Jump to content

As of July 17, 2015, the LabJack forums here at forums.labjack.com are shut down. New registrations, topics, and replies are disabled. All forums are in a read-only state for archive purposes.

Please visit our current forums at labjack.com/forums to view and make new posts. To post on the current forums, use your labjack.com login account. Your old LabJack forums login credentials have been retired. There are no longer separate logins for labjack.com and LabJack forums.


Photo

Multiprocessing - sharing U3 instance


  • Please log in to reply
5 replies to this topic

#1 Ulrich Leutner

Ulrich Leutner
  • Members
  • 5 posts

Posted 24 January 2013 - 11:12 AM

Hello together, currently i got a little problem, which i hope is difficult to fix. In my application, i need live streaming at possibly high rates. Threading is to slow and therefor creates reading errors. So I created a process with the python multiprocessing package, which does nothing than putting the packages into a queue, which should now be read and processed via U3.processStreamData() in the "parent" process. Here comes the tricky thing: processStreamData needs an instanciated U3 object, which is not aviable, because the U3 instance lives in the other process. I looked into the U3 sources and thought, it should be no problem to work with a copy of the initialized U3; because - as far as i can see - only the settings for decoding are read. So i wanted to send a copy.deepcopy over a pipe to the decoder process, but (well, quite clearly) this does not work, because the U3 instance is not pickleable. Does anybode know an easy solution for that? Greetings and thanks, Ulrich

#2 LabJack Support

LabJack Support
  • Admin
  • 8677 posts

Posted 24 January 2013 - 04:52 PM

Hello together,

currently i got a little problem, which i hope is difficult to fix.
In my application, i need live streaming at possibly high rates. Threading is to slow and therefor creates reading errors. So I created a process with the python multiprocessing package, which does nothing than putting the packages into a queue, which should now be read and processed via U3.processStreamData() in the "parent" process.
Here comes the tricky thing: processStreamData needs an instanciated U3 object, which is not aviable, because the U3 instance lives in the other process.
I looked into the U3 sources and thought, it should be no problem to work with a copy of the initialized U3; because - as far as i can see - only the settings for decoding are read.
So i wanted to send a copy.deepcopy over a pipe to the decoder process, but (well, quite clearly) this does not work, because the U3 instance is not pickleable.
Does anybode know an easy solution for that?

Greetings and thanks,

Ulrich


Which operating system is this on?

#3 Ulrich Leutner

Ulrich Leutner
  • Members
  • 5 posts

Posted 24 January 2013 - 10:36 PM

Ah, i forgot: This runs on Linux.
And - if relevant - the source is on
https://github.com/k.../Measurement.py
at the end of the file is the parent "DAQ_thread" and and the "Stream_Reader" class, which is a multiprocessing.Process.

Ulrich

#4 Ulrich Leutner

Ulrich Leutner
  • Members
  • 5 posts

Posted 25 January 2013 - 10:17 AM

Which operating system is this on?


OS is Ubuntu 12.10 (Linux 3.5.0; python 2.7.3)

#5 LabJack Support

LabJack Support
  • Admin
  • 8677 posts

Posted 25 January 2013 - 03:51 PM

Usually threading should be adequate. You may need a "USB high-high" configuration for maximum stream speeds. This helps USB communications times and clearing the U3's stream buffer at a fast enough rate. Read the Stream Mode section of the user's guide as it explains the "USB high-high" configuration and stream speeds:

http://labjack.com/s...users-guide/3.2

I would try the above first with threading. If you want to use multiple processes like you are doing, you can try making a class that is pickleable that only implements processStreamData and the class members it needs from LabJackPython.Device and u3.U3. This should hopefully gut out the non pickleable code such as ctype pointers. Copy the member variables needed to your new class' object before streaming, such as stream settings and calibration constants, and pipe.send that instead.

#6 Ulrich Leutner

Ulrich Leutner
  • Members
  • 5 posts

Posted 26 January 2013 - 11:19 AM

Usually threading should be adequate. You may need a "USB high-high" configuration for maximum stream speeds. This helps USB communications times and clearing the U3's stream buffer at a fast enough rate. Read the Stream Mode section of the user's guide as it explains the "USB high-high" configuration and stream speeds:

http://labjack.com/s...users-guide/3.2

I would try the above first with threading. If you want to use multiple processes like you are doing, you can try making a class that is pickleable that only implements processStreamData and the class members it needs from LabJackPython.Device and u3.U3. This should hopefully gut out the non pickleable code such as ctype pointers. Copy the member variables needed to your new class' object before streaming, such as stream settings and calibration constants, and pipe.send that instead.


At first, thank you for your answer. I guess your proposal is quite possible. processStreamData has only 50 lines and calls just very few methods.
Threading is too slow for me. My program is intended for to be used in school, where _live_ data analyis is needed. For a simple stopwatch or linear motion via a rotary-encoder, are running typically ten filter-threads, which process the measurement data.
And since in python, threads can not be "niced", theses filter are killing my daq thread. And the GIL prevents balancing over the other cpu cores.


0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users