multiprocessing confusion

This is the place for queries that don't fit in any of the other categories.

multiprocessing confusion

Postby metulburr » Tue Apr 01, 2014 7:53 pm

The last coupole of days i have been tinkering with these, and i am still not sure what is going on. I dont really know what process_object.join() does for example.

Basically i have this IRC bot, which does various things. I noticed the bot takes a command, and takes forever sometimes depending on certain websites responses. During this time the bot will take commands, but wait until the slow website is done before moving on to the next command. So my thought was to add threading or multiprocessing so that the bot can still execute other commands regardless of how long the slow website takes (or it to timeout).

So using this i started playing with multiprocessing:
Code: Select all
import time
import multiprocessing as mp
import random


def slow_worker(num, q):
   print('started {}'.format(mp.current_process().name))
   random_int = num #* random.randint(1,10)
   q.put(random_int)
   print('{} is {} slow'.format(mp.current_process().name, random_int))
   time.sleep(random_int)
   print('ended {}'.format(mp.current_process().name))
   
jobs = []
q = mp.Queue()
for i in range(5):
   p = mp.Process(target=slow_worker, args=(i, q,))
   jobs.append(p)
   p.start()
   
   print(q.get())
   #p.join()



Questions:

1)
One part i do not understrand is join(). It appears to do what i want it to do without join() by starting all threads at once and ending whenever the thread is done. However with join(), it appears to act as if there was no multiprocessing at all by waiting for the slow_worker to end before starting the next thread.

2)
I am having trouble following the control flow with the queue. With join() uncommented, the print for q.get() and prints inside the function before time.sleep() execute at the same time. My thought process is that it should print q.get() after the process is completed. Plus how would you know that by the time q.get() is called, that the functions q.put() has been executed?
New Users, Read This
OS Ubuntu 14.04, Arch Linux, Gentoo, Windows 7/8
https://github.com/metulburr
steam
User avatar
metulburr
 
Posts: 1415
Joined: Thu Feb 07, 2013 4:47 pm
Location: Elmira, NY

Re: multiprocessing confusion

Postby stranac » Tue Apr 01, 2014 8:26 pm

Well, I don't know much about multiprocessing, but I know enough to answer your questions:
metulburr wrote:with join(), it appears to act as if there was no multiprocessing at all by waiting for the slow_worker to end before starting the next thread

The join method waits for the process to complete. Your loop can't continue if it's waiting...
metulburr wrote:With join() uncommented, the print for q.get() and prints inside the function before time.sleep() execute at the same time. My thought process is that it should print q.get() after the process is completed.

Why would it wait for the process to complete? I thought not blocking was why you're using the multiprocessing lib?
metulburr wrote:Plus how would you know that by the time q.get() is called, that the functions q.put() has been executed?

By default, multiprocessing.Queue.get() blocks until there's something in the queue.

Btw, sounds like you want to use a library that supports asynchronous IO.
I would recommend Twisted instead of multiprocessing.
Friendship is magic!

R.I.P. Tracy M. You will be missed.
User avatar
stranac
 
Posts: 1114
Joined: Thu Feb 07, 2013 3:42 pm

Re: multiprocessing confusion

Postby metulburr » Tue Apr 01, 2014 8:59 pm

hmm, i guess maybe i should play with it more.

The bot i started (at the time) twisted was not available for python3.x and the bot is 3.x, so i used sockets. I havent checked but i am not even sure if it still is not available or not, lol. Either way i figure this would be a good time to learn and understand the concept of threading and multiprocessing anyways.
New Users, Read This
OS Ubuntu 14.04, Arch Linux, Gentoo, Windows 7/8
https://github.com/metulburr
steam
User avatar
metulburr
 
Posts: 1415
Joined: Thu Feb 07, 2013 4:47 pm
Location: Elmira, NY


Return to General Coding Help

Who is online

Users browsing this forum: Google [Bot] and 4 guests