Multiprocessing pool not processing entire list : Forums : PythonAnywhere

Multiprocessing pool not processing entire list

This simple script takes a list of integers from 0 to 16 and uses a multiprocessing Pool to map each integer to a function that prints the integer. This is done continuously in a loop. I notice that on Pythonanywhere, some iterations of the loop only print some of the integers. I don't have this issue when I run the same code locally. Any ideas why this is happening?

import multiprocessing
import time


def process(i):
    print(i)


def main():
    print("Loop")
    stuff = range(17)
    with multiprocessing.Pool() as pool:
        pool.map(process, stuff)


if __name__ == "__main__":
    while 1:
        main()
        time.sleep(1)

deleted-user-5890340 | 12 posts | Jan. 20, 2022, 6:26 p.m. | permalink

The processes may not have completed before you print "Loop" again, then they would appear in a later section.

glenn | 9718 posts | PythonAnywhere staff | Jan. 20, 2022, 7:20 p.m. | permalink

I see here that map() blocks until the result is ready, so could that really be the case? https://docs.python.org/3.7/library/multiprocessing.html#multiprocessing.pool.Pool.map

deleted-user-5890340 | 12 posts | Jan. 20, 2022, 7:38 p.m. | permalink

Additionally, I don't see any appearances in a later section.

deleted-user-5890340 | 12 posts | Jan. 20, 2022, 7:40 p.m. | permalink

I notice that if I give process() a return value and print the list returned by map(), I do get the entire list, even if some integers were not printed within process().

import multiprocessing
import time


def process(i):
    print(i)
    return i


def main():
    print("Loop")
    stuff = range(17)
    with multiprocessing.Pool() as pool:
        r = pool.map(process, stuff)
    print(r)


if __name__ == "__main__":
    while 1:
        main()
        time.sleep(1)

deleted-user-5890340 | 12 posts | Jan. 20, 2022, 9:12 p.m. | permalink

I can't reproduce the behaviour you're describing -- I'm seeing all integers printed out (usually with a different order, but all of them).

pafk | 3210 posts | PythonAnywhere staff | Jan. 21, 2022, 10:29 a.m. | permalink

I'm running this as an always-on task, not sure if that makes a difference. Here's part of an example log:

2022-01-20 21:07:28 - Task preparing to start
Jan 20 21:08:28 Loop
Jan 20 21:08:28 0
Jan 20 21:08:28 1
Jan 20 21:08:28 2
Jan 20 21:08:28 3
Jan 20 21:08:28 4
Jan 20 21:08:28 5
Jan 20 21:08:28 6
Jan 20 21:08:28 7
Jan 20 21:08:28 8
Jan 20 21:08:28 9
Jan 20 21:08:28 10
Jan 20 21:08:28 11
Jan 20 21:08:28 15
Jan 20 21:08:28 16
Jan 20 21:08:28 12
Jan 20 21:08:28 13
Jan 20 21:08:28 14
Jan 20 21:08:29 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]
Jan 20 21:08:29 Loop
Jan 20 21:08:29 0
Jan 20 21:08:29 1
Jan 20 21:08:29 2
Jan 20 21:08:29 3
Jan 20 21:08:29 4
Jan 20 21:08:29 5
Jan 20 21:08:29 6
Jan 20 21:08:29 7
Jan 20 21:08:29 8
Jan 20 21:08:29 9
Jan 20 21:08:29 10
Jan 20 21:08:29 11
Jan 20 21:08:29 12
Jan 20 21:08:29 13
Jan 20 21:08:29 14
Jan 20 21:08:29 15
Jan 20 21:08:29 16
Jan 20 21:08:30 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]
Jan 20 21:08:30 Loop
Jan 20 21:08:30 0
Jan 20 21:08:30 1
Jan 20 21:08:30 2
Jan 20 21:08:30 3
Jan 20 21:08:30 4
Jan 20 21:08:30 5
Jan 20 21:08:30 6
Jan 20 21:08:30 7
Jan 20 21:08:30 8
Jan 20 21:08:30 12
Jan 20 21:08:30 13
Jan 20 21:08:30 14
Jan 20 21:08:30 15
Jan 20 21:08:30 16
Jan 20 21:08:31 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]
Jan 20 21:08:31 Loop
Jan 20 21:08:31 3
Jan 20 21:08:31 4
Jan 20 21:08:31 5
Jan 20 21:08:31 9
Jan 20 21:08:31 10
Jan 20 21:08:31 11
Jan 20 21:08:31 12
Jan 20 21:08:31 13
Jan 20 21:08:31 14
Jan 20 21:08:31 15
Jan 20 21:08:31 16
Jan 20 21:08:32 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]
Jan 20 21:08:32 Loop
Jan 20 21:08:34 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]
Jan 20 21:08:34 Loop
Jan 20 21:08:34 0
Jan 20 21:08:34 1
Jan 20 21:08:34 2
Jan 20 21:08:34 3
Jan 20 21:08:34 4
Jan 20 21:08:34 5
Jan 20 21:08:34 9
Jan 20 21:08:34 10
Jan 20 21:08:34 11
Jan 20 21:08:34 12
Jan 20 21:08:34 13
Jan 20 21:08:34 14
Jan 20 21:08:34 15
Jan 20 21:08:34 16
Jan 20 21:08:35 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]
Jan 20 21:08:35 Loop
Jan 20 21:08:35 0
Jan 20 21:08:35 1
Jan 20 21:08:35 2
Jan 20 21:08:35 3
Jan 20 21:08:35 4
Jan 20 21:08:35 5
Jan 20 21:08:35 6
Jan 20 21:08:35 7
Jan 20 21:08:35 8
Jan 20 21:08:35 9
Jan 20 21:08:35 10
Jan 20 21:08:35 11
Jan 20 21:08:35 12
Jan 20 21:08:35 13
Jan 20 21:08:35 14
Jan 20 21:08:35 15
Jan 20 21:08:35 16
Jan 20 21:08:36 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]
Jan 20 21:08:36 Loop
Jan 20 21:08:36 0
Jan 20 21:08:36 1
Jan 20 21:08:36 2
Jan 20 21:08:36 6
Jan 20 21:08:36 7
Jan 20 21:08:36 8
Jan 20 21:08:36 12
Jan 20 21:08:36 13
Jan 20 21:08:36 14
Jan 20 21:08:37 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]
Jan 20 21:08:37 Loop
Jan 20 21:08:37 0
Jan 20 21:08:37 1
Jan 20 21:08:37 2
Jan 20 21:08:37 3
Jan 20 21:08:37 4
Jan 20 21:08:37 5
Jan 20 21:08:37 6
Jan 20 21:08:37 7
Jan 20 21:08:37 8
Jan 20 21:08:37 9
Jan 20 21:08:37 10
Jan 20 21:08:37 11
Jan 20 21:08:37 12
Jan 20 21:08:37 13
Jan 20 21:08:37 14
Jan 20 21:08:37 15
Jan 20 21:08:37 16
Jan 20 21:08:38 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]
Jan 20 21:08:38 Loop
Jan 20 21:08:38 3
Jan 20 21:08:38 4
Jan 20 21:08:38 5
Jan 20 21:08:38 6
Jan 20 21:08:38 7
Jan 20 21:08:38 8
Jan 20 21:08:38 9
Jan 20 21:08:38 10
Jan 20 21:08:38 11
Jan 20 21:08:38 12
Jan 20 21:08:38 13
Jan 20 21:08:38 14
Jan 20 21:08:38 15
Jan 20 21:08:38 16
Jan 20 21:08:39 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]
Jan 20 21:08:39 Loop
Jan 20 21:08:39 0
Jan 20 21:08:39 1
Jan 20 21:08:39 2
Jan 20 21:08:39 3
Jan 20 21:08:39 4
Jan 20 21:08:39 5
Jan 20 21:08:39 6
Jan 20 21:08:39 7
Jan 20 21:08:39 8
Jan 20 21:08:39 9
Jan 20 21:08:39 10
Jan 20 21:08:39 11
Jan 20 21:08:39 12
Jan 20 21:08:39 13
Jan 20 21:08:39 14
Jan 20 21:08:39 15
Jan 20 21:08:39 16
Jan 20 21:08:40 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]

deleted-user-5890340 | 12 posts | Jan. 21, 2022, 10:41 a.m. | permalink

Add flush=True to the print call in process and restart the task -- that should "fix" the prints.

pafk | 3210 posts | PythonAnywhere staff | Jan. 21, 2022, 12:47 p.m. | permalink

@pafk it looks like the flush did the trick, thanks!

deleted-user-5890340 | 12 posts | Jan. 21, 2022, 1:12 p.m. | permalink

Great, thanks for confirming!

giles | 12074 posts | PythonAnywhere staff | Jan. 21, 2022, 4:17 p.m. | permalink