GIL behavior in python 3.7 multithreading
--
Music by Eric Matyas
https://www.soundimage.org
Track title: RPG Blues Looping
--
Chapters
00:00 Question
02:17 Accepted answer (Score 7)
04:07 Answer 2 (Score 2)
05:37 Thank you
--
Full question
https://stackoverflow.com/questions/5549...
Question links:
[this presentation]: https://www.dabeaz.com/python/Understand...
[this video]: https://www.youtube.com/watch?
Accepted answer links:
[Python bytecode]: https://docs.python.org/3/library/dis.ht...
--
Content licensed under CC BY-SA
https://meta.stackexchange.com/help/lice...
--
Tags
#python #multithreading #pythonmultithreading #gil
#avk47
ACCEPTED ANSWER
Score 7
Python is not executed directly. It is first compiled into so called Python bytecode. This bytecode is similar in its idea to raw assembly. The bytecode is executed.
What GIL does it doesn't allow two bytecode instructions to run in parallel. Although some opeartions (e.g. io) do release the GIL internally to allow real concurrency when it can be proved that it cannot break anything.
Now all you have to know is that count -= 1 does not compile into a single bytecode instruction. It actually compiles into 4 instructions
LOAD_GLOBAL 1 (count)
LOAD_CONST 1 (1)
INPLACE_SUBTRACT
STORE_GLOBAL 1 (count)
which roughly means
load global variable into local variable
load 1 into local variable
subtract 1 from local variable
set global to the current local variable
Each of these instruction is atomic. But the order can be mixed by threads and that's why you see what you see.
So what GIL does it makes the execution flow serial. Meaning instructions happen one after another, nothing is parallel. So when you run multiple threads in theory they will perform the same as single thread minus some time spent on (so called) context switch. My tests in Python3.6 confirm that the execution time is similar.
However in Python2.7 my tests showed significant performance degradation with threads, about 1.5x. I don't know the reason for this. Something other then GIL has to happen in the background.
ANSWER 2
Score 2
Regarding Solomon's comment, the reason the code you wrote gives inconsistent results is that python does not have atomic inplace operators. The GIL does protect python's internals from getting mixed up, but your user code does still have to protect itself. If we take a look at your countdown function using the dis module, we can see where the failure can occur.
>>> print(dis(countdown))
3 0 SETUP_LOOP 24 (to 26)
2 LOAD_GLOBAL 0 (range)
4 LOAD_FAST 0 (n)
6 CALL_FUNCTION 1
8 GET_ITER
>> 10 FOR_ITER 12 (to 24)
12 STORE_FAST 1 (i)
4 14 LOAD_GLOBAL 1 (count)
16 LOAD_CONST 1 (1)
18 INPLACE_SUBTRACT
20 STORE_GLOBAL 1 (count)
22 JUMP_ABSOLUTE 10
>> 24 POP_BLOCK
>> 26 LOAD_CONST 0 (None)
28 RETURN_VALUE
None
The subtraction operation inside the loop actually takes 4 instructions to complete. If the thread is interrupted after 14 LOAD_GLOBAL 1 (count) but before the line 20 STORE_GLOBAL 1 (count), some other thread could come in and modify count. Then when execution is passed back to the first thread, the previous value of count is used for the subtraction and the result writes over whatever modification the other thread made. Like Solomon I'm not an expert on python's low level internals, but I do believe the GIL assures bytecode instructions are atomic, but not any further.