The Python Oracle

asyncio loops: how to implement asynio in an existing python program - and share variables/data?

--------------------------------------------------
Rise to the top 3% as a developer or hire one of them at Toptal: https://topt.al/25cXVn
--------------------------------------------------

Music by Eric Matyas
https://www.soundimage.org
Track title: Puzzling Curiosities

--

Chapters
00:00 Asyncio Loops: How To Implement Asynio In An Existing Python Program - And Share Variables/Data?
01:19 Accepted Answer Score 3
03:45 Thank you

--

Full question
https://stackoverflow.com/questions/7018...

--

Content licensed under CC BY-SA
https://meta.stackexchange.com/help/lice...

--

Tags
#python #pythonasyncio #asyncssh

#avk47



ACCEPTED ANSWER

Score 3


You have basically three options:

Rewrite your main loop to be asyncio compatible

A main while True loop with lots of sleeps is exactly the kind of code you want to write asynchronously. Convert this:

while True:
     task_1() # takes n ms
     sleep(0.2)
     task_2() # takes n ms
     sleep(0.4)

into this:

async def task_1():
    while True:
        stuff()
        await asyncio.sleep(0.6)

async def task_2():
    while True:
        stuff()
        await asyncio.sleep(0.01)
        other_stuff()
        await asyncio.sleep(0.8)

loop = asyncio.get_event_loop()
loop.add_task(task_1())
loop.add_task(task_2())
...
loop.run_forever()

This is the most work, but it is almost certain that your current code will be better written, clearer, easier to maintain and easier to develop if written as a bunch of coroutines. If you do this the problem goes away: with cooperative multitasking you tell the code when to yield, so sharing state is generally pretty easy. By not awaiting anything in between getting and using a state var you prevent race conditions: no need for any kind of thread-safe var.

Run your asyncio loop in a thread

Leave your current loop intact, but run your ascynio loop in a thread (or process) with either threading or multiprocessing. Expose some kind of thread-safe variable to allow the background thread to change state, or transition to a (thread safe) messaging paradigm, where the ssh thread emits messages into a queue which your main loop handles in its own time (a message could be something like ("a", 5) which would be handled by doing something like state_dict[msg[0]] == msg[1] for everything in the queue).

If you want to go this way, have a look at the multiprocessing and/or threading docs for examples of the right ways to pass variables or messages between threads. Note that this version will likely be less performant than a pure asyncio solution, particularly if your code is mostly sleeping in the main loop anyhow.

Run your synchronous code in a thread, and have asyncio in the foreground

As @MisterMiyagi points out, asyncio has loop.run_in_executor() for launching a process to run blocking code. It's more generally used to run the odd blocking bit of code without tying up the whole loop, but you can run your whole main loop in it. The same concerns about some kind of thread safe variable or message sharing apply. This has the advantage (as @MisterMiyagi points out) of keeping asyncio where it expects to be. I have a few projects which use background asyncio threads in generally non-asyncio code (event-driven gui code with an asyncio thread interacting with custom hardware over usb). It can be done, but you do have to be careful as to how you write it.

Note btw that if you do decide to use multiple threads, message-passing (with a queue) is usually easier than directly sharing variables.