I recently switched from using threads to using asyncio, even though I had never used asyncio before.
It was a combination of:
Me using cheaper “batch” LLM API calls, which can take hours to return a result
Therefore wanting to run many thousands of tasks in parallel from within one program (to make up for the slow sequential speed of each task)
But at some point, the thread pool raised a generic “can’t start a new thread” exception, without giving too much more information. It must have hit a limit somewhere (memory? hardcoded thread limit?), although I couldn’t work out where.
Maybe the general point is that threads have more overhead, and if you’re doing many thousands of things in parallel, asyncio can handle it more reliably.
I recently switched from using threads to using asyncio, even though I had never used asyncio before.
It was a combination of:
Me using cheaper “batch” LLM API calls, which can take hours to return a result
Therefore wanting to run many thousands of tasks in parallel from within one program (to make up for the slow sequential speed of each task)
But at some point, the thread pool raised a generic “can’t start a new thread” exception, without giving too much more information. It must have hit a limit somewhere (memory? hardcoded thread limit?), although I couldn’t work out where.
Maybe the general point is that threads have more overhead, and if you’re doing many thousands of things in parallel, asyncio can handle it more reliably.