Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

According to the docs :

If the call hasn’t yet completed then this method will wait up to timeout seconds. If the call hasn’t completed in timeout seconds, then a TimeoutError will be raised.

But clearly this is a False statement and doesn't represent the actual behavior:

from concurrent.futures import ThreadPoolExecutor, as_completed
from time import sleep
def my_worker(sleep_time, expected_timeout):
    sleep(sleep_time)
    print(f'slept for {sleep_time} | expected timeout: {expected_timeout}')
if __name__ == '__main__':
    timeout = 1
    with ThreadPoolExecutor(2) as executor:
        futures = [executor.submit(my_worker, 3, timeout)]
        for future in as_completed(futures):
            future.result(timeout=timeout)
which results in:
slept for 3 | expected timeout: 1
There are other questions addressing a variation of the same issue with concurrent.futures objects, but none provide any clarification of the timeout argument in Future.result.
Even the timeout argument in concurrent.futures.as_completed mentioned in the other questions is not working as expected. According to the docs:
The returned iterator raises a TimeoutError if next() is called and the result isn’t available after timeout seconds from the original call to as_completed(). timeout can be an int or float.
This sets me to expect a timeout exception is to be raised after whatever timeout is specified of as_completed call, disregarding how many tasks are pending. This also doesn't work as expected:
timeout = 1
with ThreadPoolExecutor(2) as executor:
    futures = [executor.submit(my_worker, 3, timeout) for _ in range(5)]
    for future in as_completed(futures, timeout):  # should timeout after one second
        future.result()
What actually happens: as_completed ignores the timeout and awaits all tasks to finish before raising an error:
slept for 3 | expected timeout: 1
slept for 3 | expected timeout: 1
slept for 3 | expected timeout: 1
slept for 3 | expected timeout: 1
slept for 3 | expected timeout: 1
Traceback (most recent call last):
  File "my_file.py", line 16, in <module>
    completed.__next__()
  File "/usr/local/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/_base.py", line 239, in as_completed
    raise TimeoutError(
TimeoutError: 5 (of 5) futures unfinished
Even the error message indicating that 5/5 tasks failed to complete is not true, otherwise these finished - timeout prompts wouldn't have shown.
For the first implementation, you are calling Future.result inside as_completed. As timeout starts from the moment Future.result is called, and all threads are already finished by then, no error is raised. If the keyword argument is placed in as_completed instead, the exception is thrown as expected.
As for the second implementation, documentation for ThreadPoolExecutor says that
All threads enqueued to ThreadPoolExecutor will be joined before the interpreter can exit.
So the error is raised at the expected moment, just that Python waits until all threads are finished before throwing TimeoutError.
As a sanity check,
timeout = 1
with ThreadPoolExecutor(2) as executor:
    futures = [executor.submit(my_worker, 3, timeout) for _ in range(5)]
        for future in as_completed(futures, timeout=timeout):  # should timeout after one second
            future.result()
    except Exception as e:
        print(type(e).__name__)
        print(e)
outputs
TimeoutError
5 (of 5) futures unfinished
slept for 3 | expected timeout: 1
slept for 3 | expected timeout: 1
slept for 3 | expected timeout: 1
slept for 3 | expected timeout: 1
slept for 3 | expected timeout: 1
And it seems that everything does work according to the documentation. Though I do agree with you that this isn't very intuitive.
        
Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.