Timeouts

Functions, jobs and scheduled tasks are subject to timeouts, which can be configured using the timeout keyword argument.

import multinode as mn

@mn.scheduled_task(
    timeout=timedelta(minutes=30),
    period=timedelta(days=1)
)
def run_long_task():
    # your code

@mn.job(timeout=timedelta(hours=12))
def run_very_long_job():
    # your code

The default timeouts are:

• function: 15 minutes
• job: 60 minutes
• scheduled task: 60 minutes

The maximum allowed timeout for a function, job or scheduled task is 1 day.

If a workload exceeds the timeout, a TimeoutError will be raised inside the code. You can catch it using try-except-finally blocks to terminate your computation gracefully.

import multinode as mn

@mn.function()
def run_stuff():
    try:
        # Your code
    except TimeoutError:
        # e.g. log the timeout
    finally:
        # Do some cleanup

The code will be forcibly killed if it doesn't finish within 5 minutes of the timeout.

Keep in mind that setting timeout=timedelta(hours=24) does not absolutely guarantee that your code will run uninterrupted for an entire day. Under exceptionally rare circumstances, your code may be interrupted due to hardware failures.