Elasticity

Parsl can change the amount of resources available as workloads vary over time. Allow the amount of workers used by Parsl to change elastically through two mechanisms:

Specifying the range of number of “blocks” allowed
Controlling the target ratio of number of tasks to workers.

Controlling Number of Blocks

All Providers request resources in “blocks” of nodes. The default for most Providers is to request a single block of workers at the beginning and only request another block if the current one expires.

Allow Parsl to vary the amount of blocks through the following options:

min_blocks: Minimum number of blocks to maintain.
init_blocks: Initial number of blocks to provision at initialization of workflow.
max_blocks: Maximum number of blocks that can be active.

Parallelism

The parallelism option of a Provider controls how many blocks are requested as a function of demand. Parallelism is the ratio of the number of workers available to the number of tasks running or ready to run, and should be a value between 0 and 1.

For example:

(default) parallelism=1: Request as many workers as tasks until max_blocks is met.

blocks = min(max_blocks, ceil(num_tasks / (workers_per_node * nodes_per_block))

parallelism=0: Use the fewest resources possible. Request

blocks = min_blocks if active_tasks == 0 else max(min_blocks, 1)

parallelism=0.5: Maintain one worker for every two tasks in queue.

Example Configuration

The following configuration shows a HighThroughputExecutor which will run between 1 and 2 blocks depending on the number of tasks. Each block is one node in size and each node will host 2 workers (2 workers per block). Parallelism of 0.5 means that a new block will be requested for every 4 tasks (2 workers per block / 0.5 workers per task).

from parsl.config import Config
from parsl.providers import SlurmProvider
from parsl.executors import HighThroughputExecutor

config = Config(
    executors=[
        HighThroughputExecutor(
            max_workers_per_node=2,
            provider=SlurmProvider(
                min_blocks=1,
                init_blocks=1,
                max_blocks=2,
                nodes_per_block=1,
                parallelism=0.5
            )
        )
    ]
)

The animation below shows how the number of blocks will change with the workflow. Tasks are allocated to the first block until 5 tasks are submitted, at which point Parsl requests a new block.