Execution

Dagster daemon

Several Dagster features, like schedules, sensors, and run queueing, require a long-running dagster-daemon process to be included with your deployment.

Run launchers

This article applies to Dagster Open Source (OSS) deployments. For information on Dagster+, see the Dagster+ documentation.

Run coordinators

In production Dagster deployments, there are often many runs being launched at once. The run coordinator lets you control the policy that Dagster uses to manage the set of runs in your deployment.

Customizing run queue priority

You can define custom prioritization rules for your Dagster instance using concurrency settings.

Detect and restart crashed workers with run monitoring

Dagster can detect hanging runs and restart crashed run workers. Using run monitoring requires:

Configuring run retries

If you configure run retries, a new run will be kicked off whenever a run fails for any reason. Compared to op retries, the maximum retry limit for run retries applies to the whole run instead of each individual op. Run retries also handle the case where the run process crashes or is unexpectedly terminated.

Executing Dagster on Celery

Celery is an open-source Python distributed task queue system, with support for a variety of queues (brokers) and result persistence strategies (backends).

Executing Dagster on Dask

The dagster-dask module makes a **daskexecutor available, which can target either a local Dask cluster or a distributed cluster. Computation is distributed across the cluster at the execution step level -- that is, we use Dask to orchestrate execution of the steps in a job, not to parallelize computation within those steps.