Scheduling Metaflow Flows

While the run command is convenient during development, you can't keep executing it manually in production. An old-school solution would be to use a time-based scheduler like Cron to execute the command automatically at a set schedule, but this approach has a number of serious downsides:

What if the server running cron fails? If the scheduled command fails, how do you know it has failed? How do you see its error logs? Does the Cron server have enough capacity to handle another command? And most importantly, how should you orchestrate schedules of multiple commands so that their mutual dependencies are handled correctly?

Metaflow currently integrates with Argo Workflows (a modern, Kubernetes-native workflow orchestrator), AWS Step Functions (a managed general-purpose orchestrator), and Apache Airflow (a widely-known open-source orchestrator) which can answer these questions.

Learn more about how to deploy your Metaflow flows to these orchestrators in the following subsections:

• Scheduling with Argo Workflows
• Scheduling with AWS Step Functions
• Scheduling with Airflow

tip

Note that you can manage production deployments programmatically through the Deployer API.