Scheduling Metaflow Flows
run command is convenient during development, you can't keep executing it
manually in production. An old-school solution would be to use a time-based scheduler
like Cron to execute the command automatically at
a set schedule, but this approach has a number of serious downsides:
What if the server running cron fails? If the scheduled command fails, how do you know it has failed? How do you see its error logs? Does the Cron server have enough capacity to handle another command? And most importantly, how should you orchestrate schedules of multiple commands so that their mutual dependencies are handled correctly?
Metaflow currently integrates with Argo Workflows (a modern, Kubernetes-native workflow orchestrator), AWS Step Functions (a managed general-purpose orchestrator), and Apache Airflow (a widely-known open-source orchestrator) which can answer these questions.
Learn more about how to deploy your Metaflow flows to these orchestrators in the following subsections: