Episode 8: Autopilot
Scheduling Compute in the Cloud.
This example revisits 'Episode 06-statistics-redux: Computing in the Cloud'. With Metaflow, you don't need to make any code changes to schedule your flow in the cloud. In this example, we will schedule the stats.py
workflow using the step-functions create
command-line argument. This instructs Metaflow to schedule your flow on AWS Step Functions without changing any code. You can execute your flow on AWS Step Functions by using the step-functions trigger
command-line argument. You can use a notebook to set up a simple dashboard to monitor all of your Metaflow flows.
You can find the tutorial code on GitHub.
Showcasing:
step-functions create
command-line optionstep-functions trigger
command-line option- Accessing data locally or remotely through the Metaflow Client API
Before playing this episode:
python -m pip install notebook
python -m pip install plotly
- This tutorial requires access to compute and storage resources on AWS, which can be configured by
To play this episode:
cd metaflow-tutorials
python 02-statistics/stats.py --environment=conda --with conda:python=3.7,libraries="{pandas:0.24.2}" step-functions create --max-workers 4
python 02-statistics/stats.py step-functions trigger
jupyter-notebook 08-autopilot/autopilot.ipynb
- Open autopilot.ipynb in your remote Sagemaker notebook.