Skip to main content

Roadmap

Metaflow has been used in production at Netflix since early 2018. The core Metaflow was open-sourced in December 2019. Some features of Metaflow are not available in open-source yet but we may open-source them later if there is sufficient external interest. These features are listed below.

Please click the link and comment / thumbs-up the corresponding GitHub issue if you want to see the feature open-sourced.

Support for Kubernetes​

Bring all of Metaflow's capabilities to the Kubernetes universe (Github issue)

Update - Metaflow 2.6.0 introduced integration with Kubernetes and Argo Workflows.

Metaflow UI(s)​

A variety of UI(s) for Metaflow - tracking flows, model monitoring, etc

Update - Metaflow GUI to track Metaflow flows is now available!

Faster and more flexible dependency management solutions​

Support for dependency management tools beyond conda and docker and address existing pain points (Github issue)

Flow Composition​

Support composing Metaflow flow from other flows (Github issue)

MetaflowBot​

A Slack bot for Metaflow. Use it to ask questions about past runs (Github issue)

Update - Metaflowbot is now available in Open Source!

Metaflow DataFrame​

Support in-memory processing of large data sets (Github issue)

More tutorials and recipes​

Provide advanced tutorials and documentation highlighting non-trivial use-cases (Github issue)

Support for hosting models as a micro-service​

An easy-to-use Function-as-a-Service -style microservice hosting platform for artifacts (e.g. models) produced by Metaflow runs (Github issue)

Support for R Lang​

Metaflow in the R language. Provide an idiomatic R API which uses the Python library as the backend (Github issue)

Update - Metaflow-R is now available!

Support deployments to a production DAG scheduler​

Netflix uses an internal DAG scheduler to orchestrate most modeling and ETL pipelines in production. Metaflow flows can be deployed to the production scheduler with a single command. A similar integration could be provided e.g. for AWS Step Functions (Github issue)

Update - Metaflow 2.1.0 introduced integration with AWS Step Functions.