Metaflow has been used in production at Netflix since early 2018. The core Metaflow was open-sourced in December 2019. Some features of Metaflow are not available in open-source yet but we may open-source them later if there is sufficient external interest. These features are listed below.
Please click the link and comment / thumbs-up the corresponding GitHub issue if you want to see the feature open-sourced.
Support for Kubernetes
Bring all of Metaflow's capabilities to the Kubernetes universe (Github issue)
A variety of UI(s) for Metaflow - tracking flows, model monitoring, etc
Update - Metaflow GUI to track Metaflow flows is now available!
Faster and more flexible dependency management solutions
Support for dependency management tools beyond conda and docker and address existing pain points (Github issue)
Support composing Metaflow flow from other flows (Github issue)
A Slack bot for Metaflow. Use it to ask questions about past runs (Github issue)
Update - Metaflowbot is now available in Open Source!
Support in-memory processing of large data sets (Github issue)
More tutorials and recipes
Provide advanced tutorials and documentation highlighting non-trivial use-cases (Github issue)
Support for hosting models as a micro-service
Support for R Lang
Update - Metaflow-R is now available!
Support deployments to a production DAG scheduler
Netflix uses an internal DAG scheduler to orchestrate most modeling and ETL pipelines in production. Metaflow flows can be deployed to the production scheduler with a single command. A similar integration could be provided e.g. for AWS Step Functions (Github issue)
Update - Metaflow 2.1.0 introduced integration with AWS Step Functions.