Skip to main content

Composing Flows with Custom Decorators

info

This is a new feature in Metaflow 2.16. Make sure you have a recent enough version of Metaflow to use this feature.

It is common for projects to include functionality that can be reused across multiple steps and flows. For example, you might define shared, project-specific patterns for

  • Accessing data,
  • Running ETL,
  • Tracking data and model lineage,
  • Performing feature engineering and transformations,
  • Training and evaluating a model,
  • Accessing an external service, e.g. an LLM endpoint through a model router,
  • Making tools available for agentic workflows.

You can handle cases like these by developing a shared library that encapsulates the logic and importing it in your steps. Metaflow will package the library automatically for remote execution, ensuring the logic works seamlessly from local development to production deployments.

This section introduces a powerful Metaflow feature: custom decorators and mutators. While importing and using a shared library in a step is straightforward, encapsulating common logic in a decorator offers several key advantages:

  • Clean separation of concerns: Keep shared logic out of step code, improving readability and maintainability.

  • Clarity and consistency: Applying a decorator makes the use of common patterns explicit and uniform.

  • Flexibility: Easily enable, disable, or switch between behaviors without touching step logic - great for implementing pluggable logic.

  • Correctness by default: Use mutators to apply the right patterns to all relevant steps automatically, so e.g. a centralized platform team can establish paved paths for all.

  • Reusable and portable: Distribute decorators as installable packages, whether private or public. Metaflow packages them for remote execution automatically, even if they live outside your project directory structure.

note

Custom decorators and mutators let you develop, share, and reuse components across flows - without modifying Metaflow’s core behavior. If you're looking to build a deeper infrastructure integration, such as support for a new production orchestrator, you need to use the Metaflow Extension mechanism instead. For guidance, reach out on Metaflow Slack.

Overview​

The following walkthrough illustrates the features. For technical details and examples, refer to feature-specific pages below.

Custom decorators​

Custom decorators allow you to lift common logic to custom step- and flow-level decorators which behave similarly to decorators provided by Metaflow. You can add logic to be executed before, after, and instead of the user-defined step code. Read more about custom decorators and advanced decorator patterns.

Mutators​

Mutators allow you to add and remove decorators (including custom ones), Configs and Parameters in flows programmatically. Read more about mutators.

note

Currently you are not able to alter the flow structure - add and remove steps - through a mutator, but this feature is on the roadmap.

The BaseFlow pattern​

The BaseFlow pattern allows you apply mutators, Configs, and Parameters automatically to all flows derived from the BaseFlow. This allows you to templatize flows according to your project’s best practices, ensuring that all relevant decorators are applied automatically - without requiring users to remember to add them manually. Read more about the BaseFlow pattern.