Skip to main content

Using Kubernetes

Here are some useful tips and tricks related to running Metaflow on Kubernetes. See our engineering resources for additional information about setting up and operating Kubernetes for Metaflow.

What value of @timeout should I set?

Metaflow sets a default timeout of 5 days so that you tasks don't get stuck infinitely while running on Kubernetes. For more details on how to use @timeout please read this.

How much @resources can I request?

Here are the current defaults for different resource types:

  • cpu: 1
  • memory: 4096 (4GB)
  • disk: 10240 (10GB)

When setting @resources, keep in mind the configuration of your Kubernetes cluster. Your pod will be stuck in an unschedulable state if Kubernetes is unable to provision the requested resources. Additionally, as a good measure, don't request more resources than what your workflow actually needs. On the other hand, never optimize resources prematurely.

You can place your Kubernetes pod in a specific namespace by using the namespace argument. By default, all pods execute on a vanilla python docker image corresponding to the version of Python interpreter used to launch the flow and can be overridden using the image argument.

You can also specify the resource requirements on command line as well:

$ python run --with kubernetes:cpu=4,memory=10000,namespace=foo,image=ubuntu:latest

How to configure GPUs for Kubernetes?

Metaflow compute tasks can run on any Kubernetes cluster. For starters, take a look at the Kubernetes documentation on Scheduling GPUs. The guide explains how to install Kubernetes Device Plugins so your cluster exposes a custom schedulable resource such as or, which Metaflow’s Kubernetes resources integration is already configured to call when a user specifies a decorator like @kubernetes(gpu=1).

For additional information, take a look at cloud-specific documentation:

Reach out to Metaflow Slack channel if you need help setting up a cluster.

A @kubernetes task has been stuck in PENDING forever. What should I do?

Are the resources requested in your Metaflow code/command sufficient? Especially when using custom GPU images, you might need to increase the requested memory to pull the container image into your compute environment.

Accessing Kubernetes logs

As a convenience feature, you can also see the logs of any past step as follows:

$ python logs 15/end

Disk space

You can request higher disk space for pods by using the disk attribute of @kubernetes.