@kubernetes
The @kubernetes decorator sends a step for execution on a Kubernetes cluster. For more information, see Executing Tasks Remotely.
For options related to tmpfs, see Using metaflow.S3 for in-memory processing.
from metaflow import kubernetes
Specifies that this step should execute on Kubernetes.
cpu: int, default: 1
Number of CPUs required for this step. If @resources is
also present, the maximum value from all decorators is used.
memory: int, default: 4096
Memory size (in MB) required for this step. If
@resources is also present, the maximum value from all decorators is
used.
disk: int, default: 10240
Disk size (in MB) required for this step. If
@resources is also present, the maximum value from all decorators is
used.
image: str, optional
Docker image to use when launching on Kubernetes. If not specified, and METAFLOW_KUBERNETES_CONTAINER_IMAGE is specified, that image is used. If not, a default Docker image mapping to the current version of Python is used.
image_pull_policy: str, default: KUBERNETES_IMAGE_PULL_POLICY
If given, the imagePullPolicy to be applied to the Docker image of the step.
service_account: str, default: METAFLOW_KUBERNETES_SERVICE_ACCOUNT
Kubernetes service account to use when launching pod in Kubernetes.
secrets: List[str], optional
Kubernetes secrets to use when launching pod in Kubernetes. These
secrets are in addition to the ones defined in METAFLOW_KUBERNETES_SECRETS
in Metaflow configuration.
namespace: str, default: METAFLOW_KUBERNETES_NAMESPACE
Kubernetes namespace to use when launching pod in Kubernetes.
gpu: int, optional
Number of GPUs required for this step. A value of zero implies that the scheduled node should not have GPUs.
gpu_vendor: str, default: KUBERNETES_GPU_VENDOR
The vendor of the GPUs to be used for this step.
tolerations: List[str], default: METAFLOW_KUBERNETES_TOLERATIONS
Kubernetes tolerations to use when launching pod in Kubernetes.
use_tmpfs: bool, default: False
This enables an explicit tmpfs mount for this step.
tmpfs_tempdir: bool, default: True
sets METAFLOW_TEMPDIR to tmpfs_path if set for this step.
tmpfs_size: int, optional
The value for the size (in MiB) of the tmpfs mount for this step.
This parameter maps to the --tmpfs option in Docker. Defaults to 50% of the
memory allocated for this step.
tmpfs_path: string, optional
Path to tmpfs mount for this step. Defaults to /metaflow_temp.
persistent_volume_claims: Dict[str, str], optional
A map (dictionary) of persistent volumes to be mounted to the pod for this step. The map is from persistent
volumes to the path to which the volume is to be mounted, e.g., {'pvc-name': '/path/to/mount/on'}.