@kubernetes
The @kubernetes decorator sends a step for execution on a Kubernetes cluster. For more information, see Executing Tasks Remotely.
For options related to tmpfs, see Using metaflow.S3 for in-memory processing.
from metaflow import kubernetes
Specifies that this step should execute on Kubernetes.
cpu: int, default 1
Number of CPUs required for this step. If @resources is
also present, the maximum value from all decorators is used.
memory: int, default 4096
Memory size (in MB) required for this step. If
@resources is also present, the maximum value from all decorators is
used.
disk: int, default 10240
Disk size (in MB) required for this step. If
@resources is also present, the maximum value from all decorators is
used.
image: str, optional, default None
Docker image to use when launching on Kubernetes. If not specified, and METAFLOW_KUBERNETES_CONTAINER_IMAGE is specified, that image is used. If not, a default Docker image mapping to the current version of Python is used.
image_pull_policy: str, default KUBERNETES_IMAGE_PULL_POLICY
If given, the imagePullPolicy to be applied to the Docker image of the step.
service_account: str, default METAFLOW_KUBERNETES_SERVICE_ACCOUNT
Kubernetes service account to use when launching pod in Kubernetes.
secrets: List[str], optional, default None
Kubernetes secrets to use when launching pod in Kubernetes. These
secrets are in addition to the ones defined in METAFLOW_KUBERNETES_SECRETS
in Metaflow configuration.
namespace: str, default METAFLOW_KUBERNETES_NAMESPACE
Kubernetes namespace to use when launching pod in Kubernetes.
gpu: int, optional, default None
Number of GPUs required for this step. A value of zero implies that the scheduled node should not have GPUs.
gpu_vendor: str, default KUBERNETES_GPU_VENDOR
The vendor of the GPUs to be used for this step.
tolerations: List[str], default []
The default is extracted from METAFLOW_KUBERNETES_TOLERATIONS. Kubernetes tolerations to use when launching pod in Kubernetes.
use_tmpfs: bool, default False
This enables an explicit tmpfs mount for this step.
tmpfs_tempdir: bool, default True
sets METAFLOW_TEMPDIR to tmpfs_path if set for this step.
tmpfs_size: int, optional, default: None
The value for the size (in MiB) of the tmpfs mount for this step.
This parameter maps to the --tmpfs option in Docker. Defaults to 50% of the
memory allocated for this step.
tmpfs_path: str, optional, default /metaflow_temp
Path to tmpfs mount for this step.
persistent_volume_claims: Dict[str, str], optional, default None
A map (dictionary) of persistent volumes to be mounted to the pod for this step. The map is from persistent
volumes to the path to which the volume is to be mounted, e.g., {'pvc-name': '/path/to/mount/on'}.
shared_memory: int, optional
Shared memory size (in MiB) required for this step
port: int, optional
Port number to specify in the Kubernetes job object