Azure DACS Preview: Blob Storage Cache for AKS AI Workloads, Setup and Security Limits

What DACS is​

Microsoft describes DACS as a distributed, read-through cache for Azure Blob data on AKS, and Azure Container Storage enables and manages it. In practice, cache nodes keep frequently read chunks on local NVMe disks close to the GPUs. A pod that starts later doesn't have to pull the same bytes from remote Blob Storage again.

The Azure Updates entry is only a headline plus the generic definition of "In preview," which Azure describes as available to all customers for non-production use and testing. The operational details come from Microsoft Learn and the public DACS GitHub repository.

The use cases Microsoft names are:

• Inference: repeated model reads during pod startup, node replacement and cluster scale-out.
• Training: dataset reads that repeat across epochs or between jobs that reuse a dataset.
• Prefetch: preview supports prefetch, x64 and Arm64, and clusters of up to 100 nodes.

Microsoft's documentation doesn't publish benchmarks or promise a latency or cost improvement. Treat any speed-up as something to measure on your own workload.

What it is not​

DACS is separate from the persistent-volume features of Azure Container Storage. Microsoft's documentation says it doesn't create a StorageClass, persistent volume (PV) or persistent volume claim (PVC). It installs a separate extension and doesn't create a StorageClass. Azure Blob Storage remains the durable, authoritative copy of the models and datasets. Microsoft also says the cache doesn't provide persistent storage for training outputs or checkpoints.

If your runbook checks for a StorageClass after enabling DACS, it will report a failure even when the install is healthy.

Prerequisites​

The install guidance lists these requirements:

• Azure CLI extensions: aks-preview version 22.0.0b9 or later, plus current connectedk8s and k8s-extension.
• OS: a Linux node pool, because Windows isn't supported for this Azure Container Storage 2.x path.
• Disks: local NVMe data disks dedicated to the cache.
• Node label: dacs.azure.com/processing-unit=cpu, applied through node-pool configuration. Keep any existing custom labels.

Microsoft cautions that enabling DACS installs a prerequisite DaemonSet. It discovers, formats and mounts local NVMe disks on labeled nodes. Label only nodes whose disks no other storage component has claimed.

Enabling it on an existing cluster​

The documented command is:

az aks update \
  --name <cluster-name> \
  --resource-group <resource-group> \
  --enable-azure-container-storage distributedAccelerator

• Spelling: the capital "A" is required.
• Run it alone: don't combine it with the persistent-volume storage options. A public Azure CLI extensions pull request says the same: DACS has no storage-pool construct, so storage-pool and ephemeral-disk parameters are rejected and it can't be combined with other storage types in one command.
• What gets installed: the distributedaccelerator extension, the DACS controller and NVMe prerequisite DaemonSet in kube-system, and the caches.storage.azure.com custom resource definition.
• Arc: the extension installs on the AKS managed-cluster resource, so Arc-connecting the cluster isn't required.
• Cache resources: installation doesn't create a Cache resource or cache-server pods by default. A successful install does not mean a cache is ready for application traffic.

Verifying readiness​

Check the extension, not a StorageClass.

1. Run az k8s-extension show against the managed cluster with the extension name distributedaccelerator in lowercase. The extension type should be microsoft.distributedaccelerator and the provisioning state should be Succeeded.
2. Confirm the controller with kubectl rollout status deployment/dacs-manager --namespace kube-system.
3. Confirm the prerequisite DaemonSet with kubectl rollout status daemonset/dacs-prereq --namespace kube-system, and list its pods to check the cache nodes.
4. Confirm the CRD with kubectl get crd caches.storage.azure.com.

Proceed only when the controller has a ready replica and the prerequisite DaemonSet has a Ready pod on every labeled cache node. A DaemonSet with zero desired replicas doesn't prove the NVMe disks were prepared. That usually means the node label is missing.

The security caveat​

The Azure DACS GitHub repository is blunt about this. It says DACS currently assumes a single trust domain equal to the entire Kubernetes cluster. It does not yet provide tenant isolation, and it isn't suitable for multi-tenant clusters. Any workload that can reach a DACS endpoint is treated as fully trusted with all cached data and operations. The repository also says the software is pre-release, so APIs, configuration and deployment behavior may change.

If you run shared clusters with mutually untrusted teams, this preview is probably a poor fit. A dedicated cluster for one team or one trust boundary is a safer place to evaluate it.

Removal​

Delete the DACS Cache resources while the controller is still running. Then disable DACS. Disabling removes the controller and the managed prerequisite DaemonSet, so don't remove those pieces by hand.

Who should try it​

DACS fits teams scaling GPU inference where many replicas load the same large model files, or training jobs that re-read shared datasets. It is a reasonable fit if you already use Azure Blob as the source of truth and have NVMe-equipped Linux nodes to spare. Because it is a preview with a 100-node ceiling and no tenant isolation, run it in test environments first.

Two things in the documentation are worth watching. One is the extension-version requirement, aks-preview 22.0.0b9 or later. The other is the repository's warning that behavior may change between releases.