When enforcing OCI image spec broke customer containers

calendar Jun 11, 2026 · 4 min read · Containers  ·
Share on: copy

A container platform shipped a change to detect CPU architecture mismatches at image pull time. If an image's config says arm64 but the host is amd64, fail fast instead of letting the container start and die with exec format error. The implementation is correct, tests passed and reviewers approved.

However, it caused a production impact and a quick rollback. It turns out customers had been running mislabeled images for years. The existing code is permissive: it uses the architecture in image manifest (single-arch) or manifest index (multi-arch) as a hint to pull the image. It never enforced an exact match of host arch and image arch. Now that the runtime enforced it, "wrong label" became "won't start the container." A real availability issue for customers not using blue-green deployments.

How container images declare their platform

When a runtime resolves an image reference, it fetches a JSON document first. There are two shapes: single arch and multi-arch. You can inspect the shape with crane manifest <image>:<tag> or skopeo inspect --raw docker://<image>:<tag>.

Single manifest: one image, one platform. The architecture lives in the config blob, not the manifest itself:

 1# crane inspect <image>:<tag> 
 2# The mediaType is a manifest, so it's single-arch image.
 3{
 4  "schemaVersion": 2,
 5  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
 6  "config": {
 7    "mediaType": "application/vnd.docker.container.image.v1+json",
 8    "size": 903,
 9    "digest": "sha256:80405fcefab7b13394d3d7e4ca02e93fb07ca78d241b23fbcb2cbf8b841cf6aa"
10  },
11  ...
12}
13
14# crane config <image>:<tag> 
15{
16  "architecture": "arm64",
17  "os": "linux",
18  ...
19}

The registry never validates these fields. You can push amd64 layers with an arm64 label and nobody complains.

Image index (manifest list): a manifest of manifests. Each entry has an explicit platform descriptor:

 1# crane manifest <image>:<tag>
 2# The mediaType is an image index, so it's a multi-arch image.
 3
 4{
 5  "mediaType": "application/vnd.oci.image.index.v1+json",
 6  "manifests": [
 7    { "platform": { "architecture": "amd64", "os": "linux" } },
 8    { "platform": { "architecture": "arm64", "os": "linux" } }
 9  ]
10}

The change

Before the "fix", the platform handled mismatches poorly in two ways:

  1. Index with no matching entry: failed correctly, but treated it as transient and retried for up to 3 minutes.
  2. Single manifest with wrong architecture: pulled all layers, started the container, let it crash with exec format error.

The fix: fail early and be explicit. Read the config blob before pulling layers, check the platform up front, and stop retrying the index case.

 1import (
 2  "github.com/containerd/platforms"
 3  "github.com/containerd/containerd/v2/core/remotes"
 4  spec "github.com/opencontainers/image-spec/specs-go/v1"
 5)
 6
 7func getPlatform(ctx context.Context, fetcher remotes.Fetcher, manifest spec.Manifest) (arch, os string, err error) {
 8    reader, err := fetcher.Fetch(ctx, manifest.Config)
 9    if err != nil {
10        return "", "", err 
11    }
12    defer reader.Close()
13
14    var cfg spec.Platform
15    if err := json.NewDecoder(reader).Decode(&cfg); err != nil {
16        return "", "", err
17    }
18    if !platforms.Only(platforms.DefaultSpec()).Match(cfg) {
19        return "", "", fmt.Errorf("image does not match host platform")
20    }
21    return cfg.Architecture, cfg.OS, nil
22}

What broke

Some users had single-arch images labeled arm64 running on amd64 hosts. Before the fix, nobody read the Platform field. The host was the only arbiter at exec time. If the binary ran, the container ran. The new runtime made the declared field authoritative and rejected these images immediately.

The user violated the spec. The fix enforced the spec. But production doesn't run on specs. It runs on whatever has been working. The old behavior (ignore the label) was observable, and with enough users, someone inevitably depended on it. Tightening it was a breaking change dressed as a bug fix.

Reproducing a mislabeled image

1FROM public.ecr.aws/docker/library/alpine:latest

1docker buildx build --platform linux/amd64 -t "${image}:${tag}" --push .
2# intentionally change the platform to linux/arm64
3crane mutate "${image}:${tag}" --set-platform linux/arm64 -t "${image}:${wrong_tag}"

Conclusion

When working on a long-existing product, on a code base that many developers have churned through, it is difficult to know every decision baked into the code and implicit contracts (undocumented behaviors). Any change that seems correct, e.g., following spec, must be dealt with extra care. Anything that starts rejecting inputs the system previously accepted is a breaking change, regardless of what the spec says. The user should have labeled their image correctly. But "the user was holding it wrong" has never prevented an operational event.

Measure before you enforce. Ship the check in log-only mode first. Emit a metric, log a "would reject" warning, but don't fail. Bake it across the fleet. Now you know the blast radius before you flip the switch. Enforcement becomes a config change on top of a metric you already trust.

Gate it behind a feature flag for fast rollback. If the only way to undo your change is a full platform rollback, you're under-instrumented. A feature flag that disables just this check is cheaper than reverting a release.