Skip to content

Packaging an Air-Gapped Kubernetes Stack with Kapitan and OCM

Air-gapped clusters are a recurring headache. You have a connected build host and a target cluster with no route to the internet, and somewhere between them sits a USB stick. Everything the cluster needs has to be packaged on the connected side: rendered manifests, Helm charts, and every container image those charts pull. The package must be genuinely self-contained, so that once it lands, the cluster pulls nothing from outside.

This post walks through one end-to-end approach using two tools that fit together well: Kapitan for configuration, and the Open Component Model (OCM) for transport. The example packages a typical "day-0" platform stack (cert-manager, trust-manager, Harbor, Argo CD, and Loki/Grafana), but the pattern works for anything.

The one idea that makes this clean

It is tempting to make Kapitan build the package: have kapitan compile shell out to the ocm CLI and emit a transport archive directly. It works, but it conflates two jobs.

Kapitan is a templating engine. It is deterministic, runs offline, and turns configuration into rendered files. Building an OCM package is a different kind of work: pulling gigabytes of multi-architecture images and invoking an external CLI, with all the network, side effects, and large artifacts that implies. Those two jobs belong in different layers:

Kapitan (template)                       ocm (build / package)
─────────────────────────                ─────────────────────────
renders component-constructor.yaml   ->  ocm add component-version    (build)
from inventory data                      ocm transfer --copy-resources (materialise)
                                         ocm sign / verify / push

Kapitan emits the build recipe; a handful of shell steps cook it. That keeps to how Kapitan already works: it renders manifests and scripts, and leaves running them to the right tool. The payoff is that kapitan compile stays fast and offline, and ocm does the heavy lifting where retries, caching, and registry credentials already live.

Everything below follows from that split.

Prerequisites

  • The new-generation ocm CLI (the open-component-model rewrite). Check with ocm version.
  • kapitan and helm.
  • A connected build host, and a target environment with an internal OCI registry (Harbor, Zot, or anything OCI 1.1 compliant).

Step 1 - List the charts

The bundle is plain inventory data, and for the common case it is just a list of charts. You do not enumerate images: the ones each chart pulls are discovered from the chart itself in Step 3. You say what you want to install; the tooling works out what that drags in.

# inventory/classes/components/ocm-airgap.yml
parameters:
  ocm_bundle:
    component: example.org/platform/bootstrap
    version: "0.1.0"
    provider: example

    charts:
      - name: cert-manager
        repo: https://charts.jetstack.io
        version: v1.20.2
      - name: harbor
        repo: https://helm.goharbor.io
        version: "1.19.1"
      # … the rest of your stack

Some charts point their images at odd registries. Argo CD, for instance, has historically used a vendor mirror for its Redis image. When you need to pin or relocate one, add an explicit images: list here and let it override discovery; otherwise you leave it out and ship exactly what the charts render.

Step 2 - Template the OCM recipe

OCM builds from a component-constructor.yaml. We render the chart side of it from the data above with a Jinja2 template:

{% set b = inventory.parameters.ocm_bundle -%}
# GENERATED by kapitan. Do not edit.
components:
  - name: {{ b.component }}
    version: "{{ b.version }}"
    provider:
      name: {{ b.provider }}
    resources:
{% for c in b.charts %}
      - name: {{ c.name }}-chart
        type: helmChart
        version: "{{ c.version }}"
        input:
          type: helm
          path: charts/{{ c.name }}-{{ c.version }}.tgz
{% endfor %}

Charts use input.type: helm, which embeds them by value from a local .tgz at construction time. The images come next, in Step 3, appended by reference (access.type: ociArtifact) once discovery has found them. By reference because the new-generation ocm only embeds images at construction with a separate plugin; referencing them now and materialising at transfer time (Step 5) is the idiomatic path, and it keeps kapitan compile cheap.

Wire the template into a target as an ordinary jinja2 input, with no ocm involved:

# inventory/targets/airgap-bundle.yml
classes:
  - components.ocm-airgap
parameters:
  kapitan:
    vars:
      target: airgap-bundle
    compile:
      - output_path: ocm
        input_type: jinja2
        input_paths:
          - templates/ocm/component-constructor.yaml

Render it:

kapitan compile -t airgap-bundle
# -> compiled/airgap-bundle/ocm/component-constructor.yaml

That output is a valid component-constructor.yaml and nothing else. No network, no ocm, and it finishes in well under a second. It is the recipe.

Step 3 - Fetch the charts and discover their images

Charts are embedded by value, so they have to exist locally before the build. Pull them once on the connected host:

helm repo add jetstack https://charts.jetstack.io
helm repo add harbor    https://helm.goharbor.io
# … your repos

helm pull jetstack/cert-manager --version v1.20.2 -d lib/ocm/charts/
helm pull harbor/harbor         --version 1.19.1  -d lib/ocm/charts/
# … one per chart

The template referenced them as charts/<name>-<version>.tgz, relative to the directory we run ocm from next.

Now discover the images. Render each chart, pull the image references out, and append them to the rendered constructor as ociArtifact resources. This is why the inventory only needed charts: the image list is computed, not typed by hand.

constructor=compiled/airgap-bundle/ocm/component-constructor.yaml
for tgz in lib/ocm/charts/*.tgz; do
  helm template release "$tgz" | yq '.. | .image? // empty'
done | sort -u | while read -r ref; do
  name=${ref##*/}; name=${name%%[:@]*}        # last path segment, minus tag/digest
  yq -i ".components[0].resources += [{
    \"name\": \"$name\",
    \"type\": \"ociArtifact\",
    \"version\": \"${ref##*:}\",
    \"access\": {\"type\": \"ociArtifact\", \"imageReference\": \"$ref\"}
  }]" "$constructor"
done

That snippet is deliberately small; harden it for your stack (handle digests, dedupe, relocate registries). Because it is deterministic, you can run it in CI and fail the build when a chart bump introduces an image your registry does not mirror yet. If a chart points somewhere you do not trust, the optional images: override from Step 1 wins.

Step 4 - Build the CTF archive

Now ocm enters, in a build script. ocm add component-version reads the rendered constructor and writes a CTF (Common Transport Format) archive, a directory holding an artifact-index.json and content-addressed blobs.

One gotcha: ocm resolves a helm input's relative path against the process working directory, not against a flag. So run it from where the charts live:

cd lib/ocm
ocm add component-version \
  --repository ../../compiled/airgap-bundle/transport-archive \
  --constructor ../../compiled/airgap-bundle/ocm/component-constructor.yaml \
  --component-version-conflict-policy replace

Inspect the result:

ocm get cv \
  "CTF::./compiled/airgap-bundle/transport-archive//example.org/platform/bootstrap:0.1.0" \
  -o yaml

At this point the CTF is small, a few hundred kilobytes. It contains the chart bytes and a list of image references, each already pinned to a digest. The actual image payload is not in it yet. This is the half-step that surprises people: the archive looks like a package, but it is really a manifest plus charts.

Step 5 - Materialise by value, then build the ISO

To make the archive self-contained, transfer it into a fresh CTF by value. --copy-resources downloads every referenced image and rewrites it as a local blob:

ocm transfer cv --copy-resources --recursive \
  "CTF::./compiled/airgap-bundle/transport-archive//example.org/platform/bootstrap:0.1.0" \
  "CTF::./transport-archive"

This is the slow, network-heavy step. Expect minutes and gigabytes, depending on your stack. When it finishes, check that the archive really has no external dependencies:

ocm get cv "CTF::./transport-archive//example.org/platform/bootstrap:0.1.0" -o yaml \
  | grep -c 'imageReference:'   # expect 0

Zero external references means the cluster will pull nothing from the internet. That is the air-gap guarantee, and it is worth checking rather than assuming.

Optionally sign the archive, then pack it into an ISO 9660 image. An ISO is a good fit for an air gap: a single immutable file you can mount read-only, burn to disc, or attach as virtual media to a server's BMC (iLO or iDRAC). Use -R (Rock Ridge) so the long sha256 blob filenames survive:

ocm sign cv --signature platform --private-key key.pem \
  "CTF::./transport-archive//example.org/platform/bootstrap:0.1.0"

xorriso -as mkisofs -R -J -joliet-long \
  -V AIRGAP_BOOTSTRAP_0_1_0 \
  -o bootstrap-0.1.0.iso transport-archive
sha256sum bootstrap-0.1.0.iso > bootstrap-0.1.0.iso.sha256

Step 6 - Import on the air-gapped side

Carry the ISO across. On the target host, check integrity, extract the archive, verify the signature, and transfer into the internal registry. ocm relocates each image into that registry by digest. You can mount the ISO read-only (mount -o loop,ro bootstrap-0.1.0.iso /mnt), or extract it without root using 7z:

sha256sum -c bootstrap-0.1.0.iso.sha256
7z x bootstrap-0.1.0.iso -o./extracted

ocm verify cv --signature platform --public-key key.pub \
  "CTF::./extracted/transport-archive//example.org/platform/bootstrap:0.1.0"

ocm transfer cv --copy-resources --recursive \
  "CTF::./extracted/transport-archive//example.org/platform/bootstrap:0.1.0" \
  registry.internal

From here, point Argo CD (or kubectl) at registry.internal for all images and charts. Nothing reaches outside the gap.

Gotchas worth knowing

A few things cost real debugging time.

By reference versus by value: in the new-generation ocm, images go in via access: ociArtifact, which is a reference. Embedding at construction needs a plugin. Materialise with ocm transfer --copy-resources, not at add time.

Plain-HTTP registries: ocm defaults to HTTPS. For an unencrypted local registry, give the target an explicit http:// scheme.

Registry compatibility: the archive uses OCI 1.1 artifact manifests. An old registry:2 (OCI 1.0) rejects them outright, while Harbor and Zot accept them. Some registries still trip over multi-architecture image indexes, so test your real target early.

Multi-arch bloat: --copy-resources copies every architecture of each image by default. If your cluster is single-arch, filter them out. The full set can easily triple the archive size.

Toolchain: Kapitan depends on jsonnet, which has no prebuilt wheel for the newest Python releases and compiles from source. On a host without a C++ toolchain, install one (brew install gcc, then CC=... CXX=... uv sync) before anything else.

The shape of it

The pipeline is four stages:

kapitan compile   ->  component-constructor.yaml      (template, offline)
ocm add           ->  CTF (charts + image refs)        (build)
ocm transfer      ->  self-contained CTF -> ISO        (materialise)
ocm transfer      ->  internal registry                (import)

Kapitan stays a templating engine. ocm does the packaging. The boundary is the one Kapitan already draws everywhere: render the recipe, run it elsewhere. Keeping it makes each side simpler. The configuration stays reproducible and reviewable, and the build is something you can rerun, cache, and sign without touching your templates.