Packaging an Air-Gapped Kubernetes Stack with Kapitan and OCM
Air-gapped clusters are a recurring headache. You have a connected build host and a target cluster with no route to the internet, and somewhere between them sits a USB stick. Everything the cluster needs has to be packaged on the connected side: rendered manifests, Helm charts, and every container image those charts pull. The package must be genuinely self-contained, so that once it lands, the cluster pulls nothing from outside.
This post walks through one end-to-end approach using two tools that fit together well: Kapitan for configuration, and the Open Component Model (OCM) for transport. The example packages a typical "day-0" platform stack (cert-manager, trust-manager, Harbor, Argo CD, and Loki/Grafana), but the pattern works for anything.
The one idea that makes this clean
It is tempting to make Kapitan build the package: have kapitan compile shell
out to the ocm CLI and emit a transport archive directly. It works, but it
conflates two jobs.
Kapitan is a templating engine. It is deterministic, runs offline, and turns configuration into rendered files. Building an OCM package is a different kind of work: pulling gigabytes of multi-architecture images and invoking an external CLI, with all the network, side effects, and large artifacts that implies. Those two jobs belong in different layers:
Kapitan (template) ocm (build / package)
───────────────────────── ─────────────────────────
renders component-constructor.yaml -> ocm add component-version (build)
from inventory data ocm transfer --copy-resources (materialise)
ocm sign / verify / push
Kapitan emits the build recipe; a handful of shell steps cook it. That keeps to
how Kapitan already works: it renders manifests and scripts, and leaves running
them to the right tool. The payoff is that kapitan compile stays fast and
offline, and ocm does the heavy lifting where retries, caching, and registry
credentials already live.
Everything below follows from that split.
Prerequisites
- The new-generation
ocmCLI (theopen-component-modelrewrite). Check withocm version. kapitanandhelm.- A connected build host, and a target environment with an internal OCI registry (Harbor, Zot, or anything OCI 1.1 compliant).
Step 1 - List the charts
The bundle is plain inventory data, and for the common case it is just a list of charts. You do not enumerate images: the ones each chart pulls are discovered from the chart itself in Step 3. You say what you want to install; the tooling works out what that drags in.
# inventory/classes/components/ocm-airgap.yml
parameters:
ocm_bundle:
component: example.org/platform/bootstrap
version: "0.1.0"
provider: example
charts:
- name: cert-manager
repo: https://charts.jetstack.io
version: v1.20.2
- name: harbor
repo: https://helm.goharbor.io
version: "1.19.1"
# … the rest of your stack
Some charts point their images at odd registries. Argo CD, for instance, has
historically used a vendor mirror for its Redis image. When you need to pin or
relocate one, add an explicit images: list here and let it override discovery;
otherwise you leave it out and ship exactly what the charts render.
Step 2 - Template the OCM recipe
OCM builds from a component-constructor.yaml. We render the chart side of it
from the data above with a Jinja2 template:
{% set b = inventory.parameters.ocm_bundle -%}
# GENERATED by kapitan. Do not edit.
components:
- name: {{ b.component }}
version: "{{ b.version }}"
provider:
name: {{ b.provider }}
resources:
{% for c in b.charts %}
- name: {{ c.name }}-chart
type: helmChart
version: "{{ c.version }}"
input:
type: helm
path: charts/{{ c.name }}-{{ c.version }}.tgz
{% endfor %}
Charts use input.type: helm, which embeds them by value from a local .tgz at
construction time. The images come next, in Step 3, appended by reference
(access.type: ociArtifact) once discovery has found them. By reference because
the new-generation ocm only embeds images at construction with a separate
plugin; referencing them now and materialising at transfer time (Step 5) is the
idiomatic path, and it keeps kapitan compile cheap.
Wire the template into a target as an ordinary jinja2 input, with no ocm
involved:
# inventory/targets/airgap-bundle.yml
classes:
- components.ocm-airgap
parameters:
kapitan:
vars:
target: airgap-bundle
compile:
- output_path: ocm
input_type: jinja2
input_paths:
- templates/ocm/component-constructor.yaml
Render it:
kapitan compile -t airgap-bundle
# -> compiled/airgap-bundle/ocm/component-constructor.yaml
That output is a valid component-constructor.yaml and nothing else. No network,
no ocm, and it finishes in well under a second. It is the recipe.
Step 3 - Fetch the charts and discover their images
Charts are embedded by value, so they have to exist locally before the build. Pull them once on the connected host:
helm repo add jetstack https://charts.jetstack.io
helm repo add harbor https://helm.goharbor.io
# … your repos
helm pull jetstack/cert-manager --version v1.20.2 -d lib/ocm/charts/
helm pull harbor/harbor --version 1.19.1 -d lib/ocm/charts/
# … one per chart
The template referenced them as charts/<name>-<version>.tgz, relative to the
directory we run ocm from next.
Now discover the images. Render each chart, pull the image references out, and
append them to the rendered constructor as ociArtifact resources. This is why
the inventory only needed charts: the image list is computed, not typed by hand.
constructor=compiled/airgap-bundle/ocm/component-constructor.yaml
for tgz in lib/ocm/charts/*.tgz; do
helm template release "$tgz" | yq '.. | .image? // empty'
done | sort -u | while read -r ref; do
name=${ref##*/}; name=${name%%[:@]*} # last path segment, minus tag/digest
yq -i ".components[0].resources += [{
\"name\": \"$name\",
\"type\": \"ociArtifact\",
\"version\": \"${ref##*:}\",
\"access\": {\"type\": \"ociArtifact\", \"imageReference\": \"$ref\"}
}]" "$constructor"
done
That snippet is deliberately small; harden it for your stack (handle digests,
dedupe, relocate registries). Because it is deterministic, you can run it in CI
and fail the build when a chart bump introduces an image your registry does not
mirror yet. If a chart points somewhere you do not trust, the optional images:
override from Step 1 wins.
Step 4 - Build the CTF archive
Now ocm enters, in a build script. ocm add component-version reads the
rendered constructor and writes a CTF (Common Transport Format) archive, a
directory holding an artifact-index.json and content-addressed blobs.
One gotcha: ocm resolves a helm input's relative path against the process
working directory, not against a flag. So run it from where the charts live:
cd lib/ocm
ocm add component-version \
--repository ../../compiled/airgap-bundle/transport-archive \
--constructor ../../compiled/airgap-bundle/ocm/component-constructor.yaml \
--component-version-conflict-policy replace
Inspect the result:
ocm get cv \
"CTF::./compiled/airgap-bundle/transport-archive//example.org/platform/bootstrap:0.1.0" \
-o yaml
At this point the CTF is small, a few hundred kilobytes. It contains the chart bytes and a list of image references, each already pinned to a digest. The actual image payload is not in it yet. This is the half-step that surprises people: the archive looks like a package, but it is really a manifest plus charts.
Step 5 - Materialise by value, then build the ISO
To make the archive self-contained, transfer it into a fresh CTF by value.
--copy-resources downloads every referenced image and rewrites it as a local
blob:
ocm transfer cv --copy-resources --recursive \
"CTF::./compiled/airgap-bundle/transport-archive//example.org/platform/bootstrap:0.1.0" \
"CTF::./transport-archive"
This is the slow, network-heavy step. Expect minutes and gigabytes, depending on your stack. When it finishes, check that the archive really has no external dependencies:
ocm get cv "CTF::./transport-archive//example.org/platform/bootstrap:0.1.0" -o yaml \
| grep -c 'imageReference:' # expect 0
Zero external references means the cluster will pull nothing from the internet. That is the air-gap guarantee, and it is worth checking rather than assuming.
Optionally sign the archive, then pack it into an ISO 9660 image. An ISO is a
good fit for an air gap: a single immutable file you can mount read-only, burn
to disc, or attach as virtual media to a server's BMC (iLO or iDRAC). Use -R
(Rock Ridge) so the long sha256 blob filenames survive:
ocm sign cv --signature platform --private-key key.pem \
"CTF::./transport-archive//example.org/platform/bootstrap:0.1.0"
xorriso -as mkisofs -R -J -joliet-long \
-V AIRGAP_BOOTSTRAP_0_1_0 \
-o bootstrap-0.1.0.iso transport-archive
sha256sum bootstrap-0.1.0.iso > bootstrap-0.1.0.iso.sha256
Step 6 - Import on the air-gapped side
Carry the ISO across. On the target host, check integrity, extract the archive,
verify the signature, and transfer into the internal registry. ocm relocates
each image into that registry by digest. You can mount the ISO read-only
(mount -o loop,ro bootstrap-0.1.0.iso /mnt), or extract it without root using
7z:
sha256sum -c bootstrap-0.1.0.iso.sha256
7z x bootstrap-0.1.0.iso -o./extracted
ocm verify cv --signature platform --public-key key.pub \
"CTF::./extracted/transport-archive//example.org/platform/bootstrap:0.1.0"
ocm transfer cv --copy-resources --recursive \
"CTF::./extracted/transport-archive//example.org/platform/bootstrap:0.1.0" \
registry.internal
From here, point Argo CD (or kubectl) at registry.internal for all images and
charts. Nothing reaches outside the gap.
Gotchas worth knowing
A few things cost real debugging time.
By reference versus by value: in the new-generation ocm, images go in via
access: ociArtifact, which is a reference. Embedding at construction needs a
plugin. Materialise with ocm transfer --copy-resources, not at add time.
Plain-HTTP registries: ocm defaults to HTTPS. For an unencrypted local
registry, give the target an explicit http:// scheme.
Registry compatibility: the archive uses OCI 1.1 artifact manifests. An old
registry:2 (OCI 1.0) rejects them outright, while Harbor and Zot accept them.
Some registries still trip over multi-architecture image indexes, so test your
real target early.
Multi-arch bloat: --copy-resources copies every architecture of each image by
default. If your cluster is single-arch, filter them out. The full set can easily
triple the archive size.
Toolchain: Kapitan depends on jsonnet, which has no prebuilt wheel for the
newest Python releases and compiles from source. On a host without a C++
toolchain, install one (brew install gcc, then CC=... CXX=... uv sync) before
anything else.
The shape of it
The pipeline is four stages:
kapitan compile -> component-constructor.yaml (template, offline)
ocm add -> CTF (charts + image refs) (build)
ocm transfer -> self-contained CTF -> ISO (materialise)
ocm transfer -> internal registry (import)
Kapitan stays a templating engine. ocm does the packaging. The boundary is the
one Kapitan already draws everywhere: render the recipe, run it elsewhere. Keeping
it makes each side simpler. The configuration stays reproducible and reviewable,
and the build is something you can rerun, cache, and sign without touching your
templates.