Container Image Build Optimization: Multi-Stage Builds and Layer Caching

Why Your Docker Builds Take 10 Minutes

Most Dockerfiles I encounter in production codebases are written once during initial setup and never optimized. They work — they produce a valid image that runs the application. But they take 8-12 minutes to build, produce 1GB+ images that take 30 seconds to pull, and invalidate the entire layer cache when a single source file changes. In a CI/CD pipeline that runs 50+ builds per day, those inefficiencies compound into hours of wasted developer time.

The techniques here have reduced our average build time from 9 minutes to under 2 minutes and image sizes from 1.2GB to 180MB.

Multi-Stage Builds: The Foundation

If your Dockerfile doesn't use multi-stage builds, start here. A multi-stage build separates the build environment (compilers, build tools, dev dependencies) from the runtime environment (just your application binary and runtime dependencies). The final image contains only what's needed to run.

Here's a real example for a Go service:

# Build stage
FROM golang:1.21-bookworm AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w" -o /app/server ./cmd/server

# Runtime stage
FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=builder /app/server /server
COPY --from=builder /app/migrations /migrations
EXPOSE 8080
ENTRYPOINT ["/server"]

The build stage uses golang:1.21-bookworm — a 700MB image with the full Go toolchain. The runtime stage uses distroless/static — a 2MB image with nothing but the OS libraries needed to run a static binary. The final image is roughly 15MB instead of 700MB+.

For Node.js applications, the pattern looks different because you need the Node.js runtime:

# Dependencies stage
FROM node:20-bookworm-slim AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --omit=dev

# Build stage
FROM node:20-bookworm-slim AS builder
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci
COPY . .
RUN npm run build

# Runtime stage
FROM node:20-bookworm-slim
WORKDIR /app
RUN groupadd -r appuser && useradd -r -g appuser appuser
COPY --from=deps /app/node_modules ./node_modules
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/package.json ./
USER appuser
EXPOSE 3000
CMD ["node", "dist/index.js"]

The deps stage installs only production dependencies. The builder stage installs everything (including devDependencies like TypeScript), compiles the code, then the runtime stage copies only the compiled output and production node_modules. The final image skips TypeScript, ESLint, testing frameworks — everything that's only needed at build time.

Layer Ordering for Cache Efficiency

Docker caches each layer independently. When a layer changes, every subsequent layer is rebuilt. This means the order of your COPY instructions has a massive impact on build time.

The golden rule: copy things that change infrequently first, things that change frequently last. Dependencies change less often than source code. Configuration changes less often than application logic.

Bad ordering (common in beginner Dockerfiles):

# BAD: any source file change invalidates npm install cache
COPY . .
RUN npm ci
RUN npm run build

Correct ordering:

# GOOD: npm install only re-runs when package files change
COPY package.json package-lock.json ./
RUN npm ci
COPY . .
RUN npm run build

With the correct ordering, changing a single TypeScript file doesn't re-run npm ci. The dependency layer is cached, and only the COPY and build steps re-run. This alone typically saves 2-4 minutes per build.

Granular COPY for Large Projects

For monorepos or projects with distinct build phases, you can get even more granular. Instead of COPY . ., copy specific directories in order of change frequency:

COPY package.json package-lock.json ./
RUN npm ci
COPY tsconfig.json ./
COPY src/types/ ./src/types/
COPY src/config/ ./src/config/
COPY src/ ./src/
RUN npm run build

Type definitions and configuration rarely change. Source code changes frequently. By copying them separately, a change to a source file doesn't invalidate the layers that copied types and config.

BuildKit and Cache Mounts

Docker BuildKit (enabled by default in Docker 23+) supports cache mounts — directories that persist across builds without being included in the final layer. This is transformative for package manager caches.

# syntax=docker/dockerfile:1
FROM python:3.12-slim AS builder
WORKDIR /app
COPY requirements.txt .
RUN --mount=type=cache,target=/root/.cache/pip     pip install -r requirements.txt --target=/app/deps

FROM python:3.12-slim
WORKDIR /app
COPY --from=builder /app/deps /usr/local/lib/python3.12/site-packages/
COPY . .
CMD ["python", "main.py"]

The --mount=type=cache,target=/root/.cache/pip tells BuildKit to mount a persistent cache directory at pip's cache location. The first build downloads all packages. Subsequent builds reuse cached packages and only download what's changed. With a large requirements.txt (200+ packages), this cuts dependency installation from 3 minutes to 15 seconds.

The same pattern works for every package manager:

# Go modules
RUN --mount=type=cache,target=/go/pkg/mod     go mod download

# npm
RUN --mount=type=cache,target=/root/.npm     npm ci

# Gradle
RUN --mount=type=cache,target=/root/.gradle     ./gradlew build

CI-Specific Optimizations

CI environments don't have a local Docker layer cache between builds (unless you configure one). Without external caching, every CI build starts from scratch, and your carefully ordered layers don't help because there's no cache to hit.

The solution is registry-based caching. BuildKit can push and pull cache layers from a container registry:

# Build with cache export
docker buildx build   --cache-from type=registry,ref=registry.internal/my-app:cache   --cache-to type=registry,ref=registry.internal/my-app:cache,mode=max   -t registry.internal/my-app:$SHA   --push .

The --cache-from pulls cached layers from the registry before building. The --cache-to pushes updated layers after building. The mode=max caches all layers, not just the final stage's layers — without it, intermediate build stage layers aren't cached, and your multi-stage build gets no benefit.

GitHub Actions has native support for BuildKit caching with the docker/build-push-action:

- name: Build and push
  uses: docker/build-push-action@v5
  with:
    context: .
    push: true
    tags: registry.internal/my-app:${{ github.sha }}
    cache-from: type=gha
    cache-to: type=gha,mode=max

Image Size Reduction

Beyond multi-stage builds, there are several techniques that reduce final image size. First, use slim or distroless base images. node:20 is 1.1GB. node:20-slim is 200MB. node:20-alpine is 180MB. For Go, Rust, or any language that compiles to a static binary, distroless/static at 2MB or scratch at 0 bytes.

Second, clean up in the same layer that creates the mess. Every RUN instruction creates a layer, and layers are additive — deleting a file in a later layer doesn't reduce image size because the file still exists in the earlier layer.

# BAD: apt cache persists in the layer
RUN apt-get update && apt-get install -y curl
RUN rm -rf /var/lib/apt/lists/*  # this doesn't help!

# GOOD: clean up in the same RUN instruction
RUN apt-get update &&     apt-get install -y --no-install-recommends curl &&     rm -rf /var/lib/apt/lists/*

Third, use .dockerignore aggressively. Every file in the build context gets sent to the Docker daemon, even if no COPY instruction references it. A .git directory, node_modules from local development, test fixtures — all of it slows down the build context transfer.

# .dockerignore
.git
node_modules
*.md
.env*
tests/
coverage/
.vscode/
docker-compose*.yml

Security Considerations

Build optimization and security overlap more than people expect. Running as non-root in the final image is both a security measure and a best practice that catches permission issues early:

FROM node:20-bookworm-slim
RUN groupadd -r app && useradd -r -g app -d /app -s /sbin/nologin app
WORKDIR /app
COPY --chown=app:app --from=builder /app/dist ./dist
COPY --chown=app:app --from=deps /app/node_modules ./node_modules
USER app
CMD ["node", "dist/index.js"]

Pin your base image digests in production Dockerfiles. FROM node:20-slim is a mutable tag — it points to a different image every time a security patch is released. That's good for getting patches, but bad for reproducible builds. Use the digest for production images and update it deliberately:

FROM node:20-slim@sha256:a1b2c3d4e5f6...

Scan your final images with Trivy or Grype in CI. Don't scan the build stage — it has build tools with known vulnerabilities that don't matter because those tools aren't in the final image. Scan only what ships:

trivy image --severity HIGH,CRITICAL registry.internal/my-app:$SHA

Block the deployment if critical vulnerabilities are found. The cost of fixing a vulnerability before deployment is a fraction of fixing it after an incident.

Build Parallelization

For projects with multiple services in a monorepo, build only what changed. Most CI systems support path-based triggers - only run the api-server build when files under services/api-server/ change. But even within a single service, you can parallelize build steps.

BuildKit supports parallel stage execution. If your Dockerfile has multiple independent stages (say, a frontend-build and a backend-build), BuildKit detects they're independent and runs them concurrently. On a 4-core build machine, this cuts the total build time nearly in half compared to sequential execution.

Measuring Build Performance

You can't optimize what you don't measure. Track these metrics for your Docker builds:

Total build time per service, broken down by stage. Use the progress=plain flag to see timing for each step and export this to your CI metrics system. Cache hit rate - how often are layers reused vs rebuilt? A low cache hit rate means your layer ordering needs work or your CI cache isn't configured correctly.

Image size over time. Set a size budget per service and fail the build if it's exceeded. Without a budget, image sizes creep up as developers add dependencies, and nobody notices until pull times start affecting deployment speed. Our budget is 300MB for Node.js services and 50MB for Go services. Anything over that requires justification in the PR.

Pull time on a cold node. This is the metric that actually matters for autoscaling - how long does it take for a new node to pull your image and start serving traffic? We measure this by timing docker pull on a clean instance. If pull time exceeds 30 seconds, the image needs optimization. Techniques like pre-pulling images to node groups via a DaemonSet can also help, but fixing the image size is the root cause fix.

Container Runtime Considerations

The container runtime you use affects build and pull performance. containerd (the default in most managed Kubernetes services) supports lazy pulling with the stargz snapshotter - it starts the container before the entire image is pulled, downloading layers on demand as the application accesses files. For large images, this reduces startup time from 30+ seconds to under 5 seconds.

Enabling stargz requires converting your images to the eStargz format and configuring containerd with the stargz snapshotter plugin. It's not zero effort, but for teams running large images that can't easily be shrunk (ML model serving images, for instance, which bundle model weights), it's the most effective optimization available.