Infrastructure Provisioning Without the Ticket Queue
The standard infrastructure request flow at most companies looks like this: developer fills out a form, platform team reviews it three days later, Terraform PR gets opened, another review cycle, merge, apply, notify the developer. Two weeks for an S3 bucket. Crossplane changes the model by letting developers provision infrastructure through Kubernetes-native APIs, using the same tools they already know — kubectl, Helm, GitOps.
Crossplane runs as a set of controllers inside your Kubernetes cluster. It extends the Kubernetes API with custom resource definitions (CRDs) that represent cloud resources. When a developer creates a Crossplane resource, the controller reconciles it against the cloud provider API — creating, updating, or deleting the actual infrastructure.
Provider Architecture
Crossplane uses "providers" to talk to cloud APIs. Each provider is a separate controller that handles resources for a specific cloud or service. The provider-aws handles AWS resources, provider-gcp handles GCP, and so on. There's also provider-helm and provider-kubernetes for managing Kubernetes-native resources.
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
name: provider-aws
spec:
package: xpkg.upbound.io/upbound/provider-family-aws:v1.7.0
controllerConfigRef:
name: aws-config
---
apiVersion: pkg.crossplane.io/v1alpha1
kind: ControllerConfig
metadata:
name: aws-config
spec:
podSecurityContext:
fsGroup: 2000
args:
- --max-reconcile-rate=10The --max-reconcile-rate flag is worth paying attention to. Crossplane controllers make API calls to cloud providers during every reconciliation loop. Without rate limiting, a cluster with hundreds of managed resources can hit AWS API throttling limits within minutes. I've seen this take down Terraform Cloud pipelines running in the same account because the shared API quota was exhausted.
Compositions: The Platform Team's Abstraction Layer
Raw Crossplane managed resources map 1:1 to cloud provider APIs. A Bucket resource creates an S3 bucket. An Instance creates an EC2 instance. These are powerful but too low-level for most developers — they'd need to know the same cloud-specific details that Terraform requires.
Compositions let the platform team define higher-level abstractions. Instead of exposing "create an S3 bucket with these 47 parameters," you expose "create a data store" with three options: size, encryption level, and access pattern. The Composition translates that simplified interface into the actual cloud resources with all your organization's defaults baked in.
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
name: xdatabuckets.platform.example.com
spec:
group: platform.example.com
names:
kind: XDataBucket
plural: xdatabuckets
claimNames:
kind: DataBucket
plural: databuckets
versions:
- name: v1alpha1
served: true
referenceable: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
region:
type: string
enum: [us-east-1, eu-west-1]
accessPattern:
type: string
enum: [frequent, infrequent, archive]
encrypted:
type: boolean
default: true
required: [region, accessPattern]The Composition Pipeline
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
name: databucket-aws
spec:
compositeTypeRef:
apiVersion: platform.example.com/v1alpha1
kind: XDataBucket
resources:
- name: bucket
base:
apiVersion: s3.aws.upbound.io/v1beta1
kind: Bucket
spec:
forProvider:
region: us-east-1
providerConfigRef:
name: aws-prod
patches:
- fromFieldPath: spec.region
toFieldPath: spec.forProvider.region
- name: bucket-versioning
base:
apiVersion: s3.aws.upbound.io/v1beta1
kind: BucketVersioning
spec:
forProvider:
bucketSelector:
matchControllerRef: true
versioningConfiguration:
- status: Enabled
- name: bucket-encryption
base:
apiVersion: s3.aws.upbound.io/v1beta1
kind: BucketServerSideEncryptionConfiguration
spec:
forProvider:
bucketSelector:
matchControllerRef: true
rule:
- applyServerSideEncryptionByDefault:
- sseAlgorithm: aws:kmsNow a developer creates a DataBucket claim with three fields, and Crossplane creates the S3 bucket with versioning, encryption, proper tagging, and whatever other defaults your security team requires. The developer doesn't need to know about KMS keys or versioning configuration.
GitOps Integration with ArgoCD
Crossplane works naturally with GitOps. Claims are just Kubernetes YAML — you commit them to a repository, ArgoCD syncs them to the cluster, and Crossplane reconciles the cloud resources. This gives you version-controlled infrastructure provisioning with the same review process as application deployments.
The one gotcha with ArgoCD and Crossplane: ArgoCD's health checks don't understand Crossplane's status conditions out of the box. You need custom health checks in ArgoCD's configuration to interpret Synced and Ready conditions on Crossplane resources. Without these, ArgoCD shows resources as "Progressing" indefinitely even after they're fully provisioned.
When Crossplane Doesn't Fit
Crossplane adds operational complexity. You're running additional controllers in your Kubernetes cluster, each consuming memory and making API calls. For organizations that don't have Kubernetes expertise on the platform team, introducing Crossplane means learning Kubernetes operations on top of cloud infrastructure management.
Terraform with a good module library and a self-service frontend (like Backstage or Port) can achieve similar results with a more familiar toolchain. The tradeoff is that Terraform requires a pipeline to apply changes, while Crossplane reconciles continuously. If your infrastructure changes infrequently, the pipeline model is simpler. If developers need infrastructure provisioned and deprovisioned frequently (like ephemeral environments), Crossplane's continuous reconciliation model fits better.
State management is another consideration. Crossplane's state lives in the Kubernetes etcd database. Terraform's state lives in a backend (S3, Terraform Cloud, etc.). If your organization has invested heavily in Terraform state management, Crossplane requires rethinking that entire practice. Some teams run both — Terraform for foundational infrastructure (VPCs, accounts, IAM) and Crossplane for application-level resources (databases, queues, buckets) — but maintaining two infrastructure tools has its own cost.
Environment Management with Crossplane
One of Crossplane's strongest use cases is ephemeral environments. A developer opens a PR, and the CI pipeline creates a Crossplane claim that provisions a complete environment — database, cache, DNS entry, the works. When the PR closes, the claim gets deleted, and Crossplane tears down all the cloud resources. No orphaned infrastructure, no manual cleanup.
# Ephemeral environment claim
apiVersion: platform.example.com/v1alpha1
kind: Environment
metadata:
name: pr-1234
namespace: environments
labels:
pull-request: "1234"
team: payments
spec:
type: preview
ttl: 72h
components:
database:
engine: postgresql
size: small
cache:
engine: redis
size: small
dns:
subdomain: pr-1234The TTL field is worth highlighting. Crossplane doesn't natively support TTL-based deletion, but you can add it with a small controller that watches for expired resources and deletes them. This catches the cases where a PR gets abandoned without being closed — the environment gets cleaned up automatically after 72 hours.
Handling Secrets in Compositions
Crossplane generates connection details (database passwords, endpoints, connection strings) as Kubernetes Secrets. These secrets need to reach the applications that use them, which gets tricky in multi-namespace or multi-cluster setups. The External Secrets Operator can sync Crossplane-generated secrets to other namespaces or clusters, but the wiring requires careful planning.
A pattern that's worked for our team: Crossplane writes connection secrets to a dedicated namespace, the application's Helm chart references those secrets by convention-based names, and a mutating webhook injects the secret references at deployment time. This keeps the application's Helm chart simple — it doesn't need to know about Crossplane — while maintaining the secret delivery pipeline.
Scaling Considerations
Crossplane's reconciliation model means every managed resource generates API calls to the cloud provider on every sync cycle (default: every 1-10 minutes depending on the provider). A cluster managing 500 cloud resources makes thousands of API calls per hour. At 2000+ resources, you'll start hitting cloud provider API rate limits unless you tune the reconciliation interval and max concurrency settings.
We run separate Crossplane installations per cloud account rather than one central Crossplane managing everything. This distributes the API load, isolates blast radius (a misconfigured provider in one account doesn't affect others), and aligns with most organizations' account-per-team or account-per-environment structure. The tradeoff is managing multiple Crossplane deployments, but if you're already running multiple Kubernetes clusters, it fits the existing operational model.
Debugging Crossplane Resources
When a Crossplane managed resource gets stuck in a non-ready state, the debugging process isn't always intuitive. The resource's status conditions tell you what Crossplane thinks is happening, but the actual error often lives in the provider controller's logs. A resource stuck at "ReconcileError" might mean the cloud API returned a permissions error, or it might mean the resource's spec has an invalid combination of parameters that the CRD validation didn't catch.
Start with kubectl describe on the managed resource — the events section usually contains the most useful information. If that doesn't explain the issue, check the provider pod's logs filtered by the resource's external name. For AWS resources, the provider logs include the full AWS API error response, which is more specific than the summary in the resource's status.
# Debugging a stuck Crossplane resource
kubectl describe bucket.s3.aws.upbound.io my-bucket -n crossplane-system
# Check provider logs for detailed errors
kubectl logs -n crossplane-system \
$(kubectl get pods -n crossplane-system -l pkg.crossplane.io/revision -o name | head -1) \
| grep my-bucketComposition debugging adds another layer. When a Composite Resource fails, you need to figure out which composed resource within the composition caused the failure. The composite resource's status shows the status of each composed resource, but matching them to their templates in the Composition YAML requires cross-referencing resource names — something that's tedious without tooling. The crossplane beta trace command (available in newer CLI versions) visualizes the full resource tree and makes this significantly easier.
Resource drift is another debugging scenario. Crossplane continuously reconciles managed resources against their desired state. If someone modifies a cloud resource manually (through the console or CLI), Crossplane will revert the change on the next reconciliation. This is usually desirable, but during incident response it can be frustrating — you fix something manually and Crossplane undoes your fix. Understanding this behavior before an incident saves confusion during one.