kpack via GitOps

The kpack controller is installed via GitOpsGitOps
Declarative delivery from Git — manifests live in a repository; Argo CD applies changes on sync.. Credential bootstrap (IRSAIRSA
IAM Roles for Service Accounts — pods use a Kubernetes service account annotated with an IAM role ARN., ECRECR
Amazon Elastic Container Registry — stores OCI images built by kpack; EKS pulls from here. docker secret, CodeCommitCodeCommit
AWS managed Git hosting — GitHub alternative used for app source and deploy manifests in this lab. git secret) stays outside Git.

Close

Work from the repository root.

Prerequisites

• Argo CD capability ACTIVE
• CodeCommit publish — pulse-deploy has main (includes kpack/)
• Bootstrap AWS rendered demo/.generated/kpack/image.yaml

Set variables

export AWS_PROFILE=sandbox
export AWS_PAGER=""
export AWS_REGION=ap-southeast-2
export CLUSTER_NAME=cluster-1
export ARGOCD_ROLE_NAME=ArgoCDCapabilityRole
export CODECOMMIT_DEPLOY=pulse-deploy
export CODECOMMIT_APP_URL="https://git-codecommit.${AWS_REGION}.amazonaws.com/v1/repos/pulse-app"

export AWS_ACCOUNT_ID="$(aws sts get-caller-identity --query Account --output text)"
export ACCOUNT_ID="$AWS_ACCOUNT_ID"
export ARGOCD_ROLE_ARN="arn:aws:iam::${ACCOUNT_ID}:role/${ARGOCD_ROLE_NAME}"
export CLUSTER_ARN=$(aws eks describe-cluster \
  --name "$CLUSTER_NAME" \
  --region "$AWS_REGION" \
  --query 'cluster.arn' \
  --output text)
export CODECOMMIT_DEPLOY_URL="https://git-codecommit.${AWS_REGION}.amazonaws.com/v1/repos/${CODECOMMIT_DEPLOY}"
export ECR_REGISTRY="${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com"
export ECR_URI="${ECR_REGISTRY}/pulse"

export KPACK_NAMESPACE=default
export KPACK_SERVICE_ACCOUNT=kpack-sa
export KPACK_IRSA_ROLE_NAME=cluster-1-kpack-ecr

echo "CLUSTER_ARN=$CLUSTER_ARN"

Note

Auto Mode may start with no nodes. The first builder/build schedules pods and capacity scales in — expect several extra minutes.

Register this cluster for Argo CD

Managed Argo CD does not deploy to https://kubernetes.default.svc. Register the local EKS cluster by ARN, then grant the capability role permission to sync (same step Kueue via GitOps uses before installing the controller):

mkdir -p demo/.generated/argocd

cat > demo/.generated/argocd/in-cluster-secret.yaml <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: in-cluster
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: cluster
stringData:
  name: in-cluster
  server: ${CLUSTER_ARN}
  project: default
EOF

kubectl apply -f demo/.generated/argocd/in-cluster-secret.yaml

aws eks associate-access-policy \
  --cluster-name "$CLUSTER_NAME" \
  --principal-arn "$ARGOCD_ROLE_ARN" \
  --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy \
  --access-scope type=cluster

On a re-run, associate-access-policy may report that the policy is already associated — that is fine.

See Register target clusters.

Deploy kpack controller with Argo CD

Same CodeCommit repo as Pulse (pulse-deploy), different Application path. Destination server must be the EKS cluster ARN:

cat > demo/.generated/argocd/kpack-application.yaml <<EOF
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: kpack
  namespace: argocd
spec:
  project: default
  source:
    repoURL: ${CODECOMMIT_DEPLOY_URL}
    targetRevision: main
    path: kpack
  destination:
    server: ${CLUSTER_ARN}
    namespace: kpack
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true
EOF

kubectl apply -f demo/.generated/argocd/kpack-application.yaml
kubectl -n argocd get application kpack

Wait until the Application is Synced and the controller Deployment is available:

kubectl -n argocd get application kpack -o wide
kubectl -n kpack wait --for=condition=Available deployment/kpack-controller --timeout=300s
kubectl -n kpack get pods

In the Argo CD UI you should see Application kpack owning kpack-controller / kpack-webhook (not orphan pods outside GitOps).

Note

CodeCommit auth is the capability role’s GitPull (Argo CD capability) — no Argo CD repository secret. Settings → Repositories stays empty for this path; that is expected.

Associate IAM OIDC provider

Needed for IRSA on the build service account:

eksctl utils associate-iam-oidc-provider \
  --cluster "$CLUSTER_NAME" \
  --region "$AWS_REGION" \
  --approve

Service account (IRSA)

mkdir -p demo/.generated/kpack
sed -e "s/ACCOUNT_ID/${AWS_ACCOUNT_ID}/g" -e "s/REGION/${AWS_REGION}/g" \
  demo/platform/kpack/irsa-policy.json > demo/.generated/kpack/irsa-policy.json

POLICY_ARN=$(aws iam create-policy \
  --policy-name "${KPACK_IRSA_ROLE_NAME}-policy" \
  --policy-document file://demo/.generated/kpack/irsa-policy.json \
  --query 'Policy.Arn' --output text 2>/dev/null \
  || aws iam list-policies --scope Local --query "Policies[?PolicyName==\`${KPACK_IRSA_ROLE_NAME}-policy\`].Arn | [0]" --output text)

echo "POLICY_ARN=$POLICY_ARN"

eksctl create iamserviceaccount \
  --cluster="$CLUSTER_NAME" \
  --region="$AWS_REGION" \
  --namespace="$KPACK_NAMESPACE" \
  --name="$KPACK_SERVICE_ACCOUNT" \
  --role-name="$KPACK_IRSA_ROLE_NAME" \
  --attach-policy-arn="$POLICY_ARN" \
  --approve \
  --override-existing-serviceaccounts

ECR registry secret (required)

kpack authenticates to registries with annotated secrets — IRSA alone is not enough. Without this, ClusterBuilder fails with ECR 401 Unauthorized.

ECR passwords expire (~12 hours). Re-run this block if builds start failing auth:

kubectl -n "$KPACK_NAMESPACE" create secret docker-registry ecr-credentials \
  --docker-server="$ECR_REGISTRY" \
  --docker-username=AWS \
  --docker-password="$(aws ecr get-login-password --region "$AWS_REGION")" \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl -n "$KPACK_NAMESPACE" annotate secret ecr-credentials \
  "kpack.io/docker=${ECR_REGISTRY}" --overwrite

CodeCommit git credentials

kpack clones pulse-app with a basic-auth secret annotated for the CodeCommit host.

SSO roles cannot create CodeCommit HTTPS Git credentials. Use a dedicated IAM user (lab):

GIT_USER=kpack-codecommit-git
aws iam create-user --user-name "$GIT_USER" 2>/dev/null || true
aws iam attach-user-policy --user-name "$GIT_USER" \
  --policy-arn arn:aws:iam::aws:policy/AWSCodeCommitPowerUser

CREDS=$(aws iam create-service-specific-credential \
  --user-name "$GIT_USER" \
  --service-name codecommit.amazonaws.com \
  --output json)
export GIT_USERNAME=$(echo "$CREDS" | jq -r '.ServiceSpecificCredential.ServiceUserName')
export GIT_PASSWORD=$(echo "$CREDS" | jq -r '.ServiceSpecificCredential.ServicePassword')
echo "GIT_USERNAME=$GIT_USERNAME"

kubectl -n "$KPACK_NAMESPACE" apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: codecommit-credentials
  namespace: ${KPACK_NAMESPACE}
  annotations:
    kpack.io/git: https://git-codecommit.${AWS_REGION}.amazonaws.com
type: kubernetes.io/basic-auth
stringData:
  username: ${GIT_USERNAME}
  password: ${GIT_PASSWORD}
EOF

kubectl -n "$KPACK_NAMESPACE" patch serviceaccount "$KPACK_SERVICE_ACCOUNT" \
  --type merge \
  -p '{"secrets":[{"name":"ecr-credentials"},{"name":"codecommit-credentials"}]}'

Paketo ClusterBuilder + Pulse Image

These CRs still use rendered local manifests (account-specific ECRECR
Amazon Elastic Container Registry — stores OCI images built by kpack; EKS pulls from here. / CodeCommit URLs). PaketoPaketo
Paketo Buildpacks — mature open-source builders for Go, Node, Java, Python, and more. powers ClusterBuilder default:

sed "s|ECR_URI|${ECR_URI}|g" \
  demo/platform/kpack/cluster-builder.yaml > demo/.generated/kpack/cluster-builder.yaml

grep 'tag:' demo/.generated/kpack/cluster-builder.yaml
kubectl apply -f demo/.generated/kpack/cluster-builder.yaml

Wait until the builder is Ready:

kubectl get clusterbuilder default -w
# Ready=True — Ctrl+C when done

Then apply the Image:

kubectl apply -f demo/.generated/kpack/image.yaml
kubectl get image pulse -w

Successful build pushes pulse:main to ECRECR
Amazon Elastic Container Registry — stores OCI images built by kpack; EKS pulls from here.:

aws ecr list-images --repository-name pulse

Eval tag strategy

Image tag is main. After a later rebuild, restart the workload (once Argo CD has deployed Pulse):

kubectl rollout restart deployment/pulse -n default

Next

Pulse via GitOps — second Application on the same pulse-deploy repo (path: .).