Tekton and GKE Workload Identity

Thu, Mar 4, 2021

What is Workload Identity?

When running a workload on Kubernetes, each Pod has an underlying service account identity that is used for interacting with the Kubernetes API and resources.

GCP has a similar concept of service accounts as a means for programs to authenticate to Google APIs. To make management of these secrets easier, GCP compute environments provide a mechanism called Application Default Credentials that allows users to map a VM to a particular service account. When this is done, GCP libraries running on that VM can fetch credentials for the configured service account automatically by using the VM’s baked in Metadata Service.

Although they share the same name, Kubernetes Service Accounts (KSA) are separate from Google Service Accounts (GSA). By default when you run a workload on Kubernetes, Application Default Credentials will use the service account associated to the Node (i.e. VM) regardless of the Pod’s KSA. If you wanted to ensure that different Pods used different GSA identities, you would either need to pass in service account credentials manually or separate workloads onto different Nodes (which means you need to peel back Kubernetes abstractions to be aware of the underlying node pools you are running on).

Both of these options aren’t great - ideally we would be able to use KSAs just like GSAs to scope GCP permissions for each running Pod. This is where GKE Workload Identity comes in! Workload Identity allows you to map KSAs to corresponding GSAs so that Application Default Credentials in Pods can fetch GSA credentials based on the Pod’s KSA identity.

This is a great feature that allows you to authenticate to external GCP services, without needing to worry about managing credentials. With Tekton this is particularly useful since we often need to interact with GCP resources as part of CI/CD:

Push/Pull from Artifact Registry
Read/Write from GCS
Git clone from Cloud Source Repositories
Deploy to Cloud Run / App Engine / GCE / GKE / etc.

Configuring Tekton

Since Tekton runs on Kubernetes, we can use Workload Identity to simplify credential management when running on GKE.

If you don’t have Workload Identity enabled on your cluster yet, check out this guide to get set up. The following examples assume that the Workload Identity feature is already enabled on your cluster.

Authentication in a Task

For this example, we will set up a simple Tekton build, using Workload Identity to handle pushing the artifact.

We will create a KSA named build-robot with a GSA with the same name e.g. build-robot@${PROJECT_ID}.iam.gserviceaccount.com.

For this example I am using wlynch-test as my GCP project - replace this with your own!

Create the KSA

$ kubectl create serviceaccount build-robot

Create the GSA

gcloud iam service-accounts create build-robot

Grant the GSA permissions to push to the image repository
For our example, we will use us-docker.pkg.dev/wlynch-test/test.
If you’re unfamiliar with the pkg.dev URL - this is an Artifact Registry URL, the successor to GCR. Highly recommend trying it out if you haven’t yet!
```
$ gcloud artifacts repositories add-iam-policy-binding test \
--location us \
--member=serviceAccount:build-robot@wlynch-test.iam.gserviceaccount.com \
--role=roles/artifactregistry.writer
```

Setup Workload Identity Mappings

KSA -> GSA

$ kubectl annotate serviceaccount \
build-robot \
iam.gke.io/gcp-service-account=build-robot@wlynch-test.iam.gserviceaccount.com

GSA -> KSA

gcloud iam service-accounts add-iam-policy-binding \
--role roles/iam.workloadIdentityUser \
--member "serviceAccount:wlynch-test.svc.id.goog[default/build-robot]" \
build-robot@wlynch-test.iam.gserviceaccount.com

Build and Push

Given the following taskrun.yaml:

apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  name: workload-identity
spec:
  serviceAccountName: build-robot
  taskSpec:
    steps:
      - name: checkout
        image: docker.io/alpine/git
        args: ["clone", "https://github.com/googlecloudplatform/cloud-run-hello", "."]
      # This step isn't necessary - it just prints out the
      # current authenticated user.
      - name: check-auth
        image: gcr.io/google.com/cloudsdktool/cloud-sdk
        args: ["gcloud", "auth", "list"]
      - name: build
        image: "gcr.io/kaniko-project/executor:v1.5.1"
        args: [
          "--dockerfile=Dockerfile",
          "--context=dir://.",
          "--destination=us-docker.pkg.dev/wlynch-test/test/cloud-run-hello:latest",
        ]

We can execute this by running:

$ kubectl apply -f taskrun.yaml

The kaniko step will automatically use Application Default Credentials + Workload Identity to get a credential for the configured GSA and push the image to the repository!

Using Private Images

In the example above, we used publicly available builder images to push a private image. But what if we want to use a private image in a build step?

This is a bit more complicated, since there’s actually multiple identities that need to read the image when Tasks are scheduled-

The underlying TaskRun Pod needs to fetch the image to run it.
The Tekton Pipeline controller needs to read the image metadata to properly set up the Tekton runtime environment.

Both KSAs need to be configured with Workload Identity in order for Tekton to handle private images.

Grant the TaskRun Pod Access

We already have Workload Identity set up from the last section, so we can reuse this and simply grant the GSA any additional read permissions it needs. For example, to grant read-only permissions to all Artifact Registry Docker repositories in the project:

$ gcloud projects add-iam-policy-binding wlynch-test \
--member=serviceAccount:build-robot@wlynch-test.iam.gserviceaccount.com \
--role=roles/artifactregistry.reader

Grant the Tekton Controller Access

To grant the Tekton controller access, we largely follow the same process as before, but use the controller’s namespace (tekton-pipelines) and service account (tekton-pipelines-controller).

Create the GSA

gcloud iam service-accounts create tekton-pipelines-controller

Grant the GSA read permissions.

$ gcloud projects add-iam-policy-binding wlynch-test \
--member=serviceAccount:tekton-pipelines-controller@wlynch-test.iam.gserviceaccount.com \
--role=roles/artifactregistry.reader