Kubernetes PodSpec

Kubernetes PodSpec generation

The Agent Stack for Kubernetes controller allows you to define some or all of the Kubernetes PodSpec from the following locations:

• Controller configuration: pod-spec-patch.
• Buildkite job, using the kubernetes plugin: podSpec, podSpecPatch.

With multiple PodSpec inputs provided, here is how the Agent Stack for Kubernetes controller generates a Kubernetes PodSpec:

1. Create a simple PodSpec containing a single container with the Image defined in the controller's configuration and the value of the Buildkite job's command (BUILDKITE_COMMAND).

   If the kubernetes plugin is present in the Buildkite job's plugins and contains a podSpec, use this as the starting PodSpec instead.

2. Apply the /workspace Volume.

3. Apply any extra-volume-mounts defined by the kubernetes plugin.

4. Modify any containers defined by the kubernetes plugin, overriding the command and args.

5. Add the agent container to the PodSpec.

6. Add the checkout container to the PodSpec (if skip.checkout is set to false).

7. Add init containers for the imagecheck-# containers, based on the number of unique images defined in the PodSpec.

8. Apply pod-spec-patch from the controller's configuration, using a strategic merge patch in the controller.

9. Apply podSpecPatch from the kubernetes plugin, using a strategic merge patch in the controller.

10. Ensure the a checkout container in not present after applying patching via pod-spec-patch, podSpecPatch (if skip.checkout is set to true).

11. Remove any duplicate VolumeMounts present in PodSpec after patching.

12. Create a Kubernetes Job with the final PodSpec.

PodSpec command and interpretation of arguments

In a podSpec, command must be a list of strings, since it is defined by Kubernetes. However, the Buildkite Agent Stack for Kubernetes controller runs the Buildkite Agent instead of the container's default entrypoint.

To run a command, the controller must re-interpret command into input for the Buildkite Agent. By default, the controller treats command as a sequence of multiple commands, similar to steps and commands in a pipeline.yaml file which is different to the interpretation of command (as an entrypoint vector run without a shell as a single command) in Kubernetes.

This interposer behavior can be changed using commandParams/interposer, which can have one of the following values:

• buildkite is the default, in which the Agent Stack for Kubernetes controller treats command as a sequence of multiple commands, and args as extra arguments added to the end of the last command, which are then typically interpreted by the shell.
• vector emulates the Kubernetes' interpretation in which command and args specify components of a single command intended to be run directly.
• legacy is the behavior of the Agent Stack for Kubernetes controller version 0.14.0 and earlier, where command and args are joined directly into a single command with spaces.

An example using buildkite interposer behavior:

steps:
- label: Hello World!
  agents:
    queue: kubernetes
  plugins:
  - kubernetes:
      commandParams:
        interposer: buildkite  # This is the default, and can be omitted
      podSpec:
        containers:
        - image: alpine:latest
          command:
          - set -euo pipefail
          - |-       # <-- YAML block scalars work too
            echo Hello World! > hello.txt
            cat hello.txt | buildkite-agent annotate

If you have a multi-line command, specifying the args could lead to confusion. Therefore, it is recommended to just use command.

An example using vector interposer behavior:

steps:
- label: Hello World!
  agents:
    queue: kubernetes
  plugins:
  - kubernetes:
      commandParams:
        interposer: vector
      podSpec:
        containers:
        - image: alpine:latest
          command: ['sh']
          args:
          - '-c'
          - |-
            set -eu

            echo Hello World! > hello.txt
            cat hello.txt | buildkite-agent annotate

Custom images

Almost any container image may be used, but the image must have a POSIX shell available to be executed at /bin/sh. You can specify a different image to use for a step in a step level podSpecPatch. Previously this could be done with a step level podSpec.

# pipelines.yaml
agents:
  queue: kubernetes
steps:
- name: Hello World!
  commands:
  - echo -n Hello!
  - echo " World!"
  plugins:
  - kubernetes:
      podSpecPatch:
        containers:
        - name: container-0
          image: alpine:latest

- name: Hello World from alpine!
  commands:
  - echo -n Hello
  - echo " from alpine!"
  plugins:
  - kubernetes:
      podSpecPatch:
        containers:
        - name: container-0      # <-- For the time being, specify this exactly as `container-0`.
          image: alpine:latest   #     Currently under experimentation to make this more ergonomic.