Kubernetes executor

Option	Description
`host`	Optional Kubernetes API server host URL (auto-discovery attempted if not specified).
`cert_file`	Optional Kubernetes API server user auth certificate.
`key_file`	Optional Kubernetes API server user auth private key.
`ca_file`	Optional Kubernetes API server ca certificate.

Resource	Verb (Optional Feature/Config Flags)
events	list ( `print_pod_warning_events=true` ), watch ( `FF_PRINT_POD_EVENTS=true` )
namespaces	create ( `kubernetes.NamespacePerJob=true` ), delete ( `kubernetes.NamespacePerJob=true` )
pods	create, delete, get, list ( using Informers ), watch ( using Informers , `FF_KUBERNETES_HONOR_ENTRYPOINT=true` , `FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=false` )
pods/attach	create ( `FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=false` ), delete ( `FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=false` ), get ( `FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=false` ), patch ( `FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=false` )
pods/exec	create, delete, get, patch
pods/log	get ( `FF_KUBERNETES_HONOR_ENTRYPOINT=true` , `FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=false` , `FF_WAIT_FOR_POD_TO_BE_REACHABLE=true` ), list ( `FF_KUBERNETES_HONOR_ENTRYPOINT=true` , `FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=false` )
secrets	create, delete, get, update
serviceaccounts	get
services	create, get

Setting	Description
`cpu_limit`	The CPU allocation given to build containers.
`cpu_limit_overwrite_max_allowed`	The maximum amount that the CPU allocation can be written to for build containers. When empty, it disables the CPU limit overwrite feature.
`cpu_request`	The CPU allocation requested for build containers.
`cpu_request_overwrite_max_allowed`	The maximum amount that the CPU allocation request can be written to for build containers. When empty, it disables the CPU request overwrite feature.
`helper_cpu_limit`	The CPU allocation given to build helper containers.
`helper_cpu_limit_overwrite_max_allowed`	The maximum amount that the CPU allocation can be written to for helper containers. When empty, it disables the CPU limit overwrite feature.
`helper_cpu_request`	The CPU allocation requested for build helper containers.
`helper_cpu_request_overwrite_max_allowed`	The maximum amount that the CPU allocation request can be written to for helper containers. When empty, it disables the CPU request overwrite feature.
`service_cpu_limit`	The CPU allocation given to build service containers.
`service_cpu_limit_overwrite_max_allowed`	The maximum amount that the CPU allocation can be written to for service containers. When empty, it disables the CPU limit overwrite feature.
`service_cpu_request`	The CPU allocation requested for build service containers.
`service_cpu_request_overwrite_max_allowed`	The maximum amount that the CPU allocation request can be written to for service containers. When empty, it disables the CPU request overwrite feature.

Setting	Description
`memory_limit`	The amount of memory allocated to build containers.
`memory_limit_overwrite_max_allowed`	The maximum amount that the memory allocation can be written to for build containers. When empty, it disables the memory limit overwrite feature.
`memory_request`	The amount of memory requested from build containers.
`memory_request_overwrite_max_allowed`	The maximum amount that the memory allocation request can be written to for build containers. When empty, it disables the memory request overwrite feature.
`helper_memory_limit`	The amount of memory allocated to build helper containers.
`helper_memory_limit_overwrite_max_allowed`	The maximum amount that the memory allocation can be written to for helper containers. When empty, it disables the memory limit overwrite feature.
`helper_memory_request`	The amount of memory requested for build helper containers.
`helper_memory_request_overwrite_max_allowed`	The maximum amount that the memory allocation request can be written to for helper containers. When empty, it disables the memory request overwrite feature.
`service_memory_limit`	The amount of memory allocated to build service containers.
`service_memory_limit_overwrite_max_allowed`	The maximum amount that the memory allocation can be written to for service containers. When empty, it disables the memory limit overwrite feature.
`service_memory_request`	The amount of memory requested for build service containers.
`service_memory_request_overwrite_max_allowed`	The maximum amount that the memory allocation request can be written to for service containers. When empty, it disables the memory request overwrite feature.

Setting	Description
`ephemeral_storage_limit`	The ephemeral storage limit for build containers.
`ephemeral_storage_limit_overwrite_max_allowed`	The maximum amount that the ephemeral storage limit for build containers can be overwritten. When empty, it disables the ephemeral storage limit overwrite feature.
`ephemeral_storage_request`	The ephemeral storage request given to build containers.
`ephemeral_storage_request_overwrite_max_allowed`	The maximum amount that the ephemeral storage request can be overwritten by for build containers. When empty, it disables the ephemeral storage request overwrite feature.
`helper_ephemeral_storage_limit`	The ephemeral storage limit given to helper containers.
`helper_ephemeral_storage_limit_overwrite_max_allowed`	The maximum amount that the ephemeral storage limit can be overwritten by for helper containers. When empty, it disables the ephemeral storage request overwrite feature.
`helper_ephemeral_storage_request`	The ephemeral storage request given to helper containers.
`helper_ephemeral_storage_request_overwrite_max_allowed`	The maximum amount that the ephemeral storage request can be overwritten by for helper containers. When empty, it disables the ephemeral storage request overwrite feature.
`service_ephemeral_storage_limit`	The ephemeral storage limit given to service containers.
`service_ephemeral_storage_limit_overwrite_max_allowed`	The maximum amount that the ephemeral storage limit can be overwritten by for service containers. When empty, it disables the ephemeral storage request overwrite feature.
`service_ephemeral_storage_request`	The ephemeral storage request given to service containers.
`service_ephemeral_storage_request_overwrite_max_allowed`	The maximum amount that the ephemeral storage request can be overwritten by for service containers. When empty, it disables the ephemeral storage request overwrite feature.

Setting	Description
`affinity`	Specify affinity rules that determine which node runs the build. Read more about using affinity .
`allow_privilege_escalation`	Run all containers with the `allowPrivilegeEscalation` flag enabled. When empty, it does not define the `allowPrivilegeEscalation` flag in the container `SecurityContext` and allows Kubernetes to use the default privilege escalation behavior.
`allowed_images`	Wildcard list of images that can be specified in `.gitlab-ci.yml` . If not present all images are allowed (equivalent to `["/:*"]` ). View details .
`allowed_pull_policies`	List of pull policies that can be specified in the `.gitlab-ci.yml` file or the `config.toml` file.
`allowed_services`	Wildcard list of services that can be specified in `.gitlab-ci.yml` . If not present all images are allowed (equivalent to `["/:*"]` ). View details .
`automount_service_account_token`	Boolean to control whether the service account token automatically mounts in the build pod.
`bearer_token`	Default bearer token used to launch build pods.
`bearer_token_overwrite_allowed`	Boolean to allow projects to specify a bearer token used to create the build pod.
`build_container_security_context`	Sets a container security context for the build container. Read more about security context .
`cap_add`	Specify Linux capabilities that should be added to the job pod containers. Read more about capabilities configuration in Kubernetes executor .
`cap_drop`	Specify Linux capabilities that should be dropped from the job pod containers. Read more about capabilities configuration in Kubernetes executor .
`cleanup_grace_period_seconds`	When a job completes, the duration in seconds that the pod has to terminate gracefully. After this period, the processes are forcibly halted with a kill signal. Ignored if `terminationGracePeriodSeconds` is specified.
`dns_policy`	Specify the DNS policy that should be used when constructing the pod: `none` , `default` , `cluster-first` , `cluster-first-with-host-net` . The Kubernetes default ( `cluster-first` ) is used if not set.
`dns_config`	Specify the DNS configuration that should be used when constructing the pod. Read more about using pod’s DNS config .
`helper_container_security_context`	Sets a container security context for the helper container. Read more about security context .
`helper_image`	(Advanced) Override the default helper image used to clone repositories and upload artifacts.
`helper_image_flavor`	Sets the helper image flavor ( `alpine` , `alpine3.18` , `alpine3.19` , `alpine3.21` , or `ubuntu` ). Defaults to `alpine` . Using `alpine` is the same as `alpine3.19` .
`host_aliases`	List of additional host name aliases that are added to all containers. Read more about using extra host aliases .
`image_pull_secrets`	An array of items containing the Kubernetes `docker-registry` secret names used to authenticate Docker image pulling from private registries.
`init_permissions_container_security_context`	Sets a container security context for the init-permissions container. Read more about security context .
`namespace`	Namespace in which to run Kubernetes Pods.
`namespace_per_job`	Isolate jobs in separate namespaces. If enabled, `namespace` and `namespace_overwrite_allowed` are ignored.
`namespace_overwrite_allowed`	Regular expression to validate the contents of the namespace overwrite environment variable (documented below). When empty, it disables the namespace overwrite feature.
`node_selector`	A `table` of `key=value` pairs in the format of `string=string` ( `string:string` in the case of environment variables). Setting this limits the creation of pods to Kubernetes nodes matching all the `key=value` pairs. Read more about using node selectors .
`node_tolerations`	A `table` of `"key=value" = "Effect"` pairs in the format of `string=string:string` . Setting this allows pods to schedule to nodes with all or a subset of tolerated taints. Only one toleration can be supplied through environment variable configuration. The `key` , `value` , and `effect` match with the corresponding field names in Kubernetes pod toleration configuration.
`pod_annotations`	A `table` of `key=value` pairs in the format of `string=string` . The `table` contains a list of annotations to be added to each build pod created by the runner. The value of these can include environment variables for expansion. Pod annotations can be overwritten in each build.
`pod_annotations_overwrite_allowed`	Regular expression to validate the contents of the pod annotations overwrite environment variable. When empty, it disables the pod annotations overwrite feature.
`pod_labels`	A `table` of `key=value` pairs in the format of `string=string` . The `table` contains a list of labels to be added to each build pod created by the runner. The value of these can include environment variables for expansion. Pod labels can be overwritten in each build by using `pod_labels_overwrite_allowed` .
`pod_labels_overwrite_allowed`	Regular expression to validate the contents of the pod labels overwrite environment variable. When empty, it disables the pod labels overwrite feature. Note that pod labels in the `runner.gitlab.com` label namespace cannot be overwritten.
`pod_security_context`	Configured through the configuration file, this sets a pod security context for the build pod. Read more about security context .
`pod_termination_grace_period_seconds`	Pod-level setting which determines the duration in seconds which the pod has to terminate gracefully. After this, the processes are forcibly halted with a kill signal. Ignored if `terminationGracePeriodSeconds` is specified.
`poll_interval`	How frequently, in seconds, the runner polls the Kubernetes pod it has just created to check its status (default = 3).
`poll_timeout`	The amount of time, in seconds, that needs to pass before the runner times out attempting to connect to the container it has just created. Use this setting for queueing more builds than the cluster can handle at a time (default = 180).
`cleanup_resources_timeout`	The total amount of time for Kubernetes resources to be cleaned up after the job completes. Supported syntax: `1h30m` , `300s` , `10m` . Default is 5 minutes ( `5m` ).
`priority_class_name`	Specify the Priority Class to be set to the pod. The default one is used if not set.
`privileged`	Run containers with the privileged flag.
`pull_policy`	Specify the image pull policy: `never` , `if-not-present` , `always` . If not set, the cluster’s image default pull policy is used. For more information and instructions on how to set multiple pull policies, see using pull policies . See also `if-not-present` , `never` security considerations . You can also restrict pull policies .
`resource_availability_check_max_attempts`	The maximum number of attempts to check if a resource (service account and/or pull secret) set is available before giving up. There is 5 seconds interval between each attempt. Introduced in GitLab 15.0. Read more about resources check during prepare step .
`runtime_class_name`	A Runtime class to use for all created pods. If the feature is unsupported by the cluster, jobs exit or fail.
`service_container_security_context`	Sets a container security context for the service containers. Read more about security context .
`scheduler_name`	Scheduler to use for scheduling build pods.
`service_account`	Default service account job/executor pods use to talk to Kubernetes API.
`service_account_overwrite_allowed`	Regular expression to validate the contents of the service account overwrite environment variable. When empty, it disables the service account overwrite feature.
`services`	List of services attached to the build container using the sidecar pattern . Read more about using services .
`use_service_account_image_pull_secrets`	When enabled, the pod created by the executor lacks `imagePullSecrets` . This causes the pod to be created using the `imagePullSecrets` from the service account , if set.
`terminationGracePeriodSeconds`	Duration after the processes running in the pod are sent a termination signal and the time when the processes are forcibly halted with a kill signal. Deprecated in favour of `cleanup_grace_period_seconds` and `pod_termination_grace_period_seconds` .
`volumes`	Configured through the configuration file, the list of volumes that is mounted in the build container. Read more about using volumes .
`pod_spec`	This setting is an experiment. Overwrites the pod specification generated by the runner manager with a list of configurations set on the pod used to run the CI Job. All the properties listed `Kubernetes Pod Specification` can be set. For more information, see Overwrite generated pod specifications (experiment) .
`retry_limit`	The maximum number of attempts to communicate with Kubernetes API. The retry interval between each attempt is based on a backoff algorithm starting at 500 ms.
`retry_backoff_max`	Custom maximum backoff value in milliseconds for the retry interval to reach for each attempt. The default value is 2000 ms and it can not be lower than 500 ms. The default maximum retry interval to reach for each attempt is 2 seconds and can be customized with `retry_backoff_max` .
`retry_limits`	How many times each request error is to be retried.
`logs_base_dir`	Base directory to be prepended to the generated path to store build logs. For more information, see Change the base directory for build logs and scripts .
`scripts_base_dir`	Base directory to be prepended to the generated path to store build scripts. For more information, see Change the base directory for build logs and scripts .
`print_pod_warning_events`	When enabled, this feature retrieves all warning events associated with the pod when jobs fail. This functionality is enabled by default and requires a service account with at least `events: list` permissions .

Key	Description
`project.runner.gitlab.com/id`	The ID of the project, unique across projects in the GitLab instance.
`project.runner.gitlab.com/name`	The name of the project.
`project.runner.gitlab.com/namespace-id`	The ID of the project’s namespace.
`project.runner.gitlab.com/namespace`	The name of the project’s namespace.
`project.runner.gitlab.com/root-namespace`	The ID of the project’s root namespace. For example, `/gitlab-org/group-a/subgroup-a/project` , where the root namespace is `gitlab-org`
`manager.runner.gitlab.com/name`	The name of the runner configuration that launched this job.
`manager.runner.gitlab.com/id-short`	The ID of the runner configuration that launched the job.
`job.runner.gitlab.com/pod`	Internal label used by the Kubernetes executor.

Key	Description
`job.runner.gitlab.com/id`	The ID of the job, unique across all jobs in the GitLab instance.
`job.runner.gitlab.com/url`	The URL for the job details.
`job.runner.gitlab.com/sha`	The commit revision the project is built for.
`job.runner.gitlab.com/before_sha`	The previous latest commit present on a branch or tag.
`job.runner.gitlab.com/ref`	The branch or tag name for which the project is built.
`job.runner.gitlab.com/name`	The name of the job.
`job.runner.gitlab.com/timeout`	The job execution timeout in the time duration format. For example, `2h3m0.5s` .
`project.runner.gitlab.com/id`	The project ID of the job.

Setting	Description
`name`	Name given to the custom `pod_spec` .
`patch_path`	Path to the file that defines the changes to apply to the final `PodSpec` object before it is generated. The file must be a JSON or YAML file.
`patch`	A JSON or YAML format string that describes the changes which must be applied to the final `PodSpec` object before it is generated.
`patch_type`	The strategy the runner uses to apply the specified changes to the `PodSpec` object generated by GitLab Runner. The accepted values are `merge` , `json` , and `strategic` .

Option	Type	Required	Description
`fs_group`	`int`	No	A special supplemental group that applies to all containers in a pod.
`run_as_group`	`int`	No	The GID to run the entry point of the container process.
`run_as_non_root`	boolean	No	Indicates that the container must run as a non-root user.
`run_as_user`	`int`	No	The UID to run the entry point of the container process.
`supplemental_groups`	`int` list	No	A list of groups applied to the first process run in each container, in addition to the container’s primary GID.
`selinux_type`	`string`	No	The SELinux type label that applies to all containers in a pod.

Option	Type	Required	Description
`run_as_group`	int	No	The GID to run the entry point of the container process.
`run_as_non_root`	boolean	No	Indicates that the container must run as a non-root user.
`run_as_user`	int	No	The UID to run the entry point of the container process.
`capabilities.add`	string list	No	The capabilities to add when running the container.
`capabilities.drop`	string list	No	The capabilities to drop when running the container.
`selinux_type`	string	No	The SELinux type label that is associated with the container process.

Runner pull policy	Kubernetes pull policy	Description
blank	blank	Uses the default policy, as specified by Kubernetes.
`if-not-present`	`IfNotPresent`	The image is pulled only if it is not already present on the node that executes the job. Review the security considerations before you use this pull policy.
`always`	`Always`	The image is pulled every time the job is executed.
`never`	`Never`	The image is never pulled and requires the node to already have it.

Define a list of node affinities

Define a list of node affinities to add to a pod specification at build time.

node_affinities does not determine which operating system a build should run with, only node_selectors . For more information, see Operating system, architecture, and Windows kernel version . Example configuration in the config.toml :

     concurrent = 1
[[runners]]
  name = "myRunner"
  url = "gitlab.example.com"
  executor = "kubernetes"
  [runners.kubernetes]
    [runners.kubernetes.affinity]
      [runners.kubernetes.affinity.node_affinity]
        [[runners.kubernetes.affinity.node_affinity.preferred_during_scheduling_ignored_during_execution]]
          weight = 100
          [runners.kubernetes.affinity.node_affinity.preferred_during_scheduling_ignored_during_execution.preference]
            [[runners.kubernetes.affinity.node_affinity.preferred_during_scheduling_ignored_during_execution.preference.match_expressions]]
              key = "cpu_speed"
              operator = "In"
              values = ["fast"]
            [[runners.kubernetes.affinity.node_affinity.preferred_during_scheduling_ignored_during_execution.preference.match_expressions]]
              key = "mem_speed"
              operator = "In"
              values = ["fast"]
        [[runners.kubernetes.affinity.node_affinity.preferred_during_scheduling_ignored_during_execution]]
          weight = 50
          [runners.kubernetes.affinity.node_affinity.preferred_during_scheduling_ignored_during_execution.preference]
            [[runners.kubernetes.affinity.node_affinity.preferred_during_scheduling_ignored_during_execution.preference.match_expressions]]
              key = "core_count"
              operator = "In"
              values = ["high", "32"]
            [[runners.kubernetes.affinity.node_affinity.preferred_during_scheduling_ignored_during_execution.preference.match_fields]]
              key = "cpu_type"
              operator = "In"
              values = ["arm64"]
      [runners.kubernetes.affinity.node_affinity.required_during_scheduling_ignored_during_execution]
        [[runners.kubernetes.affinity.node_affinity.required_during_scheduling_ignored_during_execution.node_selector_terms]]
          [[runners.kubernetes.affinity.node_affinity.required_during_scheduling_ignored_during_execution.node_selector_terms.match_expressions]]
            key = "kubernetes.io/e2e-az-name"
            operator = "In"
            values = [
              "e2e-az1",
              "e2e-az2"
            ]
    

Define nodes where pods are scheduled

Use pod affinity and anti-affinity to constrain the nodes your pod is eligible to be scheduled on, based on labels on other pods.

Example configuration in the config.toml :

      concurrent = 1




    

[[runners]]
  name = "myRunner"
  url = "gitlab.example.com"
  executor = "kubernetes"
  [runners.kubernetes]
    [runners.kubernetes.affinity]
      [runners.kubernetes.affinity.pod_affinity]
        [[runners.kubernetes.affinity.pod_affinity.required_during_scheduling_ignored_during_execution]]
          topology_key = "failure-domain.beta.kubernetes.io/zone"
          namespaces = ["namespace_1", "namespace_2"]
          [runners.kubernetes.affinity.pod_affinity.required_during_scheduling_ignored_during_execution.label_selector]
            [[runners.kubernetes.affinity.pod_affinity.required_during_scheduling_ignored_during_execution.label_selector.match_expressions]]
              key = "security"
              operator = "In"
              values = ["S1"]
        [[runners.kubernetes.affinity.pod_affinity.preferred_during_scheduling_ignored_during_execution]]
        weight = 100
        [runners.kubernetes.affinity.pod_affinity.preferred_during_scheduling_ignored_during_execution.pod_affinity_term]
          topology_key = "failure-domain.beta.kubernetes.io/zone"
          [runners.kubernetes.affinity.pod_affinity.preferred_during_scheduling_ignored_during_execution.pod_affinity_term.label_selector]
            [[runners.kubernetes.affinity.pod_affinity.preferred_during_scheduling_ignored_during_execution.pod_affinity_term.label_selector.match_expressions]]
              key = "security_2"
              operator = "In"
              values = ["S2"]
      [runners.kubernetes.affinity.pod_anti_affinity]
        [[runners.kubernetes.affinity.pod_anti_affinity.required_during_scheduling_ignored_during_execution]]
          topology_key = "failure-domain.beta.kubernetes.io/zone"
          namespaces = ["namespace_1", "namespace_2"]
          [runners.kubernetes.affinity.pod_anti_affinity.required_during_scheduling_ignored_during_execution.label_selector]
            [[runners.kubernetes.affinity.pod_anti_affinity.required_during_scheduling_ignored_during_execution.label_selector.match_expressions]]
              key = "security"
              operator = "In"
              values = ["S1"]
          [runners.kubernetes.affinity.pod_anti_affinity.required_during_scheduling_ignored_during_execution.namespace_selector]
            [[runners.kubernetes.affinity.pod_anti_affinity.required_during_scheduling_ignored_during_execution.namespace_selector.match_expressions]]
              key = "security"
              operator = "In"
              values = ["S1"]
        [[runners.kubernetes.affinity.pod_anti_affinity.preferred_during_scheduling_ignored_during_execution]]
        weight = 100
        [runners.kubernetes.affinity.pod_anti_affinity.preferred_during_scheduling_ignored_during_execution.pod_affinity_term]
          topology_key = "failure-domain.beta.kubernetes.io/zone"
          [runners.kubernetes.affinity.pod_anti_affinity.preferred_during_scheduling_ignored_during_execution.pod_affinity_term.label_selector]
            [[runners.kubernetes.affinity.pod_anti_affinity.preferred_during_scheduling_ignored_during_execution.pod_affinity_term.label_selector.match_expressions]]
              key = "security_2"
              operator = "In"
              values = ["S2"]
          [runners.kubernetes.affinity.pod_anti_affinity.preferred_during_scheduling_ignored_during_execution.pod_affinity_term.namespace_selector]
            [[runners.kubernetes.affinity.pod_anti_affinity.preferred_during_scheduling_ignored_during_execution.pod_affinity_term.namespace_selector.match_expressions]]
              key = "security_2"
              operator = "In"
              values = ["S2"]
     

Networking

Configure a container lifecycle hook

Use container lifecycle hooks to run code configured for a handler when the corresponding lifecycle hook is executed.

You can configure two types of hooks: PreStop and PostStart . Each of them allows only one type of handler to be set.

Example configuration in the config.toml file:

       [[runners]]
  name = "kubernetes"
  url = "https://gitlab.example.com/"
  executor = "kubernetes"
  token = "yrnZW46BrtBFqM7xDzE7dddd"
  [runners.kubernetes]
    image = "alpine:3.11"
    privileged = true
    namespace = "default"
    [runners.kubernetes.container_lifecycle.post_start.exec]
      command = ["touch", "/builds/postStart.txt"]
    [runners.kubernetes.container_lifecycle.pre_stop.http_get]
      port = 8080
      host = "localhost"
      path = "/test"
      [[runners.kubernetes.container_lifecycle.pre_stop.http_get.http_headers]]
        name = "header_name_1"
        value = "header_value_1"
      [[runners.kubernetes.container_lifecycle.pre_stop.http_get.http_headers]]
        name = "header_name_2"
        value = "header_value_2"
      

Use the following settings to configure each lifecycle hook:

Option	Type	Required	Description
`exec`	`KubernetesLifecycleExecAction`	No	`Exec` specifies the action to take.
`http_get`	`KubernetesLifecycleHTTPGet`	No	`HTTPGet` specifies the http request to perform.
`tcp_socket`	`KubernetesLifecycleTcpSocket`	No	`TCPsocket` specifies an action involving a TCP port.

KubernetesLifecycleExecAction

Option	Type	Required	Description
`command`	`string` list	Yes	The command line to execute inside the container.

KubernetesLifecycleHTTPGet

Option	Type	Required	Description
`port`	`int`	Yes	The number of the port to access on the container.
`host`	string	No	The host name to connect to, defaults to the pod IP (optional).
`path`	string	No	The path to access on the HTTP server (optional).
`scheme`	string	No	The scheme used for connecting to the host. Defaults to HTTP (optional).
`http_headers`	`KubernetesLifecycleHTTPGetHeader` list	No	Custom headers to set in the request (optional).

KubernetesLifecycleHTTPGetHeader

Option	Type	Required	Description
`name`	string	Yes	HTTP header name.
`value`	string	Yes	HTTP header value.

KubernetesLifecycleTcpSocket

Option	Type	Required	Description
`port`	`int`	Yes	The number of the port to access on the container.
`host`	string	No	The host name to connect to, defaults to the pod IP (optional).

Configure pod DNS settings

Use the following options to configure the DNS settings of the pods.

Option	Type	Required	Description
`nameservers`	`string` list	No	A list of IP addresses that are used as DNS servers for the pod.
`options`	`KubernetesDNSConfigOption`	No	A optional list of objects where each object may have a name property (required) and a value property (optional).
`searches`	`string` lists	No	A list of DNS search domains for hostname lookup in the pod.

Example configuration in the config.toml file:

        concurrent = 1
check_interval = 30




    

[[runners]]
  name = "myRunner"
  url = "https://gitlab.example.com"
  token = "__REDACTED__"
  executor = "kubernetes"
  [runners.kubernetes]
    image = "alpine:latest"
    [runners.kubernetes.dns_config]
      nameservers = [
        "1.2.3.4",
      searches = [
        "ns1.svc.cluster-domain.example",
        "my.dns.search.suffix",
      [[runners.kubernetes.dns_config.options]]
        name = "ndots"
        value = "2"
      [[runners.kubernetes.dns_config.options]]
        name = "edns0"
       

KubernetesDNSConfigOption

Option	Type	Required	Description
`name`	string	Yes	Configuration option name.
`value`	`*string`	No	Configuration option value.

Default list of dropped capabilities

GitLab Runner drops the following capabilities by default.

User-defined cap_add has priority over the default list of dropped capabilities. If you want to add the capability that is dropped by default, add it to cap_add .

NET_RAW

Add extra host aliases

This feature is available in Kubernetes 1.7 and higher.

Configure a host aliases to instruct Kubernetes to add entries to /etc/hosts file in the container.

Use the following options:

Option	Type	Required	Description
`IP`	string	Yes	The IP address you want to attach hosts to.
`Hostnames`	`string` list	Yes	A list of host name aliases that are attached to the IP.

Example configuration in the config.toml file:

         concurrent = 4
[[runners]]
  # usual configuration
  executor = "kubernetes"
  [runners.kubernetes]
    [[runners.kubernetes.host_aliases]]
      ip = "127.0.0.1"
      hostnames = ["web1", "web2"]
    [[runners.kubernetes.host_aliases]]
      ip = "192.168.1.1"
      hostnames = ["web14", "web15"]
        

You can also configure host aliases by using the command-line parameter --kubernetes-host_aliases with JSON input. For example:

          gitlab-runner register --kubernetes-host_aliases '[{"ip":"192.168.1.100","hostnames":["myservice.local"]},{"ip":"192.168.1.101","hostnames":["otherservice.local"]}]'
         

Volumes

Using the cache with the Kubernetes executor

When the cache is used with the Kubernetes executor, a volume called /cache is mounted on the pod. During job execution, if cached data is needed, the runner checks if cached data is available. Cached data is available if a compressed file is available on the cache volume.

To set the cache volume, use the cache_dir setting in the config.toml file.

If available, the compressed file is extracted into the build folder and can then be used in the job.
If not available, the cached data is downloaded from the configured storage and saved into the cache dir as a compressed file. The compressed file is then extracted into the build folder.

Configure volume types

You can mount the following volume types:

hostPath
persistentVolumeClaim
configMap
secret
emptyDir
csi

Example of a configuration with multiple volume types:

           concurrent = 4
[[runners]]
  # usual configuration
  executor = "kubernetes"
  [runners.kubernetes]
    [[runners.kubernetes.volumes.host_path]]
      name = "hostpath-1"
      mount_path = "/path/to/mount/point"
      read_only = true
      host_path = "/path/on/host"
    [[runners.kubernetes.volumes.host_path]]
      name = "hostpath-2"
      mount_path = "/path/to/mount/point_2"
      read_only = true
    [[runners.kubernetes.volumes.pvc]]
      name = "pvc-1"
      mount_path = "/path/to/mount/point1"
    [[runners.kubernetes.volumes.config_map]]
      name = "config-map-1"
      mount_path = "/path/to/directory"
      [runners.kubernetes.volumes.config_map.items]
        "key_1" = "relative/path/to/key_1_file"
        "key_2" = "key_2"
    [[runners.kubernetes.volumes.secret]]
      name = "secrets"
      mount_path = "/path/to/directory1"
      read_only = true
      [runners.kubernetes.volumes.secret.items]
        "secret_1" = "relative/path/to/secret_1_file"
    [[runners.kubernetes.volumes.empty_dir]]
      name = "empty-dir"
      mount_path = "/path/to/empty_dir"
      medium = "Memory"
    [[runners.kubernetes.volumes.csi]]
      name = "csi-volume"
      mount_path = "/path/to/csi/volume"
      driver = "my-csi-driver"
      [runners.kubernetes.volumes.csi.volume_attributes]
        size = "2Gi"
          

`hostPath` volume

Configure the hostPath volume to instruct Kubernetes to mount a specified host path in the container.

Use the following options in the config.toml file:

Option	Type	Required	Description
`name`	string	Yes	The name of the volume.
`mount_path`	string	Yes	The path where the volume is mounted in the container.
`sub_path`	string	No	The sub-path inside the mounted volume instead of its root.
`host_path`	string	No	The path on the host mounted as a volume. If you don’t specify a value, it defaults to the same path as `mount_path` .
`read_only`	boolean	No	Sets the volume in read-only mode. Defaults to `false` .
`mount_propagation`	string	No	Share mounted volumes between containers. For more information, see Mount Propagation .

`persistentVolumeClaim` volume

Configure the persistentVolumeClaim volume to instruct Kubernetes to use a persistentVolumeClaim defined in a Kubernetes cluster and mount it in the container.

Use the following options in the config.toml file:

Option	Type	Required	Description
`name`	string	Yes	The name of the volume and at the same time the name of `PersistentVolumeClaim` that should be used. Supports variables. For more information, see Persistent per-concurrency build volumes .
`mount_path`	string	Yes	Path in the container where the volume is mounted.
`read_only`	boolean	No	Sets the volume to read-only mode (defaults to false).
`sub_path`	string	No	Mount a sub-path in the volume instead of the root.
`mount_propagation`	string	No	Set the mount propagation mode for the volume. For more details, see Kubernetes mount propagation .

`configMap` volume

Configure the configMap volume to instruct Kubernetes to use a configMap defined in a Kubernetes cluster and mount it in the container.

Use the following options in the config.toml :

Option	Type	Required	Description
`name`	string	Yes	The name of the volume and at the same time the name of `configMap` that should be used.
`mount_path`	string	Yes	Path in the container where the volume is mounted.
`read_only`	boolean	No	Sets the volume to read-only mode (defaults to false).
`sub_path`	string	No	Mount a sub-path in the volume instead of the root.
`items`	`map[string]string`	no	Key-to-path mapping for keys from the `configMap` that should be used.

Each key from the configMap is changed into a file and stored in the mount path. By default:

All keys are included.
The configMap key is used as the filename.
The value is stored in the file contents.

To change the default key and value storage, use the items option. If you use the items option, only specified keys are added to the volumes and all other keys are skipped.

If you use a key that doesn’t exist, the job fails on the pod creation stage.

`secret` volume

Configure a secret volume to instruct Kubernetes to use a secret defined in a Kubernetes cluster and mount it in the container.

Use the following options in the config.toml file:

Option	Type	Required	Description
`name`	string	Yes	The name of the volume and at the same time the name of secret that should be used.
`mount_path`	string	Yes	Path inside of container where the volume should be mounted.
`read_only`	boolean	No	Sets the volume in read-only mode (defaults to false).
`sub_path`	string	No	Mount a sub-path in the volume instead of the root.
`items`	`map[string]string`	No	Key-to-path mapping for keys from the configMap that should be used.

Each key from selected secret is changed into a file stored in the selected mount path. By default:

All keys are included.
The configMap key is used as the filename.
The value is stored in the file contents.

To change default key and value storage, use the items option. If you use the items option, only specified keys are added to the volumes and all other keys are skipped.

If you use a key that doesn’t exist, the job fails on the pod creation stage.

`emptyDir` volume

Configure an emptyDir volume to instruct Kubernetes to mount an empty directory in the container.

Use the following options in the config.toml file:

Option	Type	Required	Description
`name`	string	Yes	The name of the volume.
`mount_path`	string	Yes	Path inside of container where the volume should be mounted.
`sub_path`	string	No	Mount a sub-path in the volume instead of the root.
`medium`	string	No	“Memory” provides a `tmpfs` , otherwise it defaults to the node disk storage (defaults to “”).
`size_limit`	string	No	The total amount of local storage required for the `emptyDir` volume.

`csi` volume

Configure a Container Storage Interface ( csi ) volume to instruct Kubernetes to use a custom csi driver to mount an arbitrary storage system in the container.

Use the following options in the config.toml :

Option	Type	Required	Description
`name`	string	Yes	The name of the volume.
`mount_path`	string	Yes	Path inside of container where the volume should be mounted.
`driver`	string	Yes	A string value that specifies the name of the volume driver to use.
`fs_type`	string	No	A string value that specifies the name of the file system type (for example, `ext4` , `xfs` , `ntfs` ).
`volume_attributes`	`map[string]string`	No	Key-value pair mapping for attributes of the `csi` volume.
`sub_path`	string	No	Mount a sub-path in the volume instead of the root.
`read_only`	boolean	No	Sets the volume in read-only mode (defaults to false).

Mount volumes on service containers

Volumes defined for the build container are also automatically mounted for all services containers. You can use this functionality as an alternative to services_tmpfs (available only to Docker executor), to mount database storage in RAM to speed up tests.

Example configuration in the config.toml file:

            [[runners]]
  # usual configuration
  executor = "kubernetes"
  [runners.kubernetes]
    [[runners.kubernetes.volumes.empty_dir]]
      name = "mysql-tmpfs"
      mount_path = "/var/lib/mysql"
      medium = "Memory"

Custom volume mount

To store the builds directory for the job, define custom volume mounts to the configured builds_dir ( /builds by default). If you use pvc volumes , based on the access mode , you might be limited to running jobs on one node.

Example configuration in the config.toml file:

             concurrent = 4
[[runners]]
  # usual configuration
  executor = "kubernetes"
  builds_dir = "/builds"
  [runners.kubernetes]
    [[runners.kubernetes.volumes.empty_dir]]
      name = "repo"
      mount_path = "/builds"
      medium = "Memory"
            

Persistent per-concurrency build volumes

History

The build directories in Kubernetes CI jobs are ephemeral by default. If you want to persist your Git clone across jobs (to make GIT_STRATEGY=fetch work), you must mount a persistent volume claim for your build folder. Because multiple jobs can run concurrently, you must either use a ReadWriteMany volume, or have one volume for each potential concurrent job on the same runner. The latter is likely to be more performant. Here is an example of such a configuration:

              concurrent = 4
[[runners]]
  executor = "kubernetes"
  builds_dir = "/mnt/builds"
  [runners.kubernetes]
    [[runners.kubernetes.volumes.pvc]]
      # CI_CONCURRENT_ID identifies parallel jobs of the same runner.
      name = "build-pvc-$CI_CONCURRENT_ID"
      mount_path = "/mnt/builds"
             

In this example, create the persistent volume claims named build-pvc-0 to build-pvc-3 yourself. Create as many as the runner’s concurrent setting dictates.

Use a helper image

After you set the security policy, the helper image must conform to the policy. The image does not receive privileges from the root group, so you must ensure that the user ID is part of the root group.

If you only need the nonroot environment, you can use the GitLab Runner UBI OpenShift Container Platform images instead of a helper image. You can also use the GitLab Runner Helper UBI OpenShift Container Platform images.

The following example creates a user and group called nonroot and sets the helper image to run as that user.

               ARG tag
FROM registry.gitlab.com/gitlab-org/ci-cd/gitlab-runner-ubi-images/gitlab-runner-helper-ocp:${tag}
USER root
RUN groupadd -g 59417 nonroot && \
    useradd -u 59417 nonroot -g nonroot
WORKDIR /home/nonroot
USER 59417:59417
              

Using Docker in builds

When you use Docker in your builds, there are several considerations you should be aware of.

Exposed `/var/run/docker.sock`

There is risk involved if you use the runners.kubernetes.volumes.host_path option to expose /var/run/docker.sock of your host into your build container. Be careful when you run builds in the same cluster as your production containers. The node’s containers are accessible from the build container.

Using `docker:dind`

If you run the docker:dind , also called the docker-in-docker image, containers must run in privileged mode. This may have potential risks and cause additional issues.

The Docker daemon runs as a separate container in the pod because it is started as a service , typically in the .gitlab-ci.yml . Containers in pods only share volumes assigned to them and an IP address, that they use to communicate with each other with localhost . The docker:dind container does not share /var/run/docker.sock and the docker binary tries to use it by default.

To configure the client use TCP to contact the Docker daemon, in the other container, include the environment variables of the build container:

DOCKER_HOST=tcp://docker:2375 for no TLS connection.
DOCKER_HOST=tcp://docker:2376 for TLS connection.

In Docker 19.03 and later, TLS is enabled by default but you must map certificates to your client. You can enable non-TLS connection for Docker-in-Docker or mount certificates. For more information, see Use Docker In Docker Workflow with Docker executor .

Prevent host kernel exposure

If you use docker:dind or /var/run/docker.sock , the Docker daemon has access to the underlying kernel of the host machine. This means that any limits set in the pod do not work when Docker images are built. The Docker daemon reports the full capacity of the node, regardless of limits imposed on the Docker build containers spawned by Kubernetes.

If you run build containers in privileged mode, or if /var/run/docker.sock is exposed, the host kernel may become exposed to build containers. To minimize exposure, specify a label in the node_selector option. This ensures that the node matches the labels before any containers can be deployed to the node. For example, if you specify the label role=ci , the build containers only run on nodes labeled role=ci , and all other production services run on other nodes.

To further separate build containers, you can use node taints . Taints prevent other pods from scheduling on the same nodes as the build pods, without extra configuration for the other pods.

Restrict Docker images and services

You can restrict the Docker images that are used to run your jobs. To do this, you specify wildcard patterns. For example, to allow images from your private Docker registry only:

                [[runners]]
  (...)
  executor = "kubernetes"
  [runners.kubernetes]
    (...)
    allowed_images = ["my.registry.tld:5000/*:*"]
    allowed_services = ["my.registry.tld:5000/*:*"]
               

Or, to restrict to a specific list of images from this registry:

                 [[runners]]
  (...)
  executor = "kubernetes"
  [runners.kubernetes]
    (...)
    allowed_images = ["my.registry.tld:5000/ruby:*", "my.registry.tld:5000/node:*"]
    allowed_services = ["postgres:9.4", "postgres:latest"]
                

Restrict Docker pull policies

History

In the .gitlab-ci.yml file, you can specify a pull policy. This policy determines how a CI/CD job should fetch images.

To restrict which pull policies can be used from those specified in the .gitlab-ci.yml file, use allowed_pull_policies .

For example, to allow only the always and if-not-present pull policies:

                  [[runners]]

  (...)
  executor = "kubernetes"
  [runners.kubernetes]
    (...)
    allowed_pull_policies = ["always", "if-not-present"]

If you don’t specify allowed_pull_policies , the default is the value in the pull_policy keyword.
If you don’t specify pull_policy , the cluster’s image default pull policy is used.
The job uses only the pull policies that are listed in both pull_policy and allowed_pull_policies . The effective pull policy is determined by comparing the policies in pull_policy keyword and allowed_pull_policies . GitLab uses the intersection of these two policy lists. For example, if pull_policy is ["always", "if-not-present"] and allowed_pull_policies is ["if-not-present"] , then the job uses only if-not-present because it’s the only pull policy defined in both lists.
The existing pull_policy keyword must include at least one pull policy specified in allowed_pull_policies . The job fails if none of the pull_policy values match allowed_pull_policies .

Job execution

GitLab Runner uses kube attach instead of kube exec by default. This should avoid problems like when a job is marked successful midway in environments with an unstable network.

Follow issue #27976 for progress on legacy execution strategy removal.

Configure the number of request attempts to the Kubernetes API

By default, the Kubernetes executor retries specific requests to the Kubernetes API after five failed attempts. The delay is controlled by a backoff algorithm with a 500 millisecond floor and a customizable ceiling with default value of two seconds. To configure the number of retries, use the retry_limit option in the config.toml file. Similarly, for backoff ceiling, use the retry_backoff_max option. The following failures are automatically retried:

error dialing backend
TLS handshake timeout
read: connection timed out
connect: connection timed out
Timeout occurred
http2: client connection lost
connection refused
tls: internal error
io.unexpected EOF
syscall.ECONNRESET
syscall.ECONNREFUSED
syscall.ECONNABORTED
syscall.EPIPE

To control the amount of retries for each error, use the retry_limits option. The rety_limits specifies the amount of retries for each error separately, and is a map of error messages to the amount of retries. The error message can be a substring of the error message returned by the Kubernetes API. The retry_limits option has precedence over the retry_limit option.

For example, configure the retry_limits option to retry the TLS related errors in your environment 10 times instead of the default five times:

                   [[runners]]
  name = "myRunner"
  url = "https://gitlab.example.com/"
  executor = "kubernetes"
  [runners.kubernetes]
    retry_limit = 5
    [runners.kubernetes.retry_limits]
        "TLS handshake timeout" = 10
        "tls: internal error" = 10
                  

To retry an entirely different error, such as exceeded quota 20 times:

                    [[runners]]
  name = "myRunner"
  url = "https://gitlab.example.com/"
  executor = "kubernetes"
  [runners.kubernetes]
    retry_limit = 5
    [runners.kubernetes.retry_limits]
        "exceeded quota" = 20
                   

Container entrypoint known issues

History

In 15.0, GitLab Runner uses the entrypoint defined in a Docker image when used with the Kubernetes executor with kube attach . In GitLab 15.1 and later, the entrypoint defined in a Docker image is used with the Kubernetes executor when FF_KUBERNETES_HONOR_ENTRYPOINT is set.

The container entry point has the following known issues:

If an entrypoint is defined in the Dockerfile for an image, it must open a valid shell. Otherwise, the job hangs.
- To open a shell, the system passes the command as for the build container.
File type CI/CD variables are not written to disk when the entrypoint is executed. The file is only accessible in the job during script execution.
The following CI/CD variables are not accessible in the entrypoint. You can use before_script to make any setup changes before running script commands:
- CI/CD variables defined in the settings .
- Masked CI/CD variables .

Before GitLab Runner 17.4:

The entrypoint logs were not forwarded to the build’s log.
With the Kubernetes executor with kube exec , GitLab Runner did not wait for the entrypoint to open a shell (see above ).

Starting with GitLab Runner 17.4, the entrypoint logs are now forwarded. The system waits for the entrypoint to run and spawn the shell. This has the following implications:

If FF_KUBERNETES_HONOR_ENTRYPOINT is set, and the image’s entrypoint takes longer than poll_timeout (default: 180 s), the build fails. The poll_timeout value (and potentially poll_interval ) must be adapted if the entrypoint is expected to run longer.
When FF_KUBERNETES_HONOR_ENTRYPOINT and FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY are set, the system adds a startup probe to the build container, so that it knows when the entrypoint is spawning the shell. If a custom entrypoint uses the provided args to spawn the expected shell, then the startup probe is resolved automatically. However, if the container image is spawning the shell without using the command passed in through args , the entrypoint must resolve the startup probe itself by creating a file named .gitlab-startup-marker inside the root of the build directory. The startup probe checks every poll_interval for the .gitlab-startup-marker file. If the file is not present in poll_timeout , the pod is considered unhealthy, and the system abort the build.

Restrict access to job variables

When using Kubernetes executor, users with access to the Kubernetes cluster can read variables used in the job. By default, job variables are stored in:

Pod’s environment section

To restrict access to job variable data, you should use role-based access control (RBAC). When you use RBAC, only GitLab administrators have access to the namespace used by the GitLab Runner.

If you need other users to access the GitLab Runner namespace, set the following verbs to restrict the user access in the GitLab Runner namespace:

For pods and configmaps :
- get
- watch
- list
For pods/exec and pods/attach , use create .

Example RBAC definition for authorized users:

                     kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: gitlab-runner-authorized-users
rules:
- apiGroups: [""]
  resources: ["configmaps", "pods"]
  verbs: ["get", "watch", "list"]
- apiGroups: [""]
  resources: ["pods/exec", "pods/attach"]
  verbs: ["create"]
                    

Resources check during prepare step

History

Prerequisites:

image_pull_secrets or service_account is set.
resource_availability_check_max_attempts is set to a number greater than zero.
Kubernetes serviceAccount used with the get and list permissions.

GitLab Runner checks if the new service accounts or secrets are available with a 5-second interval between each try.

In GitLab 15.0 and 15.1, you cannot disable this feature and it defaults to 5 when a negative value is set.
In GitLab 15.0.1, 15.1.1, 15.2 and later, this feature is disabled by default. To enable this feature, set resource_availability_check_max_attempts to any value other than 0 . The value you set defines the amount of times the runner checks for service accounts or secrets.

Overwrite the Kubernetes namespace

Prerequisites:

In the values.yml file for GitLab Runner Helm charts, rbac.clusterWideAccess is set to true .
The runner has permissions configured in the core API group.

You can overwrite Kubernetes namespaces to designate a namespace for CI purposes, and deploy a custom

How the runner creates Kubernetes pods

Connect to the Kubernetes API

Set the bearer token for Kubernetes API calls

Configure runner API permissions

Informers

Configuration settings

CPU requests and limits

Memory requests and limits

Helper container memory sizing recommendations

Storage requests and limits

Other config.toml settings

Configuration example

Configure the executor service account

Pods and containers

Default labels for job pods

Default annotations for job pods

Pod lifecycle

Overwrite pod tolerations

Overwrite pod labels

Overwrite pod annotations

Overwrite generated pod specifications

Merge patch strategy

JSON patch strategy

Strategic patch strategy

Best practices

Create a PVC for each build job by modifying the Pod Spec

Set a security policy for the pod

Remove old runner pods

Set a security policy for the container

Set a pull policy

Specify container capabilities

Overwrite container resources

Define a list of services

Overwrite service containers resources

Overwrite the Kubernetes default service account

Set the RuntimeClass

Change the base directory for build logs and scripts

User namespaces

Operating system, architecture, and Windows kernel version

Nodes

Specify the node to execute builds

Example for linux/arm64

Example for windows/amd64

Overwrite the node selector

Define a list of node affinities

Define nodes where pods are scheduled

Networking

Configure a container lifecycle hook

KubernetesLifecycleExecAction

KubernetesLifecycleHTTPGet

KubernetesLifecycleHTTPGetHeader

KubernetesLifecycleTcpSocket

Configure pod DNS settings

KubernetesDNSConfigOption

Default list of dropped capabilities

Add extra host aliases

Volumes

Using the cache with the Kubernetes executor

Configure volume types

hostPath volume

persistentVolumeClaim volume

configMap volume

secret volume

emptyDir volume

csi volume

Mount volumes on service containers

Custom volume mount

Persistent per-concurrency build volumes

Use a helper image

Using Docker in builds

Exposed /var/run/docker.sock

Using docker:dind

Prevent host kernel exposure

Restrict Docker images and services

Restrict Docker pull policies

Job execution

Configure the number of request attempts to the Kubernetes API

Container entrypoint known issues

Restrict access to job variables

Other `config.toml` settings

Create a `PVC` for each build job by modifying the Pod Spec

Example for `linux/arm64`

Example for `windows/amd64`

`hostPath` volume

`persistentVolumeClaim` volume

`configMap` volume

`secret` volume

`emptyDir` volume

`csi` volume

Exposed `/var/run/docker.sock`

Using `docker:dind`