gitlab on k8s

GitLab部署

helm 安装 postgresql

helmchar: github

安装日志

[root@master2 ~]#helm install gitlib-db -n gitlab /opt/helm/postgresql/
NAME: gitlib-db
LAST DEPLOYED: Mon Apr 24 09:05:58 2023
NAMESPACE: gitlab
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: postgresql
CHART VERSION: 12.1.9
APP VERSION: 15.1.0
** Please be patient while the chart is being deployed **
PostgreSQL can be accessed via port 5432 on the following DNS names from within your cluster:
    gitlib-db-postgresql.gitlab.svc.cluster.local - Read/Write connection
To get the password for "postgres" run:
    export POSTGRES_PASSWORD=$(kubectl get secret --namespace gitlab gitlib-db-postgresql -o jsonpath="{.data.postgres-password}" | base64 -d)
To connect to your database run the following command:
    kubectl run gitlib-db-postgresql-client --rm --tty -i --restart='Never' --namespace gitlab --image 10.50.10.185/postgresql/bitnami/postgresql:15.1.0-debian-11-r20 --env="PGPASSWORD=$POSTGRES_PASSWORD" \
      --command -- psql --host gitlib-db-postgresql -U postgres -d postgres -p 5432
    > NOTE: If you access the container using bash, make sure that you execute "/opt/bitnami/scripts/postgresql/entrypoint.sh /bin/bash" in order to avoid the error "psql: local user with ID 1001} does not exist"
To connect to your database from outside the cluster execute the following commands:
    kubectl port-forward --namespace gitlab svc/gitlib-db-postgresql 5432:5432 &
    PGPASSWORD="$POSTGRES_PASSWORD" psql --host 127.0.0.1 -U postgres -d postgres -p 5432

• 创建gitlab初始化数据库
  

helm 安装 redis 集群

chart: github

• 安装日志

[root@master2 ~]#helm install gitlib-redis -n gitlab /opt/helm/redis/
NAME: gitlib-redis
LAST DEPLOYED: Mon Apr 24 09:08:19 2023
NAMESPACE: gitlab
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: redis
CHART VERSION: 17.4.3
APP VERSION: 7.0.8
** Please be patient while the chart is being deployed **
Redis® can be accessed via port 6379 on the following DNS name from within your cluster:
    gitlib-redis.gitlab.svc.cluster.local for read only operations
For read/write operations, first access the Redis® Sentinel cluster, which is available in port 26379 using the same domain name above.
To connect to your Redis® server:
1. Run a Redis® pod that you can use as a client:
   kubectl run --namespace gitlab redis-client --restart='Never'  --image 10.50.10.185/redis/bitnami/redis:7.0.8-debian-11-r0 --command -- sleep infinity
   Use the following command to attach to the pod:
   kubectl exec --tty -i redis-client \
   --namespace gitlab -- bash
2. Connect using the Redis® CLI:
   redis-cli -h gitlib-redis -p 6379 # Read only operations
   redis-cli -h gitlib-redis -p 26379 # Sentinel access
To connect to your database from outside the cluster execute the following commands:
    kubectl port-forward --namespace gitlab svc/gitlib-redis 6379:6379 &
    redis-cli -h 127.0.0.1 -p 6379

gitlab manifest 部署

参考

官方的 chart超级复杂，组件超级多 。Gitlab 主要涉及到3个应用：Redis、Postgresql、Gitlab 核心程序，实际上我们只要将这3个应用分别启动起来，然后加上对应的配置就可以很方便的安装 Gitlab 了，我们这里选择使用的镜像不是官方的，而是 Gitlab 容器化中使用非常多的一个第三方镜像：sameersbn/gitlab

• 登录
• http://chot-gitlab.prod.com:32100/ 账密: root/xxx
• 使用外部redis 和pg
• 使用sameersbn的gitlab镜像
• 使用nginx ingress 向外暴露服务 kubernetes.io/ingress.class: nginx
• gitlab-prod.yaml
• 注意需要将pg和redis分别开启30300和30302 nodePort
• 这一步可优化为 域名连接避免开太多的nodeport

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: gitlab-data-pvc
  namespace: gitlab
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 200Gi
  storageClassName: nfs-storage-179sc
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gitlab
  namespace: gitlab
  labels:
    name: gitlab
spec:
  replicas: 1
  selector:
    matchLabels:
      name: gitlab
  template:
    metadata:
      name: gitlab
      labels:
        name: gitlab
    spec:
      #nodeName: 192.168.102.22  # *
      containers:
      - name: gitlab
        image: 10.50.10.185/gitlab/sameersbn/gitlab:15.8.0-1
        imagePullPolicy: IfNotPresent
        - name: TZ
          value: Asia/Shanghai
        - name: GITLAB_TIMEZONE
          value: Beijing
        - name: GITLAB_SECRETS_DB_KEY_BASE
          value: long-and-random-alpha-numeric-string  # *
        - name: GITLAB_SECRETS_SECRET_KEY_BASE
          value: long-and-random-alpha-numeric-string  # *
        - name: GITLAB_SECRETS_OTP_KEY_BASE
          value: long-and-random-alpha-numeric-string  # *
        - name: GITLAB_ROOT_PASSWORD
          value: admin123  # *
        - name: GITLAB_ROOT_EMAIL
          value: ninesun@126.com  # *
        - name: GITLAB_HOST
          value: chot-gitlab.prod.com  # *
        - name: GITLAB_PORT
          value: "30400"
        - name: GITLAB_SSH_HOST
          value: k8s-22.host.com  # *
        - name: GITLAB_SSH_PORT
          value: "30401"
        - name: GITLAB_NOTIFY_ON_BROKEN_BUILDS
          value: "true"
        - name: GITLAB_NOTIFY_PUSHER
          value: "false"
        - name: GITLAB_BACKUP_SCHEDULE
          value: daily
        - name: GITLAB_BACKUP_TIME
          value: 01:00
        - name: DB_TYPE
          value: postgres
        - name: DB_HOST
          value: gitlib-db-postgresql-hl #headless svc name: gitlib-db-postgresql-hl
        - name: DB_PORT
          value: "5432"
        - name: DB_USER
          value: postgres
        - name: DB_PASS
          value: "postgres"  # *
        - name: DB_NAME
          value: gitlab_production # 这一步如果需要重新创建数据库，就需要单独建立。否则就把gitlab 的数据存储在默认数据库postgres
        - name: REDIS_HOST
          value: gitlib-redis-headless # headless svc name: gitlib-redis-headless
        - name: REDIS_PORT
          value: "6379" # 默认端口是6379
        ports:
        - name: http
          containerPort: 80
        - name: ssh
          containerPort: 22
        volumeMounts:
        - mountPath: /home/git/data
          name: data
        livenessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 180
          timeoutSeconds: 5
        readinessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 25
          timeoutSeconds: 1
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: gitlab-data-pvc
apiVersion: v1
kind: Service
metadata:
  name: gitlab
  namespace: gitlab
  labels:
    name: gitlab
spec:
  ports:
    - name: http
      port: 80
      targetPort: http
      nodePort: 30400
    - name: ssh
      port: 22
      targetPort: ssh
      nodePort: 30401
  type: NodePort
  selector:
    name: gitlab
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: gitlab
  namespace: gitlab
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
    - host: chot-gitlab.prod.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: gitlab
                port:
                  number: 80

• gitlab需要启动多久？
  启动大约需要4mins，期间可能会导致readlines probe检查失败。可以将initialDelaySeconds 调大一些
  

gitlab如何上传项目？

• 第一步: 添加需要推送项目的服务器的公钥
  key的内容就是id_rsa.pub的内容
  
• [外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-NlPzyq3P-1683179705925)(https://s3-us-west-2.amazonaws.com/secure.notion-static.com/19224c7f-f2c7-4569-ace6-28b0a579233b/Untitled.png)]
• 第二步: git 上传项目

# 添加remote url
git remote add origin1 http://chot-gitlab.prod.com:30400/gitlab-instance-f410c318/gitlab-ci-k8s-demo.git
# 查看remote url
git remote -v
origin  https://github.com/myysophia/gitlab-ci-k8s-demo.git (fetch)
origin  https://github.com/myysophia/gitlab-ci-k8s-demo.git (push)
origin1 http://chot-gitlab.prod.com:30400/gitlab-instance-f410c318/gitlab-ci-k8s-demo.git (fetch)
origin1 http://chot-gitlab.prod.com:30400/gitlab-instance-f410c318/gitlab-ci-k8s-demo.git (push)
# 推送代码
git push -u origin1 --all

gitlab监控metrics

这部分可以接入外部的grafana监控面板进行监控

• http://chot-gitlab.prod.com:30400/-/metrics?token=zuqjYZFKMof22VkTRLek

gitlab runner helm 部署

• helm 部署
  chart github:
  
• 全局runner和项目runner
  官方建议自建的实例使用指定的runner，也就是每个项目一个单独的runner
• 两者如何选择？
  
• 全局runner token
  
• 项目runner token
  

helm部署问题

域名解析问题

• 参考链接

https://todoit.tech/k8s/gitlab-runner/

https://blog.csdn.net/boling_cavalry/article/details/106991576

报错处理: Incorrect Usage: flag provided but not defined: -template-config

版本问题

GitLab 社区版 15.8.0 需要使用对应的runner镜像(** https://docs.gitlab.com/runner/ )****

docker push 10.50.10.185/gitlab/registry.gitlab.com/gitlab-org/gitlab-runner:alpine-v15.8.0

如何查看gitlab版本: http://chot-gitlab.prod.com:32100/help

config.toml 配置 如何覆盖config.template.toml

非root用户→ 容器中~/.gitlab-runner/config.yaml

concurrent = 10
check_interval = 30
log_level = "info"
shutdown_timeout = 0
[session_server]
  session_timeout = 1800
[[runners]]
  name = "chot-gitlab-runner-minio-gitlab-runner-55564b6469-k2d6h"
  url = "http://chot-gitlab.prod.com:32100/"
  id = 8
  token = "vbR7MMTPKSL7dyPALsUN"
  token_obtained_at = 2023-02-17T03:24:19Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "kubernetes"
  [runners.custom_build_dir]
  [runners.cache]
    Type = "s3"
    Shared = true
    MaxUploadedArchiveSize = 0
    [runners.cache.s3]
      ServerAddress = "chot-minio-web.prod.com:32100"
      AccessKey = "IwA5ttRQsZlKkkQV"
      SecretKey = "C07BrPYktE997bMcWUdcHyXQPVPr3mSJ"
      BucketName = "gitlab"
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.kubernetes]
    host = ""
    bearer_token_overwrite_allowed = false
    image = "10.50.10.185/gitlab/ubuntu:16.04"
    namespace = "gitlab"
    namespace_overwrite_allowed = ""
    pull_policy = ["if-not-present"]
    node_selector_overwrite_allowed = ""
    helper_image = "10.50.10.185/gitlab/ubuntu:16.04"
    pod_labels_overwrite_allowed = ""
    service_account_overwrite_allowed = ""
    pod_annotations_overwrite_allowed = ""
    [runners.kubernetes.affinity]
    [runners.kubernetes.pod_security_context]
    [runners.kubernetes.init_permissions_container_security_context]
      [runners.kubernetes.init_permissions_container_security_context.capabilities]
    [runners.kubernetes.build_container_security_context]
      [runners.kubernetes.build_container_security_context.capabilities]
    [runners.kubernetes.helper_container_security_context]
      [runners.kubernetes.helper_container_security_context.capabilities]
    [runners.kubernetes.service_container_security_context]
      [runners.kubernetes.service_container_security_context.capabilities]
    [runners.kubernetes.volumes]
    [runners.kubernetes.dns_config]
    [runners.kubernetes.container_lifecycle]

• 跑通一个gitlab cicd
  https://github.com/yangshun2005/gitlab-cicd
  
• gitlab exector 报错处理

• • 使用k8s exector 无法拉取镜像

Running with gitlab-runner 15.8.0 (12335144)
  on chot-gitlab-runner-minio-gitlab-runner-6fbf87f59b-j6bhq S7qPFnrs, system ID: r_vB5NUhtcRQ1R
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: gitlab
Using Kubernetes executor with image ubuntu:16.04 ...
Using attach strategy to execute scripts...
Preparing environment
00:03
Waiting for pod gitlab/runner-s7qpfnrs-project-1-concurrent-0bc7zl to be running, status is Pending
WARNING: Failed to pull image with policy "IfNotPresent": image pull failed: rpc error: code = Unknown desc = Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on 10.0.2.3:53: no such host
ERROR: Job failed: prepare environment: waiting for pod running: pulling image "ubuntu:16.04": image pull failed: rpc error: code = Unknown desc = Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on 10.0.2.3:53: no such host. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information

修改runner values的文件

image:
    registry: 10.50.10.185/gitlab
    image: ubuntu
    tag: 16.04
Waiting for pod gitlab/runner-gm-nhepv-project-1-concurrent-057xgd to be running, status is Pending
ERROR: Job failed: prepare environment: waiting for pod running: image pull failed: Failed to apply default image tag "map[image:ubuntu registry:10.50.10.185/gitlab tag:16.04]": couldn't parse image reference "map[image:ubuntu registry:10.50.10.185/gitlab tag:16.04]": invalid reference format. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information

exector不支持这种image的方式

修改为

image: 10.50.10.185/gitlab/ubuntu:16.04

镜像拉取成功后会启动两个容器一个helper 一个 build镜像

如果runner分配给exector的工作1小时没有完成，这些容器就退出了。

error log

• helper 和 build镜像启动后，容器启动报错

/bin/bash: line 1: gitlab-runner-build: command not found

gitlab 流水线报错如下:

OCI runtime exec failed: exec failed: unable to start container process: exec: "gitlab-runner-helper": executable file not found in $PATH: unknown
# 从pod yaml文件中找出 其中helper 容器中执行这段脚本
if [ -x /usr/local/bin/bash ]; then
exec /usr/local/bin/bash 
elif [ -x /usr/bin/bash ]; then
exec /usr/bin/bash 
elif [ -x /bin/bash ]; then
exec /bin/bash 
elif [ -x /usr/local/bin/sh ]; then
exec /usr/local/bin/sh 
elif [ -x /usr/bin/sh ]; then
exec /usr/bin/sh 
elif [ -x /bin/sh ]; then
exec /bin/sh 
elif [ -x /busybox/sh ]; then
exec /busybox/sh 
echo shell not found
exit 1

查阅官方文档后发现这个helper 镜像还是个专有镜像，下载gitlab runner对应版本的helper, 这个helper镜像就是那个具体干活的， gitlab 的 .gitlab-ci.yml 根 据配置的时间间隔把活给gitlab runner。

gitlab runner则是让helper images 去处理。所以刚开始的时候helper的image 用ubuntu是不对的。

具体理解参考 Override the helper image 部分 :

Advanced configuration | GitLab

bitnami/gitlab-runner-helper:15.8.0

Running with gitlab-runner 15.8.0 (12335144)
  on chot-gitlab-runner-minio-gitlab-runner-75f87cfdbf-d89z6 fhyNaFUz, system ID: r_MDgwGq2YmKGA
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: gitlab
Using Kubernetes executor with image 10.50.10.185/gitlab/ubuntu:16.04 ...
Using attach strategy to execute scripts...
Preparing environment
Waiting for pod gitlab/runner-fhynafuz-project-1-concurrent-0v9gq4 to be running, status is Pending
Waiting for pod gitlab/runner-fhynafuz-project-1-concurrent-0v9gq4 to be running, status is Pending
    ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
    ContainersNotReady: "containers with unready status: [build helper]"
    ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab/runner-fhynafuz-project-1-concurrent-0v9gq4 to be running, status is Pending
    ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
    ContainersNotReady: "containers with unready status: [build helper]"
    ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab/runner-fhynafuz-project-1-concurrent-0v9gq4 to be running, status is Pending
    ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
    ContainersNotReady: "containers with unready status: [build helper]"
    ContainersNotReady: "containers with unready status: [build helper]"
/bin/bash: line 1: gitlab-runner-build: command not found

实在没办法，提个issue吧

‣

April 21, 2023

官方更新了gitlab-runner-helper镜像版本

k8s中gitlab exector架构图

一图胜千言。

The Kubernetes executor for GitLab Runner | GitLab

what is Gitlab Runner Helper?

helm部署gitlab runner的时候有一个help image。

Gitlab Runner Helper 是与 Gitlab Runner 一起使用的辅助容器。 Gitlab Runner 允许运行 CI/CD 作业并将结果发送回 Gitlab。

gitlab如何连接上k8s？ KAS Kubernetes agent server

gitlab CI

gitlab CI template

lib/gitlab/ci/templates · master · GitLab.org / GitLab · GitLab

gitlab CI Demo

上来不应该直接尝试on k8s，为啥不从最简单的开始呢？

使用 docker部署 gitlab-runner ，注册一个 exector 为shell的runner

先来个简单的.gitlab-ci.yaml

before_script:
  - echo "Before script section122333"
  - echo "For example you might run an update here or install a build dependency"
  - echo "Or perhaps you might print out some debugging details"
after_script:
  - echo "After script section"
  - echo "For example you might do some cleanup here"
build1:
  stage: build 
  script:
    - echo "Do your build heresd"
test1:
  stage: test
  script:
    - echo "Do a test here"
    - echo "For example run a test suite"
test2:
  stage: test
  script:
    - echo "Do another parallel test here"
    - echo "For example run a lint test"
deploy1:
  stage: deploy
  script:
    - echo "Do your deploy here"
  environment: production

第一步: 1. 建立 Docker Volume

目前runner部署在10.50.10.36 理论上哪台有docker环境的都可以的.

$ docker volume create gitlab-runner-config

第二步: 使用创建的卷启动GitLab Runner容器：

-env TZ=CST

docker run -d --name gitlab-runner --restart always \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v gitlab-runner-config:/etc/gitlab-runner \
        --add-host=chot-gitlab.prod.com:10.50.10.33 \
    10.50.10.185/gitlab/gitlab/gitlab-runner:v15.8.0

第三步: register 一个runner， exector 选择shell

root@27172e22cf90:/# gitlab-runner register
Runtime platform                                    arch=amd64 os=linux pid=54 revision=12335144 version=15.8.0
Running in system-mode.
Enter the GitLab instance URL (for example, https://gitlab.com/):
http://chot-gitlab.prod.com:30400/
Enter the registration token:
GR1348941BbkUVr8B1UumMfNx4LrL
Enter a description for the runner:
[27172e22cf90]:
Enter tags for the runner (comma-separated):
Enter optional maintenance note for the runner:
WARNING: Support for registration tokens and runner parameters in the 'register' command has been deprecated in GitLab Runner 15.6 and will be replaced with support for authentication tokens. For more information, see https://gitlab.com/gitlab-org/gitlab/-/issues/380872
Registering runner... succeeded                     runner=GR1348941BbkUVr8B
Enter an executor: docker-ssh, virtualbox, docker-ssh+machine, instance, ssh, docker+machine, kubernetes, custom, docker, parallels, shell:
docker
Enter the default Docker image (for example, ruby:2.7):
10.50.10.185/gitlab/ubuntu:16.04
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
Configuration (with the authentication token) was saved in "/etc/gitlab-runner/config.toml"

或者一步到位:

gitlab-runner register --url http://chot-gitlab.prod.com:30400/ --registration-tokenGR1348941BbkUVr8B1UumMfNx4LrL --executor shell

第四步 查看CI结果

• 不同exector的runner配置
  gitlab-runner register时可以指定不同的exector. 可以打不同的tag，让不同的项目或编译条件使用不同的exector.

gitlab-runner list
Runtime platform                                    arch=amd64 os=linux pid=27160 revision=12335144 version=15.8.0
Listing configured runners                          ConfigFile=/etc/gitlab-runner/config.toml
meta-162                                            Executor=docker Token=_mFCm2xiCnFd8rKaAKFg URL=http://chot-gitlab.prod.com:30400/
meta-162                                            Executor=shell Token=_yvc4o5ycSACmtyTvBVK URL=http://chot-gitlab.prod.com:30400/

CI使用minio作为cache

例如在java项目编译的时候会有很多依赖包需要下载，如果每次都从网络拉取不稳定，这时候如果把依赖包缓存起来，下次构建项目就很快.

[runners.cache]
        Type = "s3"
        Shared = true
        [runners.cache.s3]
          AccessKey = "IwA5ttRQsZlKkkQV"
          SecretKey = "C07BrPYktE997bMcWUdcHyXQPVPr3mSJ"
          BucketName = "gitlab"
          ServerAddress = "chot-minio-api.prod.com:32100"

gitlab 和gitlab-runner的关系

GitLab CI 之 Runner 的 Executor 該如何選擇？

runner和exector的关系

runner并不是实际干活的，runner 可以指定特定的exector干活，在gitlab中有不同的exector，目前有这几种exector:

• docker-ssh
• virtualbox
• docker-ssh+machine
• instance
• ssh
• docker+machine
• kubernetes
• custom
• docker
• parallels
• shell

可以结合自己的技术栈进行选择，目前我们常使用的是shell 和 docker、k8s

gitlab环境变量

这块都是gitlab预设定的一些环境变量，更多请 打开 CI_DEBUG_TRACE mode。

例如下面三个环境变量分别是当前gitlab实例名、绝对路径和相对路径。

$ echo "${CI_PROJECT_NAMESPACE}"
gitlab-instance-f410c318
$ echo "${CI_PROJECT_DIR}"
/home/gitlab-runner/builds/hs9MHCAM/0/gitlab-instance-f410c318/gitlab-ci-k8s-demo
$ echo "${CI_PROJECT_PATH}"
gitlab-instance-f410c318/gitlab-ci-k8s-demo

参考

用 GitLab CI 进行持续集成