Jenkins Pipeline实现自动构建部署到Kubernetes

阿楞Danbo丶

本文中采用jenkins pipeline执行整个jenkins的构建过程，在pipeline中使用不同的docker容器实现检出代码、代码编译、构建镜像并上传到镜像仓库、部署到kuberneres。

配置jenkins

1.插件配置

更换插件源

Manage Jenkins => Plugin Manager => Advanced => Update Site => https:// mirrors.tuna.tsinghua.edu.cn /jenkins/updates/update-center.json #清华源

安装插件

Manage Jenkins => Plugin Manager => Available => Localization: Chinese (Simplified)

Manage Jenkins => Plugin Manager => Available => Blue Ocean

Manage Jenkins => Plugin Manager => Available => Kubernetes（helm安装默认会有该插件）

Manage Jenkins => Plugin Manager => Available => docker-workflow

重启jenkins生效（可以在域名后加/restart进行重启）

2.前期配置

.kube/config

kubectl create secret generic jenkins-k8s-cfg -n jenkins --from-file=/root/.kube/config 
secret/jenkins-k8s-cfg created

k8s中使用kubectl命令时需要yaml格式的服务器及授权信息配置文件。这里将kubectl的yaml配置文件的内容以base64编码后保存在jenkins的凭据中。pipeline任务执行时，先从jenkins凭据中获取内容，进行base64解码后将配置保存为~/.kube/config文件。kubectl的配置文件的内容如下：

apiVersion: v1 
clusters: 
- cluster: 
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM2akNDQWRLZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJeU1ESXhOakEyTkRBeE5Gb1hEVE15TURJeE5EQTJORFV4TkZvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBUHEzCmRGSDJMbytrenFabzhMNEYrNUtJWUhQVDQzZk1pK1NJb2VZdDNFTE9TZ1Y5bHJLeVJLZDBrNkVpV0FNZ3U2N3EKTW1SbUc1SHJKWGgycVgwL0RXaFVmRENFL09ESFhyOCtNaTBtcGU2dmVXTi8xZ2NJS2hyaFQ2MXlPY1RWVngvYgpDS2QzaTFIVTdUR1I4T3JmQ1RleDlLUUdQbmhhbUZHcXNtNW8yaS9jNmFVdW4xeTBEYlp4NHFEcktFTG5UTytoCjlNaFJ4U2crT3lObGVyQjB5Y0xKNWZOSVJ6U1A0bzkrV3NkTDQ2bU5RaytzZ3FpaU1HdGFldEVMQ2x1WDdkK0EKamxHZy92a0pWVnBDYkNjcDJDdHpnT0tqTWd4YmI2N3I2ZnY1NndiaEoxc0R2Mkdvb1JnR0htbi9JRlhNK2xTVwpsTTlZRloxZHQwc01GSGhUdDRzQ0F3RUFBYU5GTUVNd0RnWURWUjBQQVFIL0JBUURBZ0trTUJJR0ExVWRFd0VCCi93UUlNQVlCQWY4Q0FRQXdIUVlEVlIwT0JCWUVGRFJva1d3U0VPMElpM1d6OUhLcU16bnRaYVN3TUEwR0NTcUcKU0liM0RRRUJDd1VBQTRJQkFRRGtoZHQwVEljU0hLVGVvRERpbGNpUHJ3SnJlaTJPRklhUXAvWi9pdzZ2bk5SRQpQUkZLVENSYk9IKzV1eTk5dWJKSHdMamFuMWRJYk15YzhMd2Q3WnhWMzRwNHkwNkdzUVRZVjE5aUtTWUNUOGlZCmlyWGJWRUNFNlN3SUc4WTVDL285Lys5U2tmMkN0WEd6K3pHTnptVms2c0JxaVlIMkNCVkhIaTlEOHkrNC85b1EKcWY4amNudmZzRi85QzQ2ME01SytxZG1MY3dsc29LRklBMitYb0NRRjMvcU1SR21Bay95QjlCbmxsbkxHcHRmcQpwNnc1SmtSRW9JczVwTkNvK01iWEMzT3VhZzlTZndDamgxdEtVOEdxdjdndldsa3NneWZSQVBkTTkxa2YzRk9xCndvZGVZQmZOSElpK0Y4VjR2YWdDekovaW9US2dXSUpoOThiTkNrcW0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= 
    server: https://172.172.172.172:6443 
  name: cc869q4ursfehmf5j0av0 
contexts: 
- context: 
    cluster: cc869q4ursfehmf5j0av0 
    user: "1009752" 
  name: cc869q4ursfehmf5j0av0@1009752 
current-context: cc869q4ursfehmf5j0av0@1009752 
kind: Config 
preferences: {} 
users: 
- name: "1009752" 
  user: 
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM4VENDQWRtZ0F3SUJBZ0lJVTRRbHNITGorTlV3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TWpBeU1UWXdOalF3TVRSYUZ3MDBNakF6TWpRd05qSTJOVEJhTUJJeApFREFPQmdOVkJBTVRCekV3TURrM05USXdnZ0VpTUEwR0NTcUdTSWIzRFFFQkFRVUFBNElCRHdBd2dnRUtBb0lCCkFRRFBrcGwzZmhaWStRSXh3RzNaOTNiemZaTExxemtpaFZ0N1B4WitGZXNTQ2MzNE1aZ1J5eGhiVlM5VXZ2Z20KTkNOemwzUVdVOWg2bEMvQjQ3NUh3eDFXZkZZN1ZMcjM5T1ZueWk2NE5wUENRNDd1SXVDRnhmYnBNZnlBWG4rZgpSN3RNS2VqV0tSTDFzRUhrSi85QkdiT3VEUWJGa1BmUUpoZ3VYZ0tpazFzbWQxZ2Y2OHNnclR4NHpSeHNDcTJuClU2V1dIL1lVTnQzOUIveTI5SXRYSVBMczVnb0paSVJyS2JtZEFZekExbkdGajMyc2tCRWpHU21XQnNUcTdGYXkKWXB4anBNc29LVVE3RzBkS0dsdXlDOGhySFFNdVpPblNSVkJpSkNUZ3ZycmpwMnhJOHlkZUhCOXBhVlJWdkJqQwpibzJmb3oyWFVKeUE1NkdLZnNyWklQTjNBZ01CQUFHalNEQkdNQTRHQTFVZER3RUIvd1FFQXdJRm9EQVRCZ05WCkhTVUVEREFLQmdnckJnRUZCUWNEQWpBZkJnTlZIU01FR0RBV2dCUTBhSkZzRWhEdENJdDFzL1J5cWpNNTdXV2sKc0RBTkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQTFxNjEvK3NuRFZpendhUXNWZXR3WFcwcGZJd0wzNjZBNWJabApPNzV2OUpXWmQzN3A2ZXltRkhacWx0RWNncjNMNlNJUWdzanM4MWR0dEYyTCtOSk1vQk5JbFkvY3JCREx2VDQvCjNOZUtIUEVFQlRaODZVZE9tWCsrVEJHUGhUSllaam9YRXhBOWYwaUQrTHlsTFR3YmxvMnFrY2JoZTAyaFM4Y1AKRWs3QnhhYU9HR09lYXZ3SjFLMHJEN25hWGpkWFZvTEVoUnFRZmNVWTZwQUQ4ZzQ3V3JUK3lkdEhPSHdBajhEVgpOMEFpZDVRYW5qdFE1WU40ZXdsVi92SXlLV1dzTnRYbEpYM1BiaW1lQmVuYTZTdGJlTVNvUXRyZmZteHZWNFZSCkdFVENBdFdiZE90WS90V3BTR2tkZXdDL1VVY01PUlFBU3FmK3U3d1liY1JkWDIweEpRPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= 
    client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFBS0NBUUVBejVLWmQzNFdXUGtDTWNCdDJmZDI4MzJTeTZzNUlvVmJlejhXZmhYckVnbk4rREdZCkVjc1lXMVV2Vkw3NEpqUWpjNWQwRmxQWWVwUXZ3ZU8rUjhNZFZueFdPMVM2OS9UbFo4b3V1RGFUd2tPTzdpTGcKaGNYMjZUSDhnRjUvbjBlN1RDbm8xaWtTOWJCQjVDZi9RUm16cmcwR3haRDMwQ1lZTGw0Q29wTmJKbmRZSCt2TApJSzA4ZU0wY2JBcXRwMU9sbGgvMkZEYmQvUWY4dHZTTFZ5RHk3T1lLQ1dTRWF5bTVuUUdNd05aeGhZOTlySkFSCkl4a3BsZ2JFNnV4V3NtS2NZNlRMS0NsRU94dEhTaHBic2d2SWF4MERMbVRwMGtWUVlpUWs0TDY2NDZkc1NQTW4KWGh3ZmFXbFVWYndZd202Tm42TTlsMUNjZ09laGluN0syU0R6ZHdJREFRQUJBb0lCQUM2NkFLUWVsU2RJUHM4MgpVMFJLNmRNY1JJeG1xZ0QwWmtpVFJodnlFUzNKV1QxeTVKbTB6M0hYOWtTc2pLNEU5S29Ud1FVNXNMUy9ZUlRUCmtNVVV6aWptNm42R3IyUWNFL3hPVGtBWHdTRnRGZUxUcFJsWUhDSVlwNCs2Tmg2RUJreTU3bmt2Vzk1RTVFQloKSktQaUJrbTdFdmdzanl5ak5yUlJZMlIzcVlBNTI1QStJNWhLR25oczd4Z2R1RmRZUmlleUlDN1diaTlzeTljZwpqTWRxZlJZeDF6QnJ1WUlDRi8rWmlleWFZVk15RlRwREFZVlhYeEx0RWpmOW8zenV6ek14RURTOFEwd05jRGd6Clh2c3paM0dWL2Nack5kYVNib0szRTI1TmRiZmN5NDBIYlJjUzZLOStheGQ2OGMwVEFuQkFVRitoekhGZGJvV0wKSEVBSjhiRUNnWUVBM0hja0hOVnBSTG9ML1dBeUl0L3Azc3pKak84SHdRK2g3bWJud2EvdGhNWCtLVStFa2JOZQo1c3dDMXRzRmRib1o1cHFMYkMyYXNIUGpGMDI4UEFWT1hHS0pRcG1KZmkxNkY0aVdFRHNOMXY0NGQ4b1JQSlRzCjNKMHBqdlBWTVUvK3hnNnJFWGRsaGFCN3l1b1Bxc0FLdVdmVnRJcU1vOURLUFA4cG9DUkRUS2tDZ1lFQThRZDYKSzZQc2xkRDhGOU9wZVhYTFRrSFlXUDJsQmExNUJIbW5mR3NOaGg4Qy9qUUpocFVZS2IxcGdXZDhUaHJGSkw4cwplTTcweXNEL2cyVUEyR1c0aWs2ckVNdFpjWnhvNkhHZHRvQjZIMzhFdWhqeXdsT2FuWkdmZHpTQ21rT2pVS0dkCmxDcDk1YkhVZkxQNitxcm5CWWRyaEtTdXpOcGFUVGtiSk5zdE14OENnWUVBakVMa1g4L0t0R1hsUU9OVm9taHcKbzBZSXFMaGVaNDZwaUVKQ0NqRUNYT21XWlBQcE5NVTRpSCtkQXRLL2E0SGwvcGV1UWpuVlk1T25FK2xwNjRtNApyQUFvc1kxN015Sjl2Y0V4TDh4U1pzakFaajNScjRoVUZadUxtQjQ3TWE5aU9PR1dKbnVleEdBd3dnMGR3SldECmZTZTVoNjVxQWNMUkYvT0NxUmJKRmxrQ2dZQkd4MGhUK2w0bUhHbDY0YmY1cCtLY3gyOHRxc1pENVk0TkIyQnMKZzg5RDNHVjNtblpWcEFDbGVTNjdSbFZWWmowUUxtZHp6bXBLQStURWcrdDhvRnBSbXdwVzh5Ly8wY3RvdnJ4dQpkRW5vWjlkT2dyYis0NmdhcjBHaHNyVExjaU5LTVR0aDRCeDZULzh5NDNOSEd0cWVZQWJ0dTZKdEc0MjY4Wk1MCi9xVEdSd0tCZ0JwUzc3WDZ3ald0c3RGOGRrbDVSUmdFZXhOUUcxMzdLblY1aEgxbjVnamFzYWFGYXg3SUcwSVUKZzZ1WVJJZ3VWazhEMDN1WDZRWXNGRVNEUG91RlVGTVdnWitBanVoVys3c3NVRFByZW5veklGWmgrd2xsdEwvZgpkQ1lwb2lEQTh6TmV6c0wrMDRhQ2djeVZNZUlCUnlxTzI1THpZUzNVT0g0VW9XTkdRRDNECi0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==

将以上信息保存到kube-config.yml

base64 kube-config.yml > kube-config.txt

在jenkins凭据中增加配置文件内容。在凭据设置界面，类型选择为“Secret text”，ID设置为“jenkins-k8s-config”（此处的ID必须与Jenkinsfile中的保持一致），Secret设置为上面经过base64编码后的配置文件内容。

为 maven 容器配置缓存和挂载 settings.xml

<?xml version="1.0" encoding="UTF-8"?> 
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0" 
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd"> 
    <mirrors> 
        <!-- 阿里云仓库 --> 
        <mirror> 
            <id>alimaven</id> 
            <mirrorOf>central</mirrorOf> 
            <name>aliyun maven</name> 
            <url>http://maven.aliyun.com/nexus/content/repositories/central/</url> 
        </mirror> 
        <!-- 中央仓库1 --> 
        <mirror> 
            <id>repo1</id> 
            <mirrorOf>central</mirrorOf> 
            <name>Human Readable Name for this Mirror.</name> 
            <url>http://repo1.maven.org/maven2/</url> 
        </mirror> 
        <!-- 中央仓库2 --> 
        <mirror> 
            <id>repo2</id> 
            <mirrorOf>central</mirrorOf> 
            <name>Human Readable Name for this Mirror.</name> 
            <url>http://repo2.maven.org/maven2/</url> 
        </mirror> 
    </mirrors>  
</settings>

kubectl -n jenkins create configmap maven-config --from-file=settings.xml

gitlab配置公钥

创建密钥

mkdir -p /usr/local/src/ssh/.ssh/ 
ssh-keygen -t rsa -b 2048 -C "abc@abc.com"  -N "" -f /usr/local/src/ssh/.ssh/id_rsa

gitlab配置公钥

jenkins 创建凭据

选择ssh username with private key类型添加私钥

配置镜像仓库密钥

jenkins动态slave测试

pipeline文件

def label = "jenkins-slave-${UUID.randomUUID().toString()}" // 
podTemplate(label: label, cloud: 'kubernetes') { 
    node(label) { 
        stage('Run shell') { 
            sh 'sleep 10s' 
            sh 'echo hello world.' 
}

部署项目

创建pipeline项目

参数化构建工程，选择字符参数

名称: APP_NAME

默认值: deployment-demo

描述: 项目名称，用于创建deployment名称

名称: APP_NS

默认值: default

描述: namespace名称，deployment资源创建指定的ns

名称: GIT_VER

默认值: release

描述: 项目所在git中央仓库对应项目的分支或者版本号，例如release分支

名称: GIT_REPO

默认值: ssh://git@git.abc.com:tour/java/abc.git

描述: 项目所在的git中央仓库的地址

名称: MVN_DIR

默认值: ./

描述: 编译项目目录，默认为项目的根目录

名称: MVN_CMD

默认值: mvn -B clean package -Dmaven.test.skip=true -Dautoconfig.skip

描述: 执行mvn编译所用的命令

名称: IMAGE_NAME

默认值: base

描述: docker镜像名称，格式：<仓库名>/<镜像名>

名称: IMAGE_REPO

默认值: http:// registry.cn-beijing.cr.aliyuncs.com

描述: docker镜像仓库名称

名称: BUSINESS_LINE

默认值: xundian

描述: 业务线名称

名称: BRANCH_TAG

默认值: master

描述: 代码分支名称

流水线

// 流水线的最外层结构，代表整条pipeline，包含着pipeline完整逻辑 
pipeline { 
  // 环境变量的定义 
  environment { 
   K8S_CONFIG = credentials('jenkins-k8s-config')  // 通过 ${params.xxx} 的方式对此参数进行引用 
  // pipeline中单独指令，用于指定流水的执行位置，它可能是代表着slave主机的某个物理机、虚拟机或者容器 
  agent { 
    kubernetes { 
      yaml ''' 
        apiVersion: v1 
        kind: Pod 
        metadata: 
          labels: 
            label: jenkins-mvn 
        spec: 
          containers: 
          - name: docker 
            image: registry.cn-beijing.aliyuncs.com/aquaman0319/docker:19.03-dind 
            command: 
            - dockerd 
            - --host=unix:///var/run/docker.sock 
            - --host=tcp://0.0.0.0:8000 
            securityContext: 
              privileged: true 
            tty: true 
          - name: kubectl 
            image: registry.cn-beijing.aliyuncs.com/aquaman0319/helm-kubectl-docker:v1.21.1-v3.6.0 
            command: 
            - cat 
            tty: true 
          - name: maven38 
            image: registry.cn-beijing.aliyuncs.com/aquaman0319/maven:3.8-jdk-8 
            command: 
            - cat 
            tty: true 
            volumeMounts: 
            - mountPath: "/root/.m2/repository" 
              name: "volume-0" 
            - mountPath: "/root/.m2" 
              name: "volume-1" 
          volumes: 
            - name: "volume-0" 
              persistentVolumeClaim: 
                claimName: "jenkins-m2" 
            - name: "volume-1" 
              configMap: 
                name: "maven-config"     
  // 用于包含所有的stage的定义 
  stages { 
    // 阶段，代表流水线的一个单独的功能完成时期，例如编译等 
    stage('检出代码') { 
          // 步骤，用于在stage中定义完成该阶段功能所需经历的一系列步骤 
      steps { 
        // 递归删除WORKSPACE下的文件和文件夹,避免缓存导致构建问题 
        deleteDir() 
                // credentialsId为ssh 凭据的id 
        checkout([$class: 'GitSCM', branches: [[name: "${params.GIT_VER}"]], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: "gitlabauth", url: "${params.GIT_REPO}"]]])         
    stage('构建代码') { 
      steps{ 
          // 用于创建容器的容器模板 
              container('maven38') { 
          sh "cd ${params.MVN_DIR} && ${params.MVN_CMD}" 
          // post定义将在pipeline运行或stage结束时运行的操作 
      post { 
        // 成功之后提取制品 
        success { 
          archiveArtifacts "target/*.jar" 
    stage('构建并推送镜像') { 
      steps { 
            container('docker') { 
        script { 
                    echo "构建Docker镜像" 
                    def CREATE_DATE=sh (script: "date '+%Y%m%d'",returnStdout: true).trim() 
                    TAG = "${params.BRANCH_TAG}"+"."+"${CREATE_DATE}"+"."+"${BUILD_NUMBER}" 
                    echo "${TAG}" 
                    sh (script: "cp ./Dockerfile ./target && cd ./target ") 
                    withDockerRegistry([credentialsId:'jenkins-docker-cfg', url:"https://${IMAGE_REPO}"]){ 
                        def image = docker.build("${params.IMAGE_REPO}/${params.BUSINESS_LINE}/${params.IMAGE_NAME}:${TAG}", ".") 
                        image.push("${TAG}") 
                    echo "已推送Docker镜像到镜像仓库" 
                    sh (script: "echo '${params.IMAGE_REPO}/${params.BUSINESS_LINE}/${params.IMAGE_NAME}:${TAG}' > tmp_image") 
    stage('更新服务') { 
      steps { 
            container('kubectl') { 
        sh "mkdir -p ~/.kube" 
        sh "echo ${K8S_CONFIG} | base64 -d > ~/.kube/config" 
              sh "kubectl apply -f devops/deploy-dev.yaml -n xundian"