跳转至

jenkins-pipeline 插件

jenkins-pipeline 插件用于打通 GitHub/GitLab 和 Jenkins,实现自动化创建 Jenkins Pipeline 的功能。

本文将演示:

  1. 通过 repo-scaffolding 插件在 GitLab 上创建一个 Java Spring Boot 项目脚手架;
  2. 通过 jenkins-pipeline 插件在 Jenkins 上创建一条 Java Spring Boot 的 CI 流水线;
  3. 通过 jenkins-pipeline 插件实现在 GitLab 和 Jenkins 上分别配置相应参数,实现当 GitLab 上的代码库有 push 或者 merge 事件时,自动触发 Jenkins 上的流水线运行,同时流水线的执行结果自动回写到 GitLab 上。

提示

GitHub 与 GitLab 的主要区别在于 DevStream tool config 的 options.scm.url 以及所需要的 token 等不同。

1、前置要求

必须满足

  • 一套可用的 Jenkins 环境
  • 一套可用的 GitLab 环境
  • 一套可用的 Harbor 环境
  • Jenkins 与 GitLab、Harbor 网络互通
  • 执行 dtm 的主机与 Jenkins、GitLab 网络互通

注意

当前插件暂时只支持对接使用 dtm 部署的 Jenkins。

本文基于如下环境编写:

  • 系统:一台装有 CentOS 7 的云主机;
  • k8s:minikube 方式部署的单节点集群;
  • Jenkins、Harbor 和 GitLab:使用 dtm 方式部署。

2、配置

你需要预先准备好 GitLab 的 token 和镜像仓库(Harbor 等)的密码,然后配置到环境变量里,例如(记得替换占位字符):

Bash
export IMAGE_REPO_PASSWORD=YOUR_IMAGE_REPO_PASSWORD
export GITLAB_TOKEN=YOUR_GITLAB_TOKEN

提示

如果是 GitHub,则这里的环境变量改成:

Bash
export IMAGE_REPO_PASSWORD=YOUR_IMAGE_REPO_PASSWORD
export GITHUB_TOKEN=YOUR_GITHUB_TOKEN

然后准备 DevStream 插件配置:

下面的配置文件展示的是"tool file"的内容。

关于更多关于DevStream的主配置、tool file、var file的信息,请阅读核心概念概览DevStream配置.

YAML
tools:
- name: repo-scaffolding
  instanceID: springboot
  dependsOn: []
  options:
    destinationRepo:
      owner: root
      name: spring-demo
      branch: master
      scmType: gitlab
      baseURL: YOUR_GITLAB_ADDR
      token: [[ env GITLAB_TOKEN ]]
    sourceRepo:
      owner:  devstream-io
      name: dtm-repo-scaffolding-java-springboot
      scmType: github
- name: jenkins-pipeline
  instanceID: default
  dependsOn: [repo-scaffolding.springboot]
  options:
    jenkins:
      url: YOUR_JENKINS_ADDR
      user: admin
      enableRestart: true
    scm:
      url: git@YOUR_REPO_CLONE_ADDRESS/root/spring-demo
      branch: master
      apiURL: YOUR_JENKINS_ADDR
      token: [[ env GITLAB_TOKEN ]]
    pipeline:
      jobName: test-job
      configLocation: https://raw.githubusercontent.com/devstream-io/dtm-jenkins-pipeline-example/main/springboot/Jenkinsfile
      imageRepo:
        url: YOUR_HARBOR_ADDR
        user: admin
        password: [[ env IMAGE_REPO_PASSWORD ]]

上述配置文件中使用的 GitLab、Jenkins 和 Harbor 访问地址需要替换成你的环境中实际地址。例如:

  • YOUR_GITLAB_ADDR: http://44.33.22.11:30080
  • YOUR_REPO_CLONE_ADDRESS: http://44.33.22.11:30022
  • YOUR_JENKINS_ADDR: http://44.33.22.11:32000
  • YOUR_HARBOR_ADDR: http://harbor.example.com:80

除了这几个必须修改的配置项外,其他配置项你可以在确保理解含义的前提下灵活决定是否调整。

注意:这个配置示例仅是 tool config,完整的 DevStream 配置文件还需要补充 core config 等内容,具体参考这个文档

3、开始执行

你可以使用 dtm apply 命令开始应用当前配置:

  • dtm apply -f config-jenkins-pipeline.yaml -y

这个过程的日志大致如下:

Bash
2022-09-05 09:01:08  [INFO]  Apply started.
2022-09-05 09:01:08  [INFO]  Using dir </root/.devstream/plugins> to store plugins.
2022-09-05 09:01:09  [INFO]  Using local backend. State file: devstream.state.
2022-09-05 09:01:09  [INFO]  Tool (repo-scaffolding/springboot) found in config but doesn't exist in the state, will be created.
2022-09-05 09:01:09 ℹ [INFO]  Tool (jenkins-pipeline/default) found in config but doesn't exist in the state, will be created.
2022-09-05 09:01:09  [INFO]  Start executing the plan.
2022-09-05 09:01:09  [INFO]  Changes count: 2.
2022-09-05 09:01:09  [INFO]  -------------------- [  Processing progress: 1/2.  ] --------------------
2022-09-05 09:01:09  [INFO]  Processing: (repo-scaffolding/springboot) -> Create ...
2022-09-05 09:01:10  [SUCCESS]  Tool (repo-scaffolding/springboot) Create done.
2022-09-05 09:01:10  [INFO]  -------------------- [  Processing progress: 2/2.  ] --------------------
2022-09-05 09:01:10  [INFO]  Processing: (jenkins-pipeline/default) -> Create ...
2022-09-05 09:01:10  [INFO]  Secret jenkins/docker-config has been created.
2022-09-05 09:01:14  [SUCCESS]  Tool (jenkins-pipeline/default) Create done.
2022-09-05 09:01:14  [INFO]  -------------------- [  Processing done.  ] --------------------
2022-09-05 09:01:14  [SUCCESS]  All plugins applied successfully.
2022-09-05 09:01:14  [SUCCESS]  Apply finished.

4、执行结果

你可以在 GitLab、Jenkins 上查看本次执行结果:

  • Repo Scaffolding

首先你可以在 GitLab 上可以看 repo scaffolding 的效果,dtm 为你创建了一个 Java Spring Boot 项目脚手架:

Repo scaffolding

Repo scaffolding
  • Pipeline

接着你可以在 Jenkins 上看到刚才 dtm 为你创建的 Pipeline:

Pipeline

Pipeline

如果你点开这个 test-job,就能看到它已经被触发了一次,执行结果如下:

Pipeline console

Pipeline console
  • 状态回写

然后你可以回到 GitLab,看一下 Jenkins Pipeline 的执行结果有没有被成功回写:

GitLab status

GitLab status
  • 检查镜像

通过 Jenkins 的日志你可以找到刚才 push 的镜像地址为 harbor.example.com:80/library/spring-demo:latest

Jenkins' logs

Jenkins' logs

// TODO(daniel-hutao): 补充 Harbor 截图

最后你可以通过 docker pull 下载该镜像:

Docker pulling

Docker pulling

配置详解

当前插件完整配置如下:

YAML
tools:
# name of the tool
- name: jenkins-pipeline
  # id of the tool instance
  instanceID: default
  # format: name.instanceID; If specified, dtm will make sure the dependency is applied first before handling this tool.
  dependsOn: [ ]
  # options for the plugin
  options:
    jenkins:
      # url is used to config jenkins url
      url: http://jenkins.example.com:8080
      # jenkins' user name
      user: admin
      # jenkins namespace in k8s cluster
      namespace: jenkins
      # restart jenkins if true for plugin install
      enableRestart: false
      # jenkins login password
      password: JENKINS_PASSWORD
      # if offline is true, jenkins-pipeline will not install jenkins plugins and share library
      # it will use a local Jenkinsfile for jenkins-pipeline
      offline: false
    scm:
      # scm common field
      branch: YOUR_REPO_BRANCH
      token: YOUR_REPO_SCM_TOKEN
      # you can directly use the url of repo (git@github.com/root/test-exmaple.git for example)
      url: YOUR_REPO_URL
      # or you can config detailed fields for this repo
      owner: YOUR_REPO_OWNER
      org: YOUR_REPO_ORG
      name: YOUR_REPO_NAME
      scmType: github
    pipeline:
      # jobName is jenkins's job name; <jobFolder/jobName> or <jobName>; e.g. jobs/test-job, test-job, jobs2/test-job
      jobName: test-job
      # configLocation is the location of Jenkinsfile, it can be remote or local address
      # if you don't config this field, devstream will use https://raw.githubusercontent.com/devstream-io/dtm-pipeline-templates/main/jenkins-pipeline/general/Jenkinsfile
      configLocation: https://raw.githubusercontent.com/devstream-io/devstream/main/staging/dtm-jenkins-pipeline-example/general/Jenkinsfile
      # language config is required
      language:
        framework: # support gin/flask/spring for now
        name: LANGUAGE # support go/java/nodejs/python for now
      imageRepo:
        # image repo URL for pulling/pushing
        url: http://harbor.example.com:80
        # image repo user name
        user: admin
        # image repo password
        password: YOUR_IMAGE_REPO_PASSWORD
      dingTalk:
        # dingtalk robot name
        name: YOUR_DINGTALK_ROBOT_NAME
        # dingtalk webhook
        webhook: https://oapi.dingtalk.com/robot/send?access_token=changemeByConfig
        # dingtalk securityType, we support "SECRET" and "KEY"
        securityType: YOUR_DINGTALK_SECRET_TYPE
        # dingtalk securityValue
        securityValue: YOUR_DINGTALK_SECRET_VALUE
      sonarqube:
        # sonarqube address
        url: http://sonar.example.com
        # sonarqube token
        token: YOUR_SONAR_TOKEN
        # sonarqube name in jenkins
        name: sonar_test

离线配置 (下文待进一步整理)

目前的 jenkins-pipeline 插件配置过程中会连接外部网络来安装插件和配置 pipeline 的共享库,为了支持在离线环境中使用 jenkins-pipeline 来创建和管理 jenkins 流水线,我们在 jenkins-pipeline 插件中提供了一个选项 offline 用于表示需要在离线环境下配置 jenkins-pipeline

Jenkins 插件

流水线正常运行需要依赖 jenkins 的几个插件。在离线环境中因为无法下载外部环境的插件,需要在 jenkins 预先安装好插件或者使用已经有插件的 jenkins 镜像,devstream 官方也提供了已预先安装好所有依赖的镜像 devstreamdev/jenkins:2.361.1-jdk11-dtm-0.2

依赖的插件如下: | 插件名 | 作用 | 备注 | |--------------|----------------|----------------------------| | dingding-notifications | 用于发送钉钉通知 | 如果需要使用钉钉通知,则需要安装此插件 | | github-branch-source | jenkins github 插件 | 如果代码仓库使用的是 github,则必须安装此插件 | | gitlab-plugin | jenkins gitlab 插件 | 如果代码仓库使用的是 gitlab,则必须安装此插件 | | sonar | sonar sanner 代码扫描插件 | 如果需要使用 sonarqube,则需要安装此插件 | | kubernetes | jenkins kubernetes 插件 | 用于 jenkins runner, 必须安装此插件 | | git | jenkins git 插件 | 用于代码的克隆和权限认证, 必须安装此插件 | | configuration-as-code | jenkins 配置即代码插件 | 用于 devstream 配置 jenkins 的全局配置, 必须安装此插件|

共享库

jenkins 中 devstream 默认使用共享库来管理 jenkins 流水线的共用代码,在离线环境中因为无法连接共享库,所以 devstream 提供了单独的 Jenkinfile 配置

app 示例配置

YAML
apps:
- name: test
  spec:
    language: java
    framework: springboot
  repo:
    url: gitlab.com/root/test.git
    branch: main
    token: [[ env GITLAB_TOKEN ]]
  repoTemplate:
    url: https://github.com/devstream-io/dtm-repo-scaffolding-java-springboot.git
  ci:
  - type: template
    templateName: ci-pipeline

pipelineTemplates:
- name: ci-pipeline
  type: jenkins-pipeline
  options:
    branch: main
    jenkins:
      url: jenkins.com
      user: admin
      offline: true # 在此处设置 offline 为 true, 即开启该 jenkins-pipeline 的离线模式
    imageRepo:
      user: repoUser
      password: [[ env IMAGE_REPO_PASSWORD ]]

使用该配置可得到以下输出:

Text Only
2022-12-02 19:51:52 ℹ [INFO]  Apply started.
2022-12-02 19:51:52 ℹ [INFO]  Using local backend. State file: devstream-app.state.
2022-12-02 19:51:52 ℹ [INFO]  Tool (repo-scaffolding/test) found in config but doesn't exist in the state, will be created.
2022-12-02 19:51:52 ℹ [INFO]  Tool (jenkins-pipeline/test) found in config but doesn't exist in the state, will be created.
2022-12-02 19:51:52 ℹ [INFO]  Start executing the plan.
2022-12-02 19:51:52 ℹ [INFO]  Changes count: 2.
2022-12-02 19:51:52 ℹ [INFO]  -------------------- [  Processing progress: 1/2.  ] --------------------
2022-12-02 19:51:52 ℹ [INFO]  Processing: (repo-scaffolding/test) -> Create ...
2022-12-02 19:51:52 ℹ [INFO]  github start to download repoTemplate...
2022-12-02 19:51:57 ✔ [SUCCESS]  Tool (repo-scaffolding/test) Create done.
2022-12-02 19:51:57 ℹ [INFO]  -------------------- [  Processing progress: 2/2.  ] --------------------
2022-12-02 19:51:57 ℹ [INFO]  Processing: (jenkins-pipeline/test) -> Create ...
2022-12-02 19:51:58 ℹ [INFO]  jenkins plugin imageRepo start config...
2022-12-02 19:51:58 ⚠ [WARN]  jenkins gitlab ssh key not config, private repo can't be clone
2022-12-02 19:52:00 ℹ [INFO]  jenkins start config casc...
2022-12-02 19:52:07 ✔ [SUCCESS]  Tool (jenkins-pipeline/test) Create done.
2022-12-02 19:52:07 ℹ [INFO]  -------------------- [  Processing done.  ] --------------------
2022-12-02 19:52:07 ✔ [SUCCESS]  All plugins applied successfully.
2022-12-02 19:52:07 ✔ [SUCCESS]  Apply finished.