An app in DevStream corresponds to a real-world application, and the app represents the whole software development lifecycle of that app, including source code management, code scaffolding, CI/CD (and their pipelines).
Using "App", you can easily create these for an application.
There are situations where you need to define multiple DevOps tools for an application/microservice. For example, for a web-app typed microservice, you might need the following:
- source code management, code repo scaffolding
- continuous integration (the installation of the DevOps tool, the creation of the CI pipeline)
- continuous deployment (the installation of the DevOps tool, the creation of the CD pipeline)
If you are managing more than one application/microservice (chances are, you will be managing more than one application in the real world), the configuration of DevStream can be quite long, hard to read and hard to manage if you are only using "Tools".
In order to solve this problem, DevStream provides another concept that is "App". You can easily define all DevOps tools and pipelines for an App with a couple of lines of YAML config, making the config file much easier to read and manage.
In essence, "App" will be converted to "Tool", which you do not have to worry about at all; let DevStream handle that.
pipelineTemplates define CI/CD pipelines, so that they can be referred to and shared by different DevStream Apps, reducing the length of the config file to the next level.
In the config, there is a
apps section, which is a list, with each element having the following keys:
- name: the name of the app, unique
- spec: application-specific information
- repo: info about the code repository
- repoTemplate: optional, same structure as "repo". If empty, DevStream will create/scaffold a repository from scratch.
- ci: optional, a list of CI pipelines, each element can have the following keys:
- type: the value can be a
templateor the name of a plugin
- templateName: optional, if type is
template, it defines which pipelineTemplate to use
- vars: optional, variables to be passed to the template. Only works when type is
- options: optional
- if type is the name of a plugin, the options are the options of that plugin
- if type is
template, the options here will override the ones in the template. Use full path to override, for example,
- type: the value can be a
- cd: like
ci, but stands for the list of CD pipelines. DevStream will execute CI first before CD
Defined in the
pipelineTemplates of the config, it's a list, with each element having the following keys:
- name: unique name of the pipelineTemplate, unique
- type: corresponds to a plugin's name
- options: options for that plugin
2.3 Local Variables¶
DevStream has a "var" section in the config, serving as global variables that can be referred to by all Tools and Apps.
Sometimes, however, we'd like to use the same DevOps tool with minor differences. For example, except the name of the project, everything else is different.
In this case, we can define a pipelineTemplate with a local variable, and when referring to it, we can pass different values to it:
apps: - name: my-app spec: language: java framework: springboot repo: url: https://github.com/testUser/testApp.git branch: main ci: - type: github-actions # use a plugin directly without defining pipelineTemplates cd: - type: template # use a pipelineTemplate templateName: my-cd-template # corresponds to the name of the pipelineTemplate vars: appName: my-app # a local variable passed to the pipelineTemplate pipelineTemplates: cd: - name: my-cd-template type: argocdapp options: app: name: [[ appName ]] # a local variable, passed to when referring to the template namespace: argocd destination: server: https://kubernetes.default.svc namespace: default source: valuefile: values.yaml path: charts/[[ appName ]]
3 A Demo of the Whole Config¶
A whole config for an App:
apps: - name: testApp # name of the app spec: # app-specific info language: java # programming language of the app framework: springboot # framework of the app repo: # repository-related info for the app url: https://github.com/testUser/testApp.git branch: main repoTemplate: # optional, used for repository bootstrapping/scaffolding. If not empty, a repo will be created with scaffolding code url: https://github.com/devstream-io/dtm-repo-scaffolding-java-springboot.git vars: imageRepoOwner: repoOwner # variables used for repoTemplate ci: # CI pipelines, here we use github-actions - type: github-actions - name: testApp2 spec: language: go framework: gin repo: # repository-related info for the app owner: test_user type: github branch: main repoTemplate: # optional, used for repository bootstrapping/scaffolding. If not empty, a repo will be created with scaffolding code org: devstream-io name: dtm-repo-scaffolding-java-springboot type: github ci: # CI pipelines, here we use github-actions - type: github-actions options: imageRepo: owner: repoOwner # override the plugin's options. Must use full YAML path. cd: # CD pipelines, here we use argocd - type: argocdapp
If we apply this config, DevStream will create two repositories in GitHub, with scaffolding code provided by DevStream SpringBoot. App
testApp will trigger CI in GitHub Actions upon each commit, and App
testApp2 will trigger build/push in GitHub Actions upon commit, and deploy using Argo CD.
repoTemplate in the Config represent a code repository. You can define it with a single URL or a few key/values:
two ways to configure code repo
repo: url: firstname.lastname@example.org:root/myapps.git # repo URL, supports both git and https apiURL: https://gitlab.example.com # not mandatory, if using gitlab and the URL protocol is git, here can be the GitLab API URL branch: "" # not mandatory, defaults to main for GitHub and master for GitLab
This example shows that we use GitLab
email@example.com:root/myapps.git for code clone, and DevStream uses
https://gitlab.example.com to access GitLab API. Default branch is master.
repo: org: "" # only mandatory for GitHub organization owner："test_user" # if the repo belongs to a person. If the repo belongs to an org, use the org above. name: "" # optional, defaults to the name of the app baseURL: https://gitlab.example.com # optional. If GitLab, here we can put the GitLab domain. branch: master # not mandatory, defaults to main for GitHub and master for GitLab type: gitlab # mandatory, either gitlab or github
This example shows that we use GitLab
https://gitlab.example.com, repo name is the app name, belongs to owner
test_user, with the default branch being "master".
CI section in the config supports 4 types at the moment:
template means to use a pipelineTemplate; and the other three types correspond to GitHub Actions, GitLab CI, and Jenkins, respectively.
ci: - type: jenkins-pipieline # type of the CI options: # options for CI. If empty, CI will only run unit test. jenkins: # config for jenkins url: jenkins.exmaple.com # jenkins URL user: admin # jenkins user imageRepo: # docker image repo to be pushed to. If set, Ci will push the image after build. url: http://harbor.example.com # image repo URL. Defaults to dockerhub. owner: admin # image repo owner dingTalk: # dingtalk notification settings. If set, CI result will be pushed to dingtalk. name: dingTalk webhook: https://oapi.dingtalk.com/robot/send?access_token=changemeByConfig # callback URL for dingtalk. securityType: SECRET # use secret to encrypt dingtalk message securityValue: SECRETDATA # dingtalk secret encryption data sonarqube: # sonarqube config. If set, CI will test and execute sonarqube scan. url: http://sonar.example.com # sonarqube URL token: YOUR_SONAR_TOKEN # soanrqube token name: sonar_test
The config above will trigger unit test and sonarqube code scan upon commit, then a Docker image will be built and pushed to dockerhub, and the result of the CI will be pushed to dingtalk.
If the same pipeline is required for multiple apps, the config can be long and redundant. So, DevStream provides the
template type to share similar settings for diffrent Apps. Detailed example:
apps: - name: javaProject1 spec: language: java framework: springboot repo: owner: testUser type: github repoTemplate: url: https://github.com/devstream-io/dtm-repo-scaffolding-java-springboot.git ci: - type: template # use a pipelineTemplate templateName: ci-pipeline # name of the pipelineTemplate vars: dingdingAccessToken: tokenForProject1 # variables for the pipelineTemplate dingdingSecretValue: secretValProject1 - name: javaProject2 spec: language: java framework: springboot repo: owner: testUser type: github repoTemplate: url: https://github.com/devstream-io/dtm-repo-scaffolding-java-springboot.git ci: - type: template # use a pipelineTemplate templateName: ci-pipeline # name of the pipelineTemplate vars: dingdingAccessToken: tokenForProject2 # variables for the pipelineTemplate dingdingSecretValue: secretValProject2 pipelineTemplates: # CI/CD pipeline templates - name: ci-pipeline # name of the pipelineTemplate type: jenkins-pipeline # type, supports jenkins-pipeline，github-actions and gitlab-ci at the moment options: # options, same as CI options jenkins: url: jenkins.exmaple.com user: admin imageRepo: url: http://harbor.example.com owner: admin dingTalk: name: dingTalk webhook: https://oapi.dingtalk.com/robot/send?access_token=[[ dingdingAccessToken ]] # local variable, passed to when referring to this template securityType: SECRET securityValue: [[ dingdingSecretValue ]] # local variable, passed to when referring to this template sonarqube: url: http://sonar.example.com token: sonar_token name: sonar_test
If we apply the above config, we will create two Jenkins pipelines for two apps, with the only difference being that the dingtalk notification will be sent to different groups.
At the moment, CD only supports
argocdapp. Argo CD itself can be deployed with a Tool, and
argocdapp is responsible for deploying the app in a Kubernetes cluster.
Detailed config example:
cd: - type: argocdapp options: app: name: hello # argocd app name namespace: argocd # argocd namespace destination: server: https://kubernetes.default.svc # Kubernetes cluster namespace: default # which namespace to deploy the app source: valuefile: values.yaml # helm values file path: charts/go-hello-http # helm chart path