Adding continuous integration with Jenkins pipeline and Github webhooks

Leave a reply

 

Pre requirements:

  • Windows, Linux or OSX machine running latest Java JDK or JRE.
  • Github account
  • You could fork the example project repo or you could use project of your own if you prefer.
  • if you want to set up a Webhook that will trigger automatic builds  you will need Jenkins to be accessible from outside your network. You will have to set up port forwarding in your rauther.

Example project repo:

branch-name:  
Click To Copy

 

 

Installing Jenkins

https://jenkins.io/download/

Jenkins come in two versions: Long-term Support (LTS) and Weekly releases. If you want stable version choose (LTS)

Jenkins also require Java so make sure that you have the appropriate version installed. By the time I’m writing this it requires

  • On MAC OS
    brew install jenkins-lts

https://jenkins.io/download/lts/macos/

  • On CentOS
    add Jenkins repo
    sudo rpm --import https://jenkins-ci.org/redhat/jenkins-ci.org.key
    then install it.
    yum install jenkins
    Then start the service
    brew services start jenkins-lts

Change default port (if needed, homebrew only)

Jenkins runs by default on port 8080, but I have another app running there so I had to change the default port.

Edit homebrew plist file as follows:
(replace 2.222.1 with the actual installed version)

/usr/local/Cellar/jenkins-lts/2.222.1/homebrew.mxcl.jenkins-lts.plist

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>homebrew.mxcl.jenkins-lts</string>
    <key>ProgramArguments</key>
    <array>
      <string>/usr/libexec/java_home</string>
      <string>-v</string>
      <string>1.8</string>
      <string>--exec</string>
      <string>java</string>
      <string>-Dmail.smtp.starttls.enable=true</string>
      <string>-jar</string>
      <string>/usr/local/opt/jenkins-lts/libexec/jenkins.war</string>
      <string>--httpListenAddress=0.0.0.0</string>
      <string>--httpPort=8082</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
  </dict>
</plist>

Line 18: change the port to 8082.
Line 17: Change  httpListenAddress from 127.0.0.1 to 0.0.0.0. This is necessary if you want to access Jenkins from Internet, outside of the internal network.

Now run the server as service.

brew services start jenkins-lts

Install the necessary plug-ins

Jenkins -> Manage Jenkinst -> Manage Plugins

Create First Pipeline

Create New Item

Pipeline ->Pipeline Script From SCM and put Git repository link.

Create Jenkins file with the pipeline steps

The whole pipeline should be wrapped in

pipeline {
}

A few words about pipeline syntax:

Agent is declared in the very beginning of the pipeline. This instructs Jenkins to allocate an executor (on a node) and workspace for the entire Pipeline.
An agent is typically a machine, or container, which connects to a Jenkins master and executes tasks when directed by the master.

Stage is part of Pipeline, and used for defining a conceptually distinct subset of the entire Pipeline, for example: “Build”, “Test”, and “Deploy”, which is used by many plugins to visualize or present Jenkins Pipeline status/progress.

Step A single task; fundamentally steps tell Jenkins what to do inside of a Pipeline or Project.

Full glossary could be found here

Let’s get started by creating pipeline file in the example project folder:

./jenkins/pr.groovy

pipeline {
  agent any
    
  tools {nodejs "SparkJS"}
    
  stages {
        
    stage('Cloning Git Repo') {
      steps {
        git 'https://github.com/ToniNichev/projects-sparkjs.git'
      }
    }
    stage('Install dependencies') {
      steps {
        echo '######################'              
        echo 'Building...'       
        echo '######################'                      
        sh '/usr/local/bin/yarn install'
      }
    }
     
    stage('Running Tests') {
      steps {
        echo '######################'              
        echo 'Running tests ...'          
        echo '######################'               
         sh '/usr/local/bin/yarn test'
      }
    }      
  }

  post { 
      always { 
          echo 'Starting server ...'
          sh '/usr/local/bin/yarn clean; /usr/local/bin/yarn build-prod; /usr/local/bin/yarn build-prod-ssr;'
          sh '/usr/local/bin/pm2 start ./server-build/server-bundle.js -f'
      }
  }  
}

what we just did:
– line 2, told Jenkins that it could run this pipeline for any agent. Agent basically allows you to specify where the task is to be executed. It could be Docker, Node or any agent.
– line 6, we defined our stages: Cloning Git Repo, Install dependencies, Running Tests
– line 32: finally after all stage script passed we defined the post script to run the server.

I’m using pm2 (a process manager and launcher) for running the app so if you don’t have it installed you should install it using npm or yarn.

npm install pm2@latest -g

or

yarn global add pm2

Running the pipeline task

So now everything is set up, let’s test the pipeline. Navigate to the pipeline and  from the vertical menu on the right select “build now”. If everything is good you should see a pipeline stages with progress bars filling out.

After the execution you could navigate to the log (build history in the right side -> select last job ->Console output)

There you could see a log of all stages executions including the snapshot tests

Test Suites: 2 passed, 2 total

Setting up Jenkins to listen to Github Webhook and trigger automatic builds on every commit

This is probably the best and the most tricky one to make it work. We are going to add Github Webhook that will make a post request to Jenkins every time when we push code change and this will trigger our pipeline and will rebuild the app and redeploy it. We are building so called continuous integration process. CI

Adding API key to the admin user.

Select the current (admin) user from the top right.

then on the left vertical menu choose “configure”.
Navigate to the “API Token” section and click “Add new token”

Important!!! Copy the token and save it somewhere safely because you won’t be able to see it again.

Navigate back to the pipeline that we created and click “configure” from the left vertical menu.
Scroll down to “Build Triggers” and check “Trigger builds remotely (e.g., from scripts)”

Paste the authentication token in the field and copy the example url below the text field where it says: “Use the following URL to trigger build remotely” We will need this to add it into Github webhook.

Click “save” on the bottom.

Setting up Github Webhook

Navigate to the example project in your Github space, select “settings” and “Webhooks”.

Click on “add webhook” and you will see this screen:

In the payload URL put the url that we copied from Jenkins -> Build Triggers above.

Important!!! Make sure that you replace ‘JENKINS_URL’ with the actual IP of the machine where Jenkins is running or the hostname if you set up one, and replace the token with the actual token that we generated for the ‘admin’ user. 

On the dropdown below “Content type” select “application/json”

Leave “Secret” below empty.

Next on “Which events would like to trigger this webhook” is up to you, but for simplicity I just left the default “Just push event”

Make sure that “Active” is checked and click “Add webhook”

At this point if you commit some changes to the example project and push them a webhook should fire and do a POST request to your jenkins instance, notifying it that there are code changes and triggering the pipeline process … but when I did this and looked at the response I saw: “403 No valid crumb was included in the request“

This simply means that Jenkins require another token to be sent in the headers, to make sure that only authorised cities (Github in this example) will trigger the pipeline process.

This is the most obscure and unclear part of setting up Webhooks. I google it for quite some time and figured out that there is no way to send custom header parameters (like Jenkins-crumb) from Github so the only option was to disable this security feature … which I think is fine since the pipeline is already protected with API key that we added.

Disabling CSRF Protection in Jenkins

The CSRF protection settings lives in “Manage Jenkins” under “Configure Global Security” but as it looks like the lates Jenkins releases don’t have an option to disable this, so the only alternative was to do it through the groovy script.

Go to Manage Jenkins -> Script Console
and paste the code below in the console.

import jenkins.model.Jenkins
def instance = Jenkins.instance
instance.setCrumbIssuer(null)

Click run. You will see empty result which is expected.

Go to the example project commit some change and push it again.

git commit . -m"Testing push webhook";git push

Navigate to Jenkins and you will observe that the new tack is queued in the “Build executor status” in the bottom left.

Test it

Let’s do it again by making some code changes and commit and push and observe how Jenkins will run the pipeline, test and deploy the project!

Cheers!

Leave a Reply