What is Continuous Integration and Deployment, and why use it ?
I was first attracted by Travis CI because it was really linked to Github, and it was really popular in the devops world. Travis CI provides a Continuous Integration tool, that means that each time you push commits to your Github repository, it will trigger the creation of a Virtual Machine with your code, and then run tests on it. This is Continuous Integration.
With time, Travis became more powerful, and it is now possible to build and deploy your application too on a large choice of cloud-based PaaS like Google Cloud, Amazon AWS, Microsoft Azure, … We are talking about Continuous Deployment.
To sum up, Continuous Deployment simplifies our workflow : with only one push on a git repository, tests are runned, builds are processed and everything is deployed without any intervention.
Travis CI is cool, but it’s not self hosted, and possibilities of deploying your application are not very flexible. For example, I don’t use Amazon AWS or Google Cloud, so it is a bit of a mess. I could use the SSH plugin, but it doesn’t seem very simple to me. I prefered to use Drone, a self hosted solution with a lot of plugins too, and it supports Docker.
Docker ? Yes, Drone doesn’t create a Virtual Machine like Travis, but creates containers with your code inside. Seems very cool to me, because my bare metal server is not fully equiped for virtualization.
The test section of our workflow is never a really tricky problem, you can always find a good solution to test. The big deal is with deployment. How to deploy my application on my server automatically ?
I wanted to use Docker, because all my apps run inside containers. I could use an orchestration tool like Kubernetes to run a PaaS like Deis, but I thought it was really overkill for only a few apps and one machine. So I prefered to stay with Docker-compose. Hopefully Drone can build docker images, and then push them to a registry. My solution was here.
In a nutshell, my goal is to push my freshly build docker image of my app to my docker registry, then sftp some files like a
docker-compose.yml file, and finally ssh to the machine and
docker-compose up -d. That way, nothing other than production is on the production server : the docker image is built, and there are only files that are needed for production.
Run it !
Drone and Registry part
So, there is a service on my server with two containers : Drone (because there is a official image !) and a Docker registry :
## Drone io container drone: image: drone/drone:0.4 volumes: - /var/lib/drone - /var/run/docker.sock:/var/run/docker.sock env_file: - /.env-drone restart: always ## Docker registry registry: image: registry:2 volumes: - letsencrypt-certs:/certs - /htpasswd:/htpasswd environment: - REGISTRY_HTTP_TLS_CERTIFICATE=/certs/live/???/fullchain.pem - REGISTRY_HTTP_TLS_KEY=/certs/live/???/privkey.pem - REGISTRY_AUTH=htpasswd - REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm - REGISTRY_AUTH_HTPASSWD_PATH=/htpasswd ports: - 5000:5000 restart: always
That was easy ! Some precisions are needed : the file
.env-drone contains some private keys needed for Drone. In fact, Drone uses your Github account to parameter webhooks and all you need to have a great CD. My file is like this :
The docker sock has to be provided too, because Drone will spawn containers each time it runs an integration. The registry is secured here, in order to limit access to my private images. I use HTTPS because the authentication with password needs it and the
letsencrypt-certs volume is my volume where my Let’s Encrypt certs are stored. The htpasswd file can be generated by
docker run --entrypoint htpasswd registry:2 -Bbn testuser testpassword > auth/htpasswd !
If Drone is running behind a proxy or whatever to provides a SSL connection, the SSL certs have to be valid (with Let’s Encrypt for example), or Github will not succeed to send webhooks each time a commit is done.
Every piece of my code is part of my github repository frostfire, you can see how it is done there.
The app part
Your server is now up, but you will need one more file in order to do a proper Continuous Deployment : the
.drone.yml file in each repository you want to add to Drone. This file has one goal : describe how the Integration and Deployment has to be done by Drone. You can see all the things you can do in the Drone doc, but there is a example :
build: image: node:latest commands: - npm install - npm run test - npm run package publish: docker: registry: creus.codewolf.fr:5000 username: fuego password: $$PASSWORD email: firstname.lastname@example.org repo: lupus/front-web tag: $$BRANCH file: Dockerfile insecure: false
As you can see, I pull the docker’s node image to put my code in, and then I run some npm commands I specified in my
package.json. In my repo, there is a
Dockerfile that takes the generated package (build with the last command) and puts it behind a Nginx server. Drone will publish this docker image just after building it to my registry, creus.codewolf.fr:5000, with authenfication. See Drone secrets to understand properly the
The deployment depends on how you’re doing it, but here’s mine :
deploy: sftp: host: codewolf.fr username: fuego password: $$PASSWORD destination_path: /home/fuego/Projets/lupus/lupus-front-web/ files: - cronos.conf - docker-compose.cronos.yml ssh: host: codewolf.fr user: fuego commands: - cd ~/Projets/lupus/lupus-front-web/ - docker pull creus.codewolf.fr:5000/lupus/front-web:$$BRANCH - docker-compose -f docker-compose.cronos.yml up -d - cp cronos.conf ~/Projets/frostfire/cronos/projects/lupus.conf - cd ~/Projets/frostfire/ - docker-compose up -d --build cronos-nginx
So to sum up, I sftp some files, like the docker-compose one, and a nginx configuration (it is this repository), then I ssh myself to my machine and I pull the freshly pushed image (I need to it manually because the tag is the same). I reload the docker-compose right after, and I copy the nginx configuration to Cronos, my Nginx server dedicated to projects. I rebuild this nginx with the docker-compose command and my project’s website is available with a Cronos url !
Here we go ! A full Continuous Integration and Deployment stack ready to be used by tons of projects ! Don’t hesitate to do this kind of things on your server or computer, it was very pleasant to do so. Feel free to ask me any questions :) If you want to see more of my server stack, I recommend you to read the How to organize your docker project article.
See ya !