Deploying OpenSky with Fabric
At OpenSky we use Fabric to deploy new versions of software to our servers. We deploy dozens of times a day to our testing environments, and do daily deploys to production.
Our production web nodes are split in to two groups, group1 and group2. It is setup that way so we can easily pull out a group of web nodes from the load balancer for maintenance without disrupting the site.
In this post I will take you through a hotfix scenario and the steps we take to deploy to production.
The Scenario
Imagine we just released v3.0.0 to production and we discover a critical bug that must be hotfixed.
First thing we need to do is create a hotfix branch. We use gitflow to assist with streamlining this process. I won’t talk too much about it here so I will assume you already know what it is.
Create the hotfix:
git flow hotfix start 3.0.1
Modify the opensky/config/version.ini file and bump the version number:
[parameters]
opensky.version = 3.0.1
Add the changed file, commit it and push up the hotfix:
git add opensky/config/version.ini
git commit -m"Bump version to 3.0.1"
git push origin hotfix/3.0.1
Another developer who is responsible for fixing the bug will create a new branch based off of hotfix/3.0.1 where the fix will be made:
git fetch
git checkout -b fix-the-bug origin/hotfix/3.0.1
The developer makes some changes and pushes up the new branch:
git add src/changed/file
git commit -m"Fixed nasty bug"
git push origin fix-the-bug
We use GitHub pull requests for all of our code changes to be as transparent as possible and maintain a high level of peer code review. The developer would create a pull request for the fix-the-bug branch and ask for a team mate to review. We have a special bot named @pr-nightmare that runs our tests against the branch to ensure stability before it is merged. Once the branch gets a +1 from @pr-nightmare the team mate can merge the branch in to hotfix/3.0.1.
Once it is merged we are ready to finish the hotfix:
git pull origin hotfix/3.0.1
git flow hotfix finish 3.0.1
The above command will merge hotfix/3.0.1 in to production and develop and create a new tag named v3.0.1 that can be deployed to production.
Now push the finished hotfix up to git:
git push origin develop
git push origin production
git push --tags
We are all set and ready to go to production with the v3.0.1 tag using fabric.
First thing we need to do is pull out a group of nodes from the load balancer so that we can deploy v3.0.1 to it. We will pull out group1 and leave group2 live:
fab prod proxy.group2
Now group2 is live and group1 is not receiving any traffic so we can deploy to it:
fab prod:out deploy:stable
The above command automatically determines what the latest stable tag to deploy is. In this case it will deploy v3.0.1.
Once that is done we can flip group1 live and pull out group2:
fab prod proxy.flip
Now group1 is live with the hotfix and group2 is out of rotation. To finish we run the same command as before and deploy the hotfix to group2 as well:
fab prod:out deploy:stable
We can push both groups live again and we are done:
fab prod proxy.all
The process could be even more streamlined and we’re actively working to remove steps and make it even easier to deploy to production!
