Before I can make a case for Immutable Deployable Artifacts (I’m going to use IDA for short), it is probably a good idea to define what I mean by that term.
Regardless of your technology stack, most systems follow a similar process for building deployment artifacts: get code, fetch dependencies, build code, putting together the resulting deployable artifact, whether it is .Net DLLs, .jar files, or docker images. These things are referred to as “deployable artifacts.”
To say that these are immutable, by definition, means that they cannot be changed. However, in this context, it means a little more than that: having immutable deployable artifacts means that these artifacts are built once, but deployed many times to progressive environments. In this manner, a single build artifact is deployed to QA, internally verified, promoted to staging, externally verified, and then promoted to production.
Separation of Concerns – Build vs. Deploy
First and foremost, we need to remember that build and deploy are uniquely separate concerns: Build is responsible for creating deployable artifacts, while deployment is responsible for the distribution and configuration of the artifacts in various environments.
Tools like Jenkins muddy those waters, because they allow us to integrate build and deploy into single actions. This is not a bad thing, but we need to make sure we maintain logical separation between the actions, even if their physical separation is muddied.
Gitflow
I won’t go into the merits and pain points of GitFlow here. It’s suffice to say that the majority of our teams utilized GitFlow as a branching strategy.
Gitflow and IDAs
The typical CI/CD implementations of GitFlow, particularly at my current company, map branches to environments (develop to QA, release to staging, master to production). To effect an update to an environment, a pull request is generated from one branch to another, and based on a commit to the branch. A build is then generated and deployed. This effectively means that the “build” you tested against is not the same as the build you are pushing to production. This gives development managers (current and former, such as myself) cold sweats. Why? In order to have accurate builds between branches, we rely very heavily on the following factors not changing:
- No changes were introduced to the “source branch” (master, in the case of production builds) outside of the merge request.
- No downstream dependencies changed. Some builds will pull the latest package from a particular repository (nuget, npm, etc), so we are relying that no new versions of a third party dependency have been published OR that the new version is compatible.
- Nothing has changed in the build process itself (including any changes to the build agent configuration).
Even at my best as a development manager, there is no way I could guarantee all three of those things with every build. These variables open up our products to unnecessary risks in deployment, as we are effectively deploying an untested build artifact directly into production.
Potential solutions
There is no universal best practice, but, we can draw some conclusions based on others.
Separate Branch Strategy from Deployment strategy
The first step, regardless of your branching strategy, is to separate the hard link between branch names and deployment environments. While it is certainly helpful to tag build artifacts with the branch from which they came, there is no hard and fast rule that “builds from develop always and only go to QA, never to stage or release.” If the build artifact(s) pass the necessary tests, they should be available to higher environments through promotion.
I’m not suggesting we shouldn’t limit branches which produce release candidate builds , but the notion that “production code MUST be built directly from master” is, quite frankly, dangerous.
If you are using GitFlow, one example of this would be to generate your build artifacts from ONLY your release and hotfix branches. There are some consequences that you need to be aware of with this, and they are outlined far better than i can by Ben Teese.
Simplify your branching strategy
Alternatively, if your project is smaller or your development cycles are short enough that you support fix-forward only, you may consider moving to the simpler feature branch workflow. In this branching mechanism, build artifacts come from your master branch, and feature branches are used only for feature development.
Change your deployment and promotion process
Once you have build artifacts being generated, it’s up to your deployment process to deploy and configure those artifacts. This can be as automated or as manually as your team chooses. In an ideal scenario, if all of your testing is 100% automated, then your deployment process could be as simple as deploying the generated artifacts, running all of your tests, and then promoting those artifacts to production if all your tests pass. Worst case, if your testing is 100% manual, you will need to define a process for testing and promotion of build artifacts. Octopus Deploy can be helpful in this area by creating build pipelines and allow for promotion through user interaction.
What about Continuous Integration?
There is some, well, confusion around what continuous integration is. Definitions may differ, but the basics of CI are that we should automate the build and test process so that we can verify our code state, ideally after every change. What this means to you and your team, though, may differ. Does your team have the desire, capacity, and ability to run build and test on every feature branch? Or is your team more concerned with build and test on the develop branch as an indicator of code health.
Generally speaking, CI is something that happens daily, as an automated process kicked off by developer commits. It happens BEFORE Continuous Deployment/Continuous Delivery, and is usually a way to short circuit the CD process to prevent us from deploying bad code.
The changes above should have no bearing on your CI process in that whatever build/test processes you do now should continue.
A note on Infrastructure as Code
It’s worth mentioning that IaC projects such as terraform deployment projects should have a direct correlation between branch and environment. Since the code is meant to define an environment and no build artifacts, that link is logical and necessary.
Leave a Reply