Continuous integration & delivery

How do we deliver software to users as quickly as possible? We have to focus on the build, test and release process. Fast delivery is also useful because it allows to verify if features and bugfixes; it also minimizes the time of feedback from the client. Releases need to be automated and frequent. Software doesn't have to be waiting uselessly until it is deployed.

The keys to an iterative development process are the prioritization of the most valuable features and parallelization of objectives by assigning them to the adequate number of developers.

Continuous integration is the practice where the members of a team integrate their work in the repository often. Each integration is verified by a tool that automatically performs tests and detect integration errors as soon as possible.

On the other hand, continuous delivery allows developing in such way that the code can be deployed in production at any time. For automatic deploys we use the term continuous deployment.

Benefits

• faster delivery
• reduce errors
• empowering teams
• lowering stress
• deployment flexibility

Continuous integration

The goal of continuous integration is that software is in a working state all the time. Without CI, your software must be broken until somebody proves it works.

When code is in the repository, we have to execute actions in the repository to ensure a stable quality of the code through the execution of tests and code analysis. This must be automated to ensure it is always executed.

Having the code directly integrated into the trunk, that main branch, on regular basis keeps the software working and integrated at all times. It reduces the possibility of large merge conflicts, ensures that integration problems are caught immediately and results in higher-quality software.

Requirements

• version control: everything must be in a single version control repository and in a working state. Many times it is not in a working state because nobody runs the whole application until a task is finished and nobody implements "invisible code" into production, leaving the whole test to the end.
• automated build: be able to build your software and run tests. It must be also tested and refactored when needed. If a build brakes, it is THE priority to fix it.
• agreement of the team: CI is a practice, it requires commitment to make small incremental changes by checking in regularly
• development workspace: code, configuration and data must be always updated locally. Tests must run on every commit. One sign of a good application architecture is that it allows the application to be run seamlessly in a development machine.
• having a comprehensive test suite: provides confidence that application is actually working. They can't take long (seconds / minutes magnitude) to execute or people will stop running them locally. Tests must pass before moving on to another task. These tests must run frequently as we need the trunk stable.
• be prepared to revert to previous version: if we can't fix the problem quickly, we should revert to the previous changeset in version control and remedy the problem locally. Assume that you might break something nd know what to do to revert them. You know previous version was good because it passd all tests.
• time-box fixing before reverting: set with the team an amount of time to try to fix an error during the commit. If you don't succeed in that time, revert to previous code version.
• don't comment failing tests: fix them

Code analysis

During the integration of code, code analysis must be run for ensuring code quality.

Managing code

Working on the mainline integrating directly into the master branch has its benefits:

• code is continuously integrated
• developers get each other's changes immediately
• detect earlier possible merge and integration problems For keeping mainline releasable, we need to run tests on it frequently, have it divided on components, make small changes and hide incomplete features

By working with featured branches, we ensure that the mainline is always releasable without interfering other developers:

• changes into the mainline must be merged frequently
• nobody should start a new branch until previous one is merged into mainline
• refactors must be tested and merged immediately

Having long-lived branches is bad because of the combinatorial problem of merging. If you have four branches, each of them will only be merging from mainline, not with each other. All four branches are diverging. I only takes two branches performing a refactoring in a tightly coupled codebase to bring the entire team to a halt when one of them merges. It bears repeating that branching is fundamentally antithetical to continuous integration. Even if you perform CI on every branch, it doesn't actually address the problem of integration, since you are not really integrating your branch. The nearest you can get to true CI is to have your system to merge every branch into a trunk that represents what mainline would look like if everybody we to merge, and run all automated tests against that. Such a merge would likely fail most of the time.

Essential practices

when do I have to create a PR?

Ship / show / ask

CI Software

Basically it polls code from the version control system when a commit is made, run the tests and notify the results. Most of them include a list of builds that have been run and check the reports. They are mainly configured through web interfaces. Visibility is an important benefit of using a CI server, allowing to constantly monitor the status of the current build.

Ensure local code quality with testing tools (phpunit, Jest) to ensure the reliability of the code we are integrating.

Continuous delivery

Principles of software delivery

• create a repeatable, reliable process
• automate almost everything
• keep everything in version control
• if it hurts, do it more frequently
• build quality in
• done means released
• everybody is responsible for the delivery process

Deploying risks

• break existing expected behaviours of the system
• introduce new unexpected behaviours of the system
• corrupt the data because of a scheme design mistake or failed migration
• introduce performance problem

We need safety measures to prevent this. We have continuous delivery to increase the safety and speed of changes.

To be able to build and release a version continuously we need:

• a suite of thorough and fast tests (unit, component and smoke functional)
• build the version
• second suite of tests: end to end and non-functional (performance, etc)
• if tests succeed, we can deploy with certainty of positive outcome. It can be deployed directly to production or to a user acceptance environment (pre-production).

Having the code validated, we proceed to prepare production environment, both application and infrastructure for the delivery. Configuration can be made through Ansible and architecture with Terraform.

We need to have automated they change in the four main components:

• code
• configuration
  • Anything that changes between deployments needs to be captured as configuration and not baked in when the application is packaged.
  • Keep the keys in the source code but the values somewhere else
  • configuration settings have a lifecycle completely different from the source code
  • it must be version-controlled, that way you can rollback at anytime
  • Do not fall into the antipattern Ultimate configurability, where we want everything to be configurable but we end falling on analysis paralysis, where we don't do anything.
• host environment
  • the key to manage them is to make their creation a fully automated process:
    • random pieces of infrastructure around
    • fixing can take many hours. It is better to rebuild it from scratch
    • Allows the creation of multiple copies for testing or escalation purposes.
  • good software has installers that can be run from command line
  • any change must be made through the automated process, nobody can make a change without using it.
• data

Release strategy

• Define roles in charge of the deployment, although every team member needs to be able to do it
• Have a documentation of the technology used and the pipeline
• list environments for different testings
• have a proper tool to monitor the application
• define a disaster recovery plan (AKA rollback)
• define a release plan:
  • steps required for deploying
  • deploy to stage environments
  • environment very similar to production
  • have an automated process that deploys releases into the environments
  • execute simple smoke tests
• Emergency fixes need to follow the same release strategy

Deployment pipeline practices

• Build once: The build that gets deployed into production should be exactly the same as those that went through the acceptance tests.
• Deploy the same way to every environment: it ensures that the build and deployment process is tested effectively. When you have testes the deployment process hundreds of times on many environments you can eliminate the deployment script as your source of error
• smoke test deployments: make sure that everything is up and running
• deploy into a copy of production: test it on an environment the most similar as possible to the production one in levels of infrastructure, OS, application stack and data.

Managing data

When integrating a new data structure and populating it, we can have this new data stored as an event in case we need a rollback. Application must be able to work with new and old database structures.

New functionalities by iterations

They can be developed incrementally and hidden from the users until it is fully functional.

If several developers routinely touch the same lines of code, it is a sign that the code is poorly structured and not enough encapsulated, being highly coupled. We must apply Demeter law, where objects must communicate directly only with those ones that affect them.

Antipatterns

• Deploy software manually:
  • extensive detailed documentation describing steps to take and how may go wrong.
  • manual testing to confirm it works
  • Frequent corrections to the release process
  • environments in a cluster that differ in configuration
  • Releases with unpredictable outcome Instead deploy should be pressing a button, everything packaged and automated. It should be the only way in which software is ever deployed.
• deploying to a production-like environment only after development is complete: Team responsible of making the deploy needs to have access to the version before the release process. Instead we have to integrate the testing, deployment and release activities into the development process.
• manual configuration management of production environment: some application changes need to be made directly on the environment. Instead the configuration should be applied from version control through an automated process.

Common problems

• The deploy process is very complicated due to huge testing time, because of new bugs and not having enough trust on current tests. This might be due to non-automated processes, not enough knowledge of the process or team is not collaborating with the deployment flow.
• The quality of the application is poor because of increasing bugs, developers fear new features due to low maintainability and increasing time of development. For soling that developers need to collaborate with testers, tests are no implemented, or test team has not enough knowledge.