A Tale of Two Deployments Part 2 - Why Agile Infrastructure Decisions Matter

This article is a follow-up to the blog post A Tale Of Two Deployments - Machine Images, Immutable Servers And Green/Blue Deployment , and is a 'prequel' sequel, in the best traditions of Hollywood. In the original, we heard the story of Ali and Sally.

Ali's team decided to push code to manually maintained servers, and had the worst day of their career during a live deployment, when everything that could go wrong, did go wrong. Sally's team, however, having implemented an immutable-server based delivery pipeline, was able to avoid the issues that plagued Ali and have a stress free deployment, even when everything that could go wrong had gone wrong.

The allure of pushing code to manually maintained servers (or "snowflake servers", because each one is different) is always there at the beginning of a new project. Tools like Capistrano (for deploying Ruby based web apps) have made it trivial to set this scheme up.

Ali's team believed that using a PaaS solution like Heroku will become very expensive later on, even though it is the least intensive way to host their app. If they tied their app to Heroku, it would be really hard to move to another hosting solution, they thought.

After all, they were an AGILE team - they were just starting out, and since they had not launched, and did not actually have any real users, there was not any need to worry about infrastructure right now, just developing features to please investors. They could think about this later, when it became a high priority issue. Like just before launch.

So, they did the easiest thing (without tying themselves to Heroku or another expensive PaaS) and brought up a server on AWS. They edited some config files on it, install some packages, and push their code to it every time they wished to deploy.

Part 1 of this article series already shows how this decision caused so many problems! Having to add another server to the cluster was problematic, as was fixing existing servers that went down. Deployment was an act of replacing running code with new code, a process that could be described as "push and pray". New dependencies had to be manually installed or upgraded, which caused issues with keeping them in sync across all servers, or when rolling them back along with the code.

Sally's team also wished to be agile when they were deciding on how to host their new app. They too wanted to avoid the exorbitant costs that PaaS solutions rack up as traffic increases.

However, they also were wise enough to realise that, with the right early decisions, they could move their app to different hosting solutions easily, and this meant they could choose what was good for them right now, without having to worry about what their infrastructure needs would be in the future.

They made sure their app was "12-factor compliant" from the beginning - it did not make any assumptions about where it was hosted, but instead conformed to a set of requirements which, if also followed by the hosting infrastructure, meant the app would run on it with little work. This way, the app was decoupled from its hosting infrastructure, making moving it to new infrastructures trivial in the future.

So they decided to host on Heroku for now, using Heroku's Pipelines feature, so they did not need to manage any infrastructure at all. Then, if the costs became too high, and they decided to move to a different hosting provider, or implement their own infrastructure solution, they could do so easily without having to make any changes to their app, as it was already agnostic to any underlying infrastructure.

The key points of a 12-factor app's hosting strategy are:

  • All build steps, like asset precompilation, gathering or generating any required files, installing dependencies, are done once, in an environment-agnostic manner. They should not require any connections to external services which vary per environment (like the main database), need any API keys, or need anything else that ties them to a particular environment. This results in a "deployment artifact" (like a machine image) that can be carried through different environments in a pipeline.
  • All settings should be injected into the app, for example by using environment variables, and not hardcoded in any files.
  • The app should output 1 log line per event to standard output, not to a file.
  • The app should never assume its server will be around for long - it could be taken down and replaced at any time!

The moral of this story is: NEVER just push code to manually managed servers! Whether you choose to use a PaaS hosting solution, create your own, or license an existing one, remember that moving a properly configured 12-factor app to a different solution is many times better than ever having to depend on the "push and pray" method of deployment. Then you, too, can live happily ever after (as far as your deployments go, anyway).

Asfand Qazi runs The DevOps Doctors, a DevOps consultancy. Click the button below to contact him for a free consultation if you need help with your infrastructure and deployment process.