Blue-green deployments on AWS

Before we do this, make sure that you have a service that you want to deploy. To keep things simple, I followed the spring boot tutorial on making a restful web service. It was quick and the app worked like a charm. As usual, I went a bit extra and made my app return a stubbed list of users. You don’t have to. Make sure you have a /healthcheck endpoint and another endpoint that you can test with. In my case, I have /users which returns a list of users.

All righty then. Lets get a high level overview of what things are and how they are going to work. But before we do that, lets go through a quick real-ish life scenario.

Say you have a service that you have deployed onto AWS. Now you have a newer version of that service that you’d like to test. Since you never know if something works without actually trying it out, normally, after exhaustive testing in staging and other environments, you’d deploy that service into production to all your users. But ah ha! That one guy in your team forgot that one test case which made it blow up which means every single user of yours is now seeing error pages everywhere. This is bad so you roll it back to the previous version. Doesn’t sound too bad yet but by the time you do this, you’d have lost a couple of hours in time which would translate into actual money lost to the company which could eventually make a dent in your end of the year bonus.

Continue reading

AWS cloud formation describe* permission error

So you know sometimes, its difficult to work with AWS cloud formation scripts. On surface, errors are seemingly random and unrelated to the script. This is what happened today. I wrote an awesome parameterised cloud formation script. I was quite proud of it, mainly because most of my parameters were typed. This mean that the parameters like, ec2_security_groups had a type of List<AWS::EC2::SecurityGroup::Id>. This not only makes it easier to work with that cloud formation script from the AWS console, but also makes it very easy to work with from a CI/CD pipeline as if those parameters are invalid, the script will fail instantly, instead of waiting for resources to deploy.

However, while doing this, I completely missed the fact that in order for cloud formation to validate your input parameters, it needs to look them up first. What IAM policy permission does a resource lookup need? describe!

For once, my IAM role had tightly restricted permissions and because of this I had to go on to expand them slightly to allow for describe permissions.

I kept getting this error about the role not having the describe permission and I kept wondering why it needed that. Well now you know too!