Episode 3 – Continuously Integrating, All The Time

It’s episode 3 of the Rent, Buy, Build Cloud Native podcast. This time, we’re talking about continuous integration and continuous delivery, for automating up the code and the platform.

It’s episode 3 of the Rent, Buy, Build Cloud Native podcast.  This time, we’re talking about continuous integration and continuous delivery, for automating up the code and the platform.

James Hunt
Welcome to Rent Buy Build. I’m James Hunt.
Brian Seguin
And I’m Brian Seguin.
James Hunt
Each week we’ll discuss the pieces and parts of Cloud-Native platforms and answer the question “Should you rent this, buy this, or build it yourself?” This week, we’re going to talk about CI/CD, continuous integration and continuous delivery.
Brian Seguin
You know, James, when I first heard the term CI/CD, I thought it was a military term. I thought it was something about, you know, continuously integrating new troops and continuously rotating them through some type of, of shifts. And then, you know, then perhaps I thought maybe it was a sales term, or you know,
James Hunt
you are you saying you’ve watched too much Battlestar Galactica and the CIC command Information Center? Give me a firing solution, Andrea, stat.
Brian Seguin
Yeah, exactly. Right. So but you know, what I, when I learned it was a, it was a software term, I had flashbacks to my college days as a Rails programmer. And I remember how angry customers would get every time we deployed something, because Sure enough, everything would fail like it, or at least it felt like everything would fail, there would be things QA missed, and all of these other things would happen, and there would just be so many people upset. So I’d like to start off by, you know, just kind of discussing CI/CD and what it means and why they are, they can be referred to as two different things.
James Hunt
Right? So yeah, you’ve probably heard the phrase works on my machine. That’s the problem that you’re talking about with we went to deploy something into production, we spent weeks working on it, tweaking it, doing bug fixes, doing QA testing, and then we went to deploy, it turns out, it doesn’t actually work. And, and that’s what, that’s what CI/CD is attempting to fix. By running the test suites and the QA suites and the deployments all the time, that continuous part of continuous integration, is you don’t need a person at the helm, saying, Okay, now we’re going to run the tests, you don’t do days, hours or weeks of development, and then run the tests, you run the tests on every commit, you run the integration, every time you go to cut a release, or every time someone wants to merge new feature code into a main branch. And but that the idea being that you can kind of control the quality and assess minimum standards for the whole code base by doing it in baby steps a little bit at a time.
Brian Seguin
And that is CI/CD together. So that’s continuous integration and continuous delivery. Continuous Delivery, if from my understanding is just delivering code continuously, but there still might be manual testing processes. Is that correct? Right. Yeah.
James Hunt
So continuous integration is regularly testing your code for internal consistency, do the unit tests still pass? Or did we break something in one part of this code while trying to fix something and another part that that’s the unit tests, it’s also integrating the component into a larger software product. So if you’re working on if you have lots of microservices, or if you have lots of modularized codebase in a monolith, you are going to have changes that may have wide reaching effects, and continuous integration on the integration. Testing end of the spectrum, lets you try the thing you’re doing in the context of all the rest of the things that’s going to run next to in production. So that’s all about testing. That is an augmentation to your other testing. In very few circumstances, it makes sense to say, well, we have ci, so I guess we never have to try and test this thing manually. Again, we never have to do usability testing, we never have to do QA testing. We never have to do browser acceptance, testing readiness, you’re still going to do all those things. But you can then focus at a higher level of testing because you’re not constantly testing, does the login button actually log me in, right? That’s what ci does. The continuous deployment or continuous delivery, the CD end of CI/CD, is to my mind more about infrastructure as code. And it’s taking the same tenants the same lessons learned from managing code bases, and applying it to operations work and platform work. So rather than say, we will make a whole bunch of changes to how we’re doing maybe TLS certs or upgrading Postgres versions, or, or even just deploying big sets of changes for an app into production. You’re doing these steps one at a time, constantly. Et CIE and and other i think i think Etsy the code is craft blog is where I first saw this in, in real world fullscale. Use Etsy being the online shop, handmade craft marketplace. What they do is every time a developer commits code That code gets deployed into production. And if you think about the fact that developers commit code regularly throughout the day, you might be deploying to prod two or three times a day as a single developer. And that’s scary. That was really scary when it first hit the scene because prod deployments were managed by this high priesthood of of release engineers and said, I need 15
Brian Seguin
layers of approval. James,
James Hunt
right, we got to go before the change approval board, the change approval board has to weigh this against all the other change control that’s going on at the time, are you there’s a freeze, like Marina Yeah, we’re in a blackout period, a code, freeze the challenge, freeze CIC D together, kind of kind of throw all that out the window and say, the best time to find out something is broken is right after you’ve made the change, not two weeks later, right? If you go out to production, and you have two weeks a whole Sprint’s worth of a team’s changes, they may interact in ways that weren’t tested because they weren’t developed together. And also you don’t know, which changes at fault, right? Was it?
Brian Seguin
Yeah, in a way as the CI CD process, you’re testing micro bits of code and production more frequently.
James Hunt
Yep. So the CI CD or the CI, part of CI/CD is testing software in a vacuum. And the CD is more testing and running the deployment. So it’s much testing but actually moving those changes into prod, quickly and regularly. So that you always have the latest and you’re only changing very small subsets of change.
Brian Seguin
That makes sense. So in a rent scenario, are we talking about what is the SaaS CI/CD solution?
James Hunt
Right, so And the interesting thing is, even though ci and CD are both at very different ends of what they do, and what they’re concerned with, they’re both built on top of automation, and workflow engines. Because, uh, see, if you look at a CI task, like, hey, somebody pushed a commit, let’s run the unit tests, that is an input, or an inciting event, right? A commit was made to master and a response task, take that, clone that down, run, you know, run your jest, unit tests, run your rake commands, run your make targets, whatever. And if that succeeds, then you’re you’re good. If not send an email and let somebody know or pop into slack and say, Hey, this broke the build. That’s a workflow engine. And the same thing happens with CD where somebody you know, an operator pushes a change to a manifest, or a change to a released package. And that’s an event that occurs, it causes the orchestration the workflow engine, to notice that do some task and response and either succeed in the deployment or fail in the deployment. And oftentimes, you can chain a bunch of these things together, right? So if we’re looking at the software testing side of CI/CD, the first step is always unit tests. Does the piece of software that we’re we’re looking at as a self contained unit, Does it still work? Does it do what this contract says it’s going to do? If this is a Twitter API, Does it still work with Twitter? Right?
Brian Seguin
So in the SaaS solutions, are there out of the box tests that you can kind of throw onto your your codebase?
James Hunt
Well, so the SaaS offering is really someone else’s running the workflow engine. And you’re bringing the pipelines? Because in no case, are you really ever going to walk into a rental solution with CI/CD, and somebody says, yep, and this is we know how to test your code, because nobody knows how to test your code except your developers who wrote it. But a lot of the SaaS offerings will provide language stacks, right environments where node j s is installed or a certain version of PHP is available. Travis does this thing where you can specify a matrix of bills. So I have a couple of go programs that are ci through Travis from a testing perspective, and I test against like the last three major releases of go. So right now that’s go 115 14, and 13, soon to be 116, and dropping off 113. And Travis manages all that. That’s all part of the SaaS offering. But this stuff also that all runs in the cloud. And that’s on someone else’s hardware, someone else’s software and someone else’s maintenance of that. And
Brian Seguin
right so in the SaaS case, you just have to worry about how you’re going to code the tests specific to your application. You throw them over the fence to your Travis your circle ci your GitHub actions your app there. Yep. And you say run these for me. I I don’t I don’t care how
James Hunt
that’s that makes a Cloud Foundry cone. If ever I’ve heard one. Yeah, and that’s the idea. So So the big players in the market and there’s a ton of them, right. There are literally dozens and dozens of these new ones are popping up. All the time. But the big names of Travis Travis CI, circle ci, app vejer is a big one. And GitHub actions, those are kind of what I would say the Big Four, I’m sure somebody is going to reach out and say, oh, what about my favorite one? That’s this, there are tons of these, right? do research, look into what’s available. But most of them all have the same basic structure, they usually will respond to a file in repo. For Travis, for example, it’s the dot Travis dot yamo file. And then that’s the only point of integration, you write your dot Travis dot yamo file, Travis picks up the rest and does it they all do usually containerized builds these days, where they’re pulling down Docker images to run things. Which means you have the ability in your unit tests to install packages, download binary files from the internet, do whatever you need to do to build up the environment in which to test so that it’s reproducible and so that you don’t have to maintain the fleets of these test servers yourself.
Brian Seguin
So are most of these licensed plus consumption models, almost your
James Hunt
all of them are consumption
Brian Seguin
models, okay?
James Hunt
Usually, most of these are a free tier plus some limitation for once you hit this point, you need to start paying. So GitHub actions, for example, is the one I’m I’m most familiar with, personally, because I’ve done a fair amount of work, both building extensions to it and using it as his GitHub actions. If you are an open source repository, it is free. It is free, quote forever. I don’t, I’m sure there’s an upper limit, if you actually tried to continuously continuously integrate where you never stopped running, I’m sure that they would, you know, you’d get a friendly email that says, hey, what’s going on? Can you please stop doing this, but they don’t charge you per the minutes, if you’re not an open source project, which on github.com means you are a private repo, you get a certain number of runner minutes. So you can run your pipelines for I think, 2000, maybe 3000 minutes per month, at which point they start charging you extra. So I don’t actually remember the math on the 784 hours, see, times 24. So there’s 18, almost 19,000 minutes in a month, you get about two to run, which is enough to run developer pushed changes, as long as you don’t have too many developers. And then they have plans for if you want, you know, 5000 or more minutes, above and beyond. Usually, it’s mid job runs.
Brian Seguin
So how does a SaaS solution for CI/CD integrate with on prem environments like data centers or colo facilities that you might have your code deployed on?
James Hunt
That’s a great question, right? Because all of these SaaS offerings, they are in the public cloud, right? They are on the public Internet, and the only thing you really have access to is stuff that you can access from the public Internet. But if you have on prem, or VP seed workloads, and you need to kind of reach inside of those environments, most of these SaaS solutions have what’s called a federated runner model or a federated agent model, where they’re gonna run the workflow somewhere, they’re gonna run your task somewhere, usually, it’s on a fleet of runners that they run, right, they have a bunch of hosts in Amazon or, or Google Cloud or somewhere, or even in their own data centers that they don’t work out to, but you can bring your own runner, and you can run that runner anywhere you want. Case in point, Stark, and Wayne has a vSphere lab where we run a bunch of integration tests for some of our deployment software. And we needed the ability to react to github.com repo pushes, to run the tests inside that highly segregated and walled off environment. So what we did is we put a GitHub actions runner on that vSphere cluster, registered it with our account and GitHub comm and said any of the GitHub actions jobs that need to do this, can use this runner. And so in that way, we get the benefit of SaaS for all of the feature set and the maintenance of the availability of the public components. And we’re on the hook only for the little bit of a build agent that we’re running.
Brian Seguin
So I’d assume if you’re going into a buy scenario, here, it’s some some type of CI/CD solution you want to run on prem in your data centers without these runners that you’re adding on from the SaaS solution. You’re going to be managing that more yourself. There might be you might be licensing that I would assume it wouldn’t be consumption based,
James Hunt
right and that’s the chief differentiate between buy and and rent. Not only you’re actually still running the runners in a buy scenario, but you’re also running the control plane. I have the CI CD solution. So if you’re using, for example, Jenkins is probably the most popular on prem CI/CD solution, whether it’s Jenkins or Jenkins x, that is a, they run the workflow engine on the server. And you can hook your own runners up to that. But you’re running all of that. It’s all on your infrastructure. And it’s however much usage, you can cram through that capacity that you’ve provisioned. So if you’ve got
Brian Seguin
Are you going to need a good are you gonna need a lot of extra infrastructure to run CI/CD in your lab,
James Hunt
it really depends on the scale. And what you’re doing with ci, CD CI/CD is a very elastic offering, or an elastic practice, because you can do something as simple as we’ve just run the unit tests. And if the unit tests pass, we let the PR go through. Or you can do all kinds of crazy stuff. I once worked with a customer who was using their own on prem CI/CD, to react to developer changes in a local Git repo, pull that down, build images, run tests, publish the images to a Docker registry, at which point another pipeline would pick up that the new Docker image was there, run that integration scenario to make sure that that candidate image would work well with all the other images that had been shipped. And if it did, would bless it for deployment into production, at which point, another pipeline would come in, pick that up and do the deployment, which was the Etsy Coda’s craft story. It wasn’t Etsy, but it was their story of a developer commits, and it ends up in production without anybody having to do anything. And for them, they had a very large footprint of CI/CD, that their CSV servers were not trivial, we spent a fair amount of time trying to improve the efficiency of their pipelines because we got such a value gain from reducing the infrastructure spent, even though it was an on prem vSphere.
Brian Seguin
So that is one of the kind of complications with the BI scenario, you’re taking your code, you’re writing your own tests for the code for the CI CD solution. And then you’re also managing the CI CD solution environment inside of your on prem environment alongside everything else that you have to manage inside of that on prem environment.
James Hunt
Right? That that is the crux of by that’s by in a nutshell. Or like what’s building, pay as you deploy as a new term, as you deploy episode, in contrast to the pay as you go model, the more servers you have to build out as your usage of CI/CD increases, or the cadence, right, if you’re if you want to do scheduled builds, and you want to do them quicker, more often, then you’re going to need more infrastructure. Interesting.
Brian Seguin
So what’s a build scenario? I, I feel like there’s there’s got to be not many
James Hunt
builders a lot harder to pin down. The obvious one is, well, you built your own continuous integration solution from first principles, you started with a language and you said, here’s how we’re going to watch get. And here’s how we’re going to watch an image registry. Here’s how we’re going to do the deployments to our EC to account to our AWS count as Easter. But in reality, I think what build actually is is accidental CI/CD. And this is a lot of legacy code. This is a lot of
Brian Seguin
they just tripped and fell into CI/CD themselves pretty
James Hunt
much. And it’s one of those things like if if you if you look at CI/CD as the automation of testing, integration and deployment tasks, these are things we’ve been doing in the industry for decades. You mentioned your rails phase in college, which everyone goes through a Rails phase. I love the rails programmers, by the way, that’s not a slight against rails. rails, I went through rails phase and when I was doing rails, what revolutionized the deployment for me was Capistrano. And what Capistrano did was make the deployments. predictable. You always deployed this way you had a rollback path, in a sense, that is CD. But all of the Capistrano deployments I ever did had custom, everything in them. And I didn’t have anything that would watch an SCM repository and then automatically do Capistrano deploys. And if I did, I would have written it myself. And to me, that’s what billed for CI/CD is, at some point in your developer life, you’ve half automated, some aspect of code base or production, deployment management. And the the build scenario is something that I think you’ve inherited, not so much that you set out to do. And I think most people that are in a build situation, are looking to either buy and unify, right? Oh, we’re gonna move everything. Jenkins that’ll take all this technical debt and essentially cancel it out. Because now we’ve got all the scripts are in a common place in a common format that everybody understands. That’s not, you know, Frank is the only one who understands the CI solution, it’d be everybody, or you’re looking to rent and do even more uniformity. Like, we’re just gonna move to GitHub, we’re gonna move to App there, it’s, you know, it takes no time at all to do the move. And then we’re not going to have to worry about either running the servers or maintaining this old code. So
Brian Seguin
if I’m going to understand the build scenario here, please correct me. It’s kind of an unconscious building of glue as one off use cases are determined before a actual CI/CD tool is selected as the uniform way of doing it,
James Hunt
right. It’s, it’s a, I need to solve this problem. And then you solve that problem. And then you go back to writing code for the business logic aspects of the application. And then something else pops up, oh, the deployment failed, because we didn’t run our database migrations, oh, yeah, we can, you know, put that into the make file, or the rails or a rake migrate DB task. And eventually you look back and you go, oh, wow, we’ve actually built our own ci solution. Everybody, this is awful, we need to fix this, the best way forward is to either rent something, you know, we’ll move to Travis CI, because easy, or bi and run it on prem.
Brian Seguin
So build might be an accidental way for companies to that happens on a company’s journey to a Cloud-Native type. environment. So when do you rent when you buy when you build in this scenario, like I said, so so build,
James Hunt
you don’t, don’t don’t build your own ci unless, unless you trip and fall into it you unless you have right, if you’ve already done it, at least, if you’ve already done it, take an inventory of what you have and how your processes play out today, and try and figure out where you might move those to get them out of your cars, you’re basically locked into your yourself as the vendor, you’re either going to buy or you’re going to rent I think, nine times out of 10, you’re going to rent. I say this in all capital letters always in parentheses, almost. If you have little to no legacy and a moderate run volume rental is going to be the best bang for your buck, it’s going to get you to ease into CI/CD at whatever pace makes sense to you. So maybe you start with you take one app that constantly has is responsible for production deployment failures, and you integrate unit tests on every commit. You’d be surprised how many bugs you will catch. And how many production outages you can avoid just doing the one repo is it’s preto principle, right? 80% 20% 80% of the problems are caused by 20% of the repos. So if you target those 20% of the repos with CI/CD in a rental scenario, you get really good value. If you are if you do have some legacy, or you have higher volume, like lots and lots of run volume, you’re gonna blow out past the 2000 or 3000, runner minutes a month on GitHub actions, then you might need to look at the by scenario, only if you can overcome the cost of holding the servers by not paying for the rental pays your usage, right, the idle point, you have to hit that and you have to hit it every month. If we’re talking strictly from a fin ops sense from a financial operation sense. If your solution that if your scenario, your situation is 100% public cloud, then SaaS offerings will be great. No matter which one you choose, they all integrate with github.com or Bitbucket, org, they can all talk to G ke and aka s and EC two and all those places. You can you can do terraform CFDs, all that stuff. If you’re hybrid rental is probably still the best option. If you have on premise deployments or on premise things. Let’s say for example, you have a Docker registry that you run inside your network in a data center, you can still use the cloud based CI/CD SaaS offerings, you just have to run that agent inside that data center.
Brian Seguin
Right. And for the bias scenario, we’re going back to the air gapped environments or the extremely limited environments or the extremely, extremely restricted networking environments that you know, you may not have all the access that you want to assess solution. There may be security concerns or maybe other concerns inside your organization. In those cases, you’re going to actually buy and license a solution that you’re going to implement yourself. You got Well, you’re going to at least you’re going to manage it yourself and and let it kind of build all of your testing processes around it and then use that.
James Hunt
Yep. Well, that’s it you’ve wasted another perfectly good half hour listening to rent buy build. Join us next week as we talked about source control. Management SCM get get ups and whether you should rent it, buy it or build it yourself. You can find all episodes of rent buy build online at rvb that Stark and wayne.com or wherever podcasts are sold