Episode 10: A Time To Scan

Security! It's on everyone's mind (or at least it should be), and nowhere is that truer than in the containers space. In this episode, Brian and James discuss image scanning, security postures, and we take bit of a detour through security theater to screen Fast & Furious 6: Container Drift.

Security! It’s on everyone’s mind (or at least it should be), and nowhere is that truer than in the containers space. In this episode, Brian and James discuss image scanning, security postures, and we take bit of a detour through security theater to screen Fast & Furious 6: Container Drift.

James Hunt
Hello, you’re listening to Rent, Buy, Build – the Cloud-Native podcast where we talk about the pieces and parts of cloud platforms and ask the question: “should you rent this, buy this, or build it yourself?” I’m your co-host, James Hunt
Brian Seguin
and I’m Brian Seguin.
James Hunt
And today we’re following up on our last episode about image registries by covering the topic of image security scanning. Brian, Brian has done a whole pile of research in preparation for this specific episode. So I’m gonna let him kind of talk about what would you learn Brian,
Brian Seguin
I love security scanning, it’s a fascinating thing to research and look into. We’ve worked with to the major security scanning companies in the past, and it’s It was really fun just learning about some of the vulnerabilities and the vulnerabilities that you’re surprised that happened in production code that, you know, from the 90s. Like, that’s, that’s the most shocking thing as you’re going through some of these things. But I think before we get into, you know, whether we run by or build the security scanning, let’s talk about what it is and why it’s important, and maybe some of the weird nuances surrounding it. So there’s really two main types of security scanning, there’s your operating system vulnerability scanning, where you’re, you’re, you’re scanning for the the OS for vulnerabilities. And then there’s language specific vulnerability scanning. And there’s two ways to do security scanning. One is your scan on your build image, and then another, as you scan during your runtime, both are important. And actually, all of those different methods are important. Especially, you know,
James Hunt
so I can scan my base image to find stuff, vulnerabilities that were put there by other people. And I can scan my software that I have written and I’m deploying to find the vulnerabilities that I accidentally included.
Brian Seguin
No, no, that’s not how it works, James, you’re only scanning that like the language packets that that you’re using.
James Hunt
Right, my–
Brian Seguin
not actually scanning your codes, vulnerability. I mean, I don’t think there’s any software out there that does scanned your codes vulnerability is there. I mean,
James Hunt
There’s static analysis tools that can look for, you know, problematic pattern behavior. But really, the software as it were, that finds vulnerabilities is human security researchers.
Brian Seguin
Does anybody do that?
James Hunt
A lot of people do that.
Brian Seguin
Oh, interesting,
James Hunt
especially if the vendors have good bug bounty programs. And aren’t, you know, jerks about the disclosure process, Google has a whole thing called Project Zero, where they just find vulnerabilities and things and report them upstream. But that’s actually where your CVEs come from, right from from the hard work of security researchers. No- what I was getting at is there’s two areas in which we’re scanning for known vulnerabilities, the operating system packages, you know, the Debian stuff or the the RPMs that were installed in your base Ubuntu or base, CentOS image, and the vulnerabilities in the dependencies of the code that you are writing. So not the code you wrote, but the code you’re using, so that you don’t have to write it your NPM modules, your Ruby gems, your, your CPAN modules, those kinds of things that that sum it up?
Brian Seguin
Yes, it does.
James Hunt
And then what was the thing about live versus static?
Brian Seguin
Yeah, so you can scan your image prior to building or upon building and then you can also scan your image while it’s running in, in production, for example. And it’s important to scan your image both places in production, mostly because you have something called container creep. Right?
James Hunt
Are we talking about container drift?
Brian Seguin
Oh, container drift; container creep, that I think that’s my brother.
James Hunt
Container Drift – the 6th Fast and the Furious movie, I believe? Oh, yeah. No. So container drift is interesting because container drift is the tendency of software to or the bits in a container image, as the container itself is running to change. So anytime you modify the file system of a running container, we consider that drift. This can be usually this is a sign, especially in the modern world with immutable containers or immutable infrastructure. Even if you’re not enforcing read only file systems. We don’t log into a container to update code, right in Kubernetes. We don’t take the existing pod and push new Ruby code into it, we build another container to replace the previous version. So that immutable infrastructure means that container drift is usually a sign of compromise. It’s an IOC or an indicator of compromise. And that was easily you know, you don’t even have to scan for that if you’re using immutable containers, or read only containers and Kubernetes, where the the software can’t modify the file system. So an example of container drift, that would be an indicator of compromise is: all of a sudden, the kubectl binary shows up in /usr/bin that that wasn’t there in the initial image that we scanned. In in the myopic view, you could say, “well, we need to scan that cube CTL binary and see if it has any vulnerabilities.” But the bigger question is, “who put the kubectl binary there? And why did they do it? And what are they doing?” because that, that reeks of an attack that that’s a compromise, someone has somehow convinced the container to do something it wasn’t designed to do. The other part, it’s not so much drift. But the other part of scanning images is not just to scan them at build and ingestion time, so not just as they get pushed into your registry, which if you remember back to last episode, we strongly encouraged you to buy and run your own. So that you can do things like as Docker images get pushed into the registry, we scan them, and if they fail scans, we don’t take them. That keeps your day-of-push, stuff secure. But vulnerabilities are being found all the time
Brian Seguin
Right.
James Hunt
Human researchers are constantly looking for and trying to exploit bugs and buffer overruns and all kinds of stuff. And they’re finding things in software that you thought was secure. So just because there’s no vulnerability in for example, nginx. today that you’re finding when you push your custom build of nginx images up to your registry, doesn’t mean that next week, that version isn’t going to be now vulnerable to a newly found flaw. So you have to keep re scanning your images regularly. As the database sources update, as new vulnerabilities are found, you need to reassess your existing inventory of images. So that’s, that’s a third, I guess.
Brian Seguin
Yeah, I think it’s important to point out a couple of, you know, best practices here, when it comes to running your containers that prevents this, especially if you’re not scanning. If you’re not doing both types of scanning, scanning on the image, build side and scanning, running containers. And the best practices are, you know, you want to read a play often, right, you don’t want to have your container stagnant out there. So you’re actually scanning every time you read a play in that case. The other thing here is Don’t you know, is the implementation of getups. It’s, it’s where developers don’t actually push their containers from local to, you know, production or non prod. But they’re actually pushing their code into repos, which is then getting picked up by pipelines getting scanned, and then going into production. The biggest reason why that’s important is, you know, what I called container creep, which is where it’s my that’s my interpretation of developers, adding extra things locally into their container image that they don’t actually need, and pushing that into production. And that’s one of the biggest things for vulnerabilities is, a lot of times, people without knowing put extra code into their container that they don’t actually need, and actually have vulnerabilities in it. So when the seabees come back, you can actually look through them and say, we can actually delete all of these different things inside of this image, because our source code is not using it. And it’s important to go through that process on an ongoing basis. Because what’ll end up happening with employee turnover, people won’t want to change the image in the future, and they’re going to get a little bit nervous about that. And that’s where you get a lot of really old and outdated images running in production nowadays that aren’t getting scanned and actually have a bunch of vulnerabilities in them.
James Hunt
Right, the normal use case there is I started with the Ubuntu base image in 20.04. It’s an LTS, and that comes with a set of base packages. And maybe I add, maybe I add a couple of packages to make my application do something but they pull in additional libraries that then have vulnerabilities. You call it container creep, I believe I’ve always called a container bloat either way. Yes, and being able to see, hey, here’s all the stuff that is vulnerable that we know of right that has a listed known vulnerability upstream in either NIST or or mitre or any of the CVE databases is helpful. Because as you mentioned, there’s a resistance to changing the base image of an application container once it’s working. Unfortunately, that’s also a best practice to do that right upgrade if you’re if you are using an operating system image. Like Ubuntu or CentOS, or even Alpine, you do need to bump those base images regularly. The question is, do you actually need all that stuff? Or was that just easy, right? So when we when we talk remediation, one of the easiest ways, when I go through and do these security scans for people, the easiest thing that we have found to remediate is to change the base image, do you actually need Ubuntu? Or was that just the easiest way to get GCC installed so that you could install native extensions for Ruby. And then there’s a couple of other things, you know, multistage builds are helpful here. Because your build dependencies evaporate in a multistage build. So if you’re compiling, for example, native Perl extensions, or native Ruby extensions, you can get the compiler and all the build tools in the first stage of the build image, and then copy over the compiled assets into the final image and not take the build tools. So that’s a whole bunch of fewer packages, which, you know, contributes to cutting down on your image size and also contributes to your security posture. Right. So when we talk about rental, so now, I guess we should instead talk about how these things work. So
Brian Seguin
Oh, yeah, that was a so how these things work.
James Hunt
So how do these things work, Brian?
Brian Seguin
I don’t know.
James Hunt
(laughter)
Brian Seguin
I mean, my understanding of how these things work is there’s people out there that are constantly looking for vulnerabilities, and then they publish them to different vulnerability projects. And there’s different vulnerability sources that log what the vulnerabilities are. And then basically, you’re looking at your images, you’re looking at your code, and you’re, you’re going back, and you’re comparing it to see if there’s any compatible versions between what you have deployed and what has been registered as a CVE, for any of any of those open source, you know, vulnerability sources. The interesting thing that I learned is that not every scanning solution out there or not every company is actually using multiple different scanning sources, some might be using one or another. But it also might depend on you know, the type of code you’re deploying or the type of infrastructure, you’re not infrastructure.
James Hunt
The type of OS platform
Brian Seguin
type of thing you it also might depend on the type of OS platform that you’re actually using.
James Hunt
Right. So as a case in point, the Ubuntu security, there’s a USC Vulnerability Database for here’s known problems with Ubuntu packages. Because the platforms themselves, the distributions are always patching their packages. So if you install the same version of postfix, on CentOS, as you do on Ubuntu, you might not actually get the same set of patches, because they’re different patch sets. So the distributions are maintaining their own, here’s what we’re our version of postfix has been patched to not be vulnerable to this CVE, or that CVE. And those are generally included as the relationships between the scanning solution provider and those distributions are hammered out. So canonical, for instance, provides that vulnerability information to snick, and Aqua and all of these things. But the mechanics of how this scan happen, I think are less important than how the integration works. I noticed the other day and I’m going to dig into this for probably a future blog post. But I did a Docker build with on a newer version of Docker, and it already has snick built in. So you can actually scan while or after you’ve built it before you do a manual push to Docker Hub. But what we’re talking about when we talk about security scanning and images is really an add on to the registry. And that is because the scanning solution needs to be able to scan not just on ingestion, right? Not just when we push the image, but also needs to be able to enumerate all of the images in your inventory and scan those. So for that it needs a registry. And the the cloud providers are generally offering scanning solutions as part of their hosted registries. So if you’re renting your image registry, you’re also most likely going to be renting the security scanning because it’s included. And with the exception, I think of Google. And Brian, you have way more information on that than I do with the exception of maybe Google they’re all free.
Brian Seguin
Yeah, from a rental standpoint, just to dive in there. You’re really looking at–
James Hunt
Speaking of segues…
Brian Seguin
From a rental standpoint, you’re really looking at Aqua, Snyk; those are rental SAS solutions. But also, as James was saying, the major cloud providers, Google, Azure, Amazon, they all have their own rental format. But basically, it’s tied to their, their registry offering. And so what you’re really actually doing for most of them is paying per consumption. That’s mainly the case with Azure that your Azure defender and same with Amazon, it’s you’re paying for the consumption. But the services for the most part, free
James Hunt
The consumption is the consumption of registry,
Brian Seguin
Right
James Hunt
Whether you’re scanning or not, you’re paying that,
Brian Seguin
Right, Google is the odd ball, which has 26 cents per scan container image for their scanning option.
James Hunt
That’s an interesting take, like as far as positioning of, we’re going to charge you for the security. I wonder what they’re using, they don’t have those details available publicly. I don’t think Amazon ECR does use Claire, which we’ll talk about shortly as one of our buy options. But yeah, these are usually included in the registry, because they already have access to the data, you’re already paying to store and transfer the data so they can hook something internal up to that registry and scan it for you. So if you remember back to last episode, we talked about whether you should rent buy or build the image registry itself. And I think the recommendation we came up with was, in most cases, you should be buying it because it gives you the option to add these types of things on as you see fit. Not just as the providers use fit, so Google decides to charge you per scan. And invariably, I feel that that will end in a disincentivizing of scanning things inside of Google. Whereas with ECR from Amazon and Azure with Azure defender, as part of the the offering just part and parcel, we are going to see more scanning done, because it’s free, which will lead to a more secure and more, hopefully better internet, as people use these scan things and find things. But above and beyond just the scanning solution. The by recommendation for image registry allows you to not just implement the scanning solution you’re going to buy or rent, but also allows you to provide the controls and the gatekeeping mechanisms to ensure that just because you scanned an image, it doesn’t stop there, right? You want to halt the deployment pipeline if a certain number of CVS are found or a certain severity of CVE is found. And I want to take this,
Brian Seguin
that’s a really good point.
James Hunt
And I want to take this time to talk a little bit about security theater. Because it with All Things security, it’s very easy to fall into the hole of we’re doing this because it makes us feel secure. Not because it actually improves our security posture. Scanning images can very quickly become while we scanned all the images. And then what did you do? Well, then we push them all to prod. Okay, what happens if there’s a CVE? Well, we, we tell the developers, they need to fix that and we push it to prod like you’re not, you’re not creasing your security posture doing this
Brian Seguin
On the flip side of the coin, you can actually implement procedures where the developer doesn’t push to prod they push to the Git repo and then their, their push to prod can be blocked if it has security vulnerabilities. But that can also be an overreaction, because not all the vulnerabilities are equal. And there’s actually different ways of measuring how bad a vulnerability is. And there’s whole scales for that,
James Hunt
So the CVE databases usually come in, you know, low, medium high. But really the the take with that is to not overreact and say if there’s any CVE ease at all, nobody’s going to prod because there are vulnerabilities that are just technically vulnerabilities. But they are so difficult to take advantage of that you may not, may not fit into your risk portfolio of things you care about. Things like local file buffer overflows in libjpeg, just because you installed ImageMagick, so that you could identify types of images as they’re uploaded, doesn’t mean that you’re going to run afoul of some of these. Some of them have very elaborate conditions that have to be met in order to be usable by the attacker and they have a limited blast radius. So it’s important when we talk about security and image scanning that you have the, the the wherewithal to go through what you’re doing and make sure that it’s helping your security not hindering your agility. And that’s not an easy recommendation. I can’t give you a piece of software that if you just install this, and then set the make it work equals true in the config and you’ll be off to the races and you’ll be happy. So in the BI space, we’re talking about I’m going to talk about two specific open source projects. They’re in the CNCF landscape. They are not graduated or incubating but they are seeing widespread use. They also proceeded the CNCF landscape, I believe one of them is Claire, which was a core OS. Or you might pronounce it CoreOS, a core OS project. That was what powered the key.io security solution, if you remember to last episode we talked key was one of the rental options. And Claire comes with key. But you can also run Claire on your own on-prem Docker registry. So that’s one of them an anchor anchor. Anchor, is the other one. And they’re both. They’re both container scanning image scanning solutions, they both have a lot of features and bells and whistles. And they’re all they’ve been very valuable to me personally, as a way to spin up at local registry, push some images to it and find out what what’s wrong with these images so that I can fix them. But they’re also useful components of a larger platform strategy.
Brian Seguin
And those two are buys because you’re buying them with your time they’re open source software, you need to implement them, you need to make sure they’re updated and running in your infrastructure or what have you. There’s also two options for by that also are options for rent. And that’s Aqua and Snyk.
James Hunt
Is it “sneak” or “snick”?
Brian Seguin
“Snick”? I don’t know.
James Hunt
Because the the name is actually an acronym.
Brian Seguin
Oh
James Hunt
Yeah, I spoke to the I don’t know if it was found it was one of the we we really had a run in at. I think KubeCon I talked to somebody from snake and was you know, just talking shop with them. And I asked him what is what is Snyk is it “sneak”? Because that seems like a weird thing to name a security project. What’s your what’s what’s check in your security? Sneak! It sneaks around the perimeter and tries to break in like, No, I think it’s pronounced “snick”. And it stands for “so now you know”. Interesting, which apparently is a an IRC or Slack slang in use in various parts of Silicon Valley. Yeah, it was an interesting little tidbit, fun fact thing you didn’t know. And now you do.
Brian Seguin
So both Snyk and Aqua, are fantastic security scanning solutions. They both offer SaaS offerings, but they both offer a biab by offering where you can run it in your own infrastructure on prem.
James Hunt
And a lot of those also have integration with the container runtimes. So we’ve been talking and we’re focusing in reps primarily on the image registry add on nature, but Aqua in particular has add on agents for Kubernetes Cloud Foundry and other container orchestration platforms so that it can scan not just the images at the image registry and not just the build process of images, but also the runtime for container drift.
Brian Seguin
Aqua and Snyk are really nice in the buy solution because you can actually buy them or run them as SaaS alongside your wherever your registry is. So even if you’re consuming one of these public cloud providers, container registries, you can run Aqua or sneak alongside that. And you can have a different costing structure for how those image containers are run and different features. Because each one of those has different features above and beyond what the regular scanning options provide for the other security scanning solutions. Rental for sneaking Aqua, just to touch on that is mostly per user, it pretty much follows the per user and then you can upgrade different paths for different features, you know, it has a tiered structure for that. And their consumption is kind of built into that per user per tier class. The buy scenario is, you know, you’re licensing their software, you have to talk to them for enterprise grade buy scenarios, right they’ll so you basically will be working out with them for that. Just to kind of go back to the other rental options you if you’re consuming one of the public, public cloud providers, Container Registry solutions. Each one of them has their own add on solution that’s there. Like I said, Google is 26 cents per scanned container as your defender is a consumption model. So you’re basically paying for the compute. And Amazon actually uses Claire which is a surprisingly a Red Hat slash IBM project. But Claire I believe is open source James
James Hunt
primarily through acquisition, right because Claire was Claire was CoreOS and then CoreOS got bought by Red Hat and then Red Hat got bought by Big Blue and yeah, yes, it is open source. So that’s one of the reasons I think Amazon’s not charging you for the scans because they’re not paying licensing or anything to run Claire behind. scenes for the ECR offering.
Brian Seguin
Right. And we did talk about Claire a few minutes ago. But I was just reiterating that even though it’s run by Red Hat, or it’s owned by Red Hat, now, it is still open source and it is consumed by Amazon for their scanning solution. I think from a build standpoint, I mean, I think you’re mostly building your remediation steps you might be
James Hunt
right, don’t go out and build your own scanner, and right. Unless you’re trying to compete with Aqua and Snyk. And these other open source projects Don’t, don’t build it.
Brian Seguin
Right. There’s other scanning solutions out there that are open source that you can implement if you need to, which goes into more of a bi scenario. Build might be manual scanning, right? Build might be just implementing best practices where you say, Alright, every day, we’re going to re push our image, and we’re going to make sure it’s updated with all the latest code. That could be a remediation route.
James Hunt
Yeah, I think for build, you’re really going down the path of ci integration with your image registry, and your scanning solution, all working in concert, to make sure that stuff that is tedious and boring, and that, you know, humans don’t want to do regularly rescanning, the same thing looking for new stuff gets done by something, an automation engine or workflow engine, scheduled cron jobs, something you’re gonna you’re probably gonna have to build that. And then that whole inflow process, as you mentioned, Brian, the get ups, gate gating and gatekeeping to get into production based on you know, are you secure enough, right? We start to think of security as an agile practice, similar to TDD or unit testing, right? When you push your code nowadays, 2021, something runs your unit tests and your integration tests and your functional tests and verifies that you didn’t accidentally break something. Well, one of the things you can break is the security posture of the application by introducing new dependencies, or, or, yeah, really, just by introducing new dependencies. So having security be just another test in your workflow pipeline, will do a lot of good and increasing your overall security posture. And that’s really where build fits into this.
Brian Seguin
Interesting. So it’s more about all the tooling surrounding your scanning and making sure it’s automated. Pretty much. Well, I think so from our recommendation standpoint,
James Hunt
this one’s a slam dunk. It’s a sports ball term.
Brian Seguin
Is it a buy them?
James Hunt
I think it’s a buy. And I think the recommendation here that I know for a fact that the recommendation here follows the same recommendation from our last episode, our image registry episode, if you’re going to buy your image registry, you’re probably going to want to buy your security scanning, to fit into it so that you can build that inflow process. If you’re already renting, ECR or or ACR GCR. avail yourself with the fact that they’ve already included it. And if the pricing on a per scan is too much of a hassle, then you’re now looking at a buy for both.
James Hunt
(promotion) Rent Buy Build is brought to you in part by Hunt Productions, Incorporated. Hey, that’s my company! Hunt Productions specializes in web presence and application development on top of WordPress, Shopify and other major platforms. Through our carefully cultivated network of designers, web hosting partners and SEO professionals. We can help you with all things web. Hunt Productions: we make technology make sense