Episode 6 – Shhh! It’s a Secret!

This week on Rent, Buy, Build, we're talking about secrets management. We'll explore your options through the Tower of Secrets, and then talk about Vault, Vault, Vault!

Secrets! They’re everywhere! And they’re kind of a pain in the butt to keep safe and secure. In this episode of Rent / Buy / Build, we discuss the Tower of Secrets, touch on the problem of Secret Zero, and swear off building our own cryptographically secure key-value store!

Brian Seguin
Hello, you’re listening to Rent / Buy / Build, the Cloud Native podcast that asks the question: should you rent this, buy this, or build it yourself? I’m Brian Seguin.
James Hunt
And I’m James Hunt.
Brian Seguin
Today we’re talking about secrets! Specifically, I think we’re talking about the secrets to Mission Control — the control keys, right, James?
James Hunt
Right. Every cloud native platform needs nuclear launch codes for deployment. That way, we don’t accidentally deploy the code and start World War III.
Brian Seguin
But it realistically, what we’re actually talking about is privileged account credentials: passwords, certificates, SSH keys, API keys, encryption keys.
James Hunt
All the keys, really.
Brian Seguin
All the keys, except except kind of the Mission Control launch code, right? Those we don’t want to double keys. Yeah, we don’t want to control them. So I kind of want to start this by saying, or asking the question: what happens if you do not protect your secrets? I mean, you have people hacking in to mine cryptocurrency on your, on your Amazon infrastructure, right? We have ransomware, we have application, you know, data loss, there’s a lot of different things that that could happen if you don’t manage your secrets well, and for the sake of this podcast (episode), we’re actually talking about cloud infrastructure secrets and cloud application secrets. So this is secrets that tie into your database or your, you know, infrastructure, your CI/CD pipelines; am I missing anything, James?
James Hunt
There’s a lot of TLS in modern clouds, a lot of mutual TLS, a lot of edge TLS. So there’s a lot of private keys to manage. And talking about threat vectors that you brought up, you know, what happens if you don’t manage your passwords or your secrets? Well, a whole variety of things from the the more obvious things like attacks, advanced persistent threats, you know, attackers getting into your infrastructure and just kind of sitting there collecting data, maybe destroying data. There’s also impersonation. Man-in-the-Middle (MitM) attacks, if, for example, you leak, edge TLS secrets, you give an attacker the ability to impersonate you on the web and pretend to be you and therefore, build on the trust you’ve built with your users to extract data and information from them passwords, personal identifying information, that sort of thing.
Brian Seguin
And any of these things, could — any of these breaches could be a business-ending event for the, for the right company or for the wrong company.
James Hunt
Right. And that’s really why we see such a focus and a fervor in this area of cloud native platforms; the secrets are one of the easiest ways if they if they leak. If you commit and commit them to GitHub, push them up for the world to see, you could be looking at either astronomical charges from your infrastructure provider that essentially end you from a cash flow perspective, you could be talking about loss of–
Brian Seguin
Wait, wait, wait, wait, wait, if someone hacks into my Amazon account, it doesn’t work like the credit card, where if someone you know gets your credit card credentials, and Amazon doesn’t hold you liable for compute,
James Hunt
You can. You can appeal to your infrastructure provider, most of them understand that nobody’s perfect, and that eventually an IAM key or an access key is going to leak or an employee’s going to go rogue. They will refund or cancel out those cloud charges. What they will usually require in return, though, is that you demonstrate that you have taken precautions to avoid that from happening.
Brian Seguin
Okay, so let’s talk about some of those precautions. Right. And I think that’s what we’re getting into from the rent, buy, build, you know, standpoint. I think there’s
James Hunt
Exactly!
Brian Seguin
Well, I guess before we get into rent, buy, build, we kind of have to talk about the different levels of where people put their secrets where people put their keys. I think I like to leave mine underneath the driver’s seat because above the above the
James Hunt
Right everybody looks under the visor so put it under the seat.
Brian Seguin
I also have Bluetooth keys so I don’t know how that works.
James Hunt
You have to have a Bluetooth rock to put the key in.
Brian Seguin
I think that parallels the first level of where people store their their secrets or their passwords to like databases and that’s that they don’t really store them anywhere secure, they just kind of throw them in the application code.
James Hunt
Right. You’re talking about the what I like to term the “Tower of Secrets Management,” and that first level — so if you if you imagine a tower where it takes effort to climb all the way to the top floor. No elevators in the Tower of Secrets — that’s a security precaution. It takes effort to climb from floor to floor. So we’ll we’ll use this mental model kind of to explore what’s available and what people do in the wild; what they should be doing what they shouldn’t be doing. And that first level, the one that requires basically zero effort is to not manage your secrets. You don’t ever rotate them, you don’t ever expire out access keys. Once you’ve used an S3 key, for example, for accessing a bucket, you keep that forever. And please don’t ever revoke that. Because we don’t know what exactly will break. That key, those passwords, those encryption secrets have been baked into the application code, and have been scattered across our very large microservices framework or our very large application. And we don’t have any sort of idea of where things are. And there’s really two main problems with that. One–
Brian Seguin
They’re exposed!
James Hunt
Right! The biggest problem here
Brian Seguin
There is no security. So what’s the second problem, James?
James Hunt
What I mentioned about rotation, it’s hard to change a password, if you’ve hard-coded it in three dozen places, especially when you don’t remember if three dozen is the right number, or if it’s higher. So what happens is you ossify your credentials, and you just kind of don’t ever change this password, right? A lot of horror stories from on-prem IT from you know, when I first got into the scene here are around, you know, we changed Joe’s password — he quit seven years ago, and we kept his account around because he kept the PDI process running for our vendors. And we had to change the password because security orders came in, and we were down for two months chasing where Joe had put his password. This is a terrible place to be on the Tower.
Brian Seguin
So then I guess a lot, it’s a lot safer for me to actually just put my my secrets in my code and then throw it behind a private Git repo, right, that works. James, right? That’s the net layer?
James Hunt
That will mitigate the exposure. Because you’re not, you know, pushing code out into public. You still have the problem of access. Anyone who has access to write the code, read the code, or deploy the code now has access to all the secrets, which isn’t really great–
Brian Seguin
Then everybody in my organization who has access to my private Git repo also has access to all the keys, and potentially contractors, or other people that are that are brought in.
James Hunt
And what we usually see with these is people think of, of application code as code, not as a security primitive. So you may have the best of intentions and say, well, we’re going to do, we’re going to lock down access to the reconciliation ledger application because it has keys in it. And that’s great, day 1. But come day 275, you’ve forgotten that you have done that. And somebody, somewhere is going to need to hire a contractor to fix some code in the reconciliation ledger application, they’re going to hire that contractor, they’re going to give them access. And now, inadvertently, you’ve exposed those credentials. Which leads me to the next floor.
Brian Seguin
Before we get to the next step, I just kind of want to say that it is a thing that happens even in Fortune 2000 companies, you know, these tier one and two, where secrets are not, you know, actually protected and secrets are just put behind a private Git repo
James Hunt
Bigger isn’t always better.
Brian Seguin
I mean, this, these are two methods that actually happen. Now, I think that the next two tiers are the target for a lot of organizations. But I think the the third tier is where most organizations kind of sit.
James Hunt
Right. So the second tier is where you introduce your version control system as an infrastructure as code strategy. So rather than putting the keys in the application code and making the application code repository private, and having to control access to that, you pull the secrets out into an infrastructure-as-code repository and make that private. That has the added benefit of not requiring your developers to have access to the production keys. And you see this a lot in enterprises that have release engineering teams, SREs and the like. Where the people doing the deployments the “DevOps teams”, as it were, those people have access to the production keys, separate from application code access. You’re still exposing your secrets, both at rest and actively because you know, the people interacting with Git can see those-
Brian Seguin
It’s not encrypted at all.
James Hunt
Right. And there’s no real access control, although it’s better, right and that’s the real story of the secrets towers: as you go up, you put in more effort, it does get better. And at some point, you have to make the decision: do we keep climbing, or are we comfortable where we’re at? Because security is not a Boolean, or not a binary thing. You can’t just say, “Oh, yeah, hey, (snaps fingers) now we’re secure.” You’re you’re on a spectrum of security. And you’re always-
Brian Seguin
Is it one of those exponential curves, where-
James Hunt
it usually is, like with most things, right, your last, it’s Pareto principle — your last 20%. of “securing things” is going to take massively more effort than the first 80%. So with floor two, second floor, the next step, we’re putting secrets in an infrastructure repository and having the SREs with access to that. The other thing to keep in mind is we’re still exposing secrets at the application deployment instance. So if the reconciliation ledger app I talked about has access to a database, that database password is in memory as the application spins, which I want to point out, it seems a lot of people go, “Well, duh, of course, it’s there. That’s how the how else is the app supposed to get into the database.” But it is important as we get into the top levels of this tower.
Brian Seguin
And from a software standpoint, or from a code standpoint, these first two tiers, they don’t require any additional code or anything like that to be, you know, kind of put in this is, these are steps that people do out of the box without actually needing to put it in, plug it in. So let’s talk about where software gets in here. And that’s, I think, the next two stages, right?
James Hunt
Yep, the top two floors of the Secrets Tower. The first of which is, is moving the secrets into the deployment system itself. I’m going to talk specifically about Kubernetes, because it has first-class primitives for this, but we also can see this in other containerization platforms and other PaaSes. So when we talk about Kubernetes, and secrets, we’re actually talking about an aptly named object called Secret. And a Secret in Kubernetes is a separate thing, a key-value, if you will, object in the API that you can assess additional access control to. Otherwise it’s functionally equivalent to a ConfigMap. And it’s essentially a way of saying I have an image in my deployment. And the image needs environment variables, and I need to get those environment variable values from somewhere. You either get them from a literal, encoded in the deployment, which is not of interest to us, or you get it from a ConfigMap or a Secret. And the nice thing about that is you can have a completely separate team, managing your Secrets.
James Hunt
Secret team.
James Hunt
Yeah, you’ve got a secret team. Right? We don’t talk about secret club, it’s the first rule. Ah, you have a separate team, your security team, or your AWS team manages subsets of secrets in your Kubernetes cluster, separate even from the operators of that cluster. So now you’ve got three teams, roughly; three roles. You’ve got The Operator, you’ve got The Application Developer, and you’ve got The Secrets Manager. That gives you a little bit more security in depth, right? You’re talking defense-in-depth here, because now operators don’t have direct access to secrets, app devs don’t have direct access to secrets, and the secrets team (one assumes) has gone through proper training, and they’re incentivized by the organization to properly handle that secure material.
Brian Seguin
That seems fairly secure. How do you get more secure beyond that?
James Hunt
As I mentioned, in the last step, you’re you’re still exposing the secrets inside the application environment. That Secret object in Kubernetes translates almost invariably into either a file in the container, so the application would spin up and read a TLS private key from the disk that was put there by Kubernetes by way of the Secret object, or it’s going to read it out of the environment, the Unix/POSIX environment variables. So it’s still present in memory footprint, and the application isn’t really doing anything to mitigate security, just out of the box, because you haven’t told it to. Unix doesn’t differentiate between (a) this environment variable tells me what port to bind, and (b) this environment variable is the keys to all of the important sensitive data. For that we have to go to the top floor of the Secrets Tower, and that is: integrating the secrets management into the application itself. And this one’s fairly new. It doesn’t see a lot of adopted because like I said the Tower is is an effort continuum. The higher up the tower you go, the more effort you’re expending to get more security. Most people are going to stop at deployment orchestration of secrets. And anything above and beyond if you have an attacker — if your threat model involves somebody being able to break into your containerization environment, and then scan memory or or scan files, then you need to build additional defensive positions around that; firewalls, auditing and systems like falcom, those kinds of things, pod security policies, etc. But if you really do want to go that extra mile, the application integration is where you provide out-of-band access to a standalone system whose whole job is to broker access to secrets.
Brian Seguin
So what do you mean by out of band access?
James Hunt
So it gets to one of the problems we have in secrets management that I like to call Secret Zero. We’re going to talk about a product called Vault. It’s an open source security secrets store by Hashicorp. And Vault really is just let’s encrypt a key-value store with some access control. What we’ll do is we’ll say: instead of having all of our applications read their database passwords and their private keys and their RSA keys and all this stuff from files or environment variables, we’ll have them go talk to Vault over an HTTP API. They will authenticate to the Vault, say, “Hey, I’m this app or that piece of infrastructure, I need access to these secrets” and Vault will be the deputy who operates on our behalf saying “yes, you can have this” or “no, you cannot, and I have alerted the security team of your attempts to access out of band stuff or out of bounds and stuff.” The problem with that is: how do you get the application to authenticate to the Vault. That’s the Secret Zero, right? If there’s no difference between being the application, and having the applications credentials — from a security breach perspective. If the application has a token — a randomly-generated and difficult to brute force token for getting into the Vault — I don’t have to attack the Vault, I have to attack the application to get the token to then pretend to be the Vault. And for Vault in particular, it solves this with a bunch of auth back-ends, one of which used to be called Role ID, I think it’s now called App ID. But the idea is that you give different pieces of credential via different mechanisms. So you might bake one half of the key into an image, and the other half comes from the orchestration platform. So you have to put both of these things together; similar — or not dissimilar, at least — to the two keys for nuclear launch. Right, you have to have two people agree that this is what we’re going to do. And the other thing that Vault does to mitigate that is — because it’s a robot, right? Ultimately, it’s a piece of software. It’s going to excel at tedious, repetitive work on very tight timescales. So one of the things you can do once you’ve integrated a secrets manager into the infrastructure side, for example, Postgres has a plugin, or rather Vault has a plugin to talk to Postgres, you can give Vault root access to the Postgres database server, and have it hand out very short-term database credentials to the application. So an application would boot up and would talk to the Vault, say, Hey, I’m the — we’ll go back to the reconciliation ledger app. I’m the reconciliation ledger app, instance number 36, I need access to my database and Vault would actually go to the database as the superuser, provision an account, give the username and credentials and host information to the app, the app would connect, and that credential would only be good for 30 minutes, and at the end of 30 minutes, the application would have to reprove its identity to Vault. And what this gets you from a security perspective is: if somebody steals the credentials out of the application’s memory, they’re only good for so long. They’re gonna get shut down in a half an hour, and you’ve kind of sealed off any holes that do appear. And that’s really the name of the game with security. It’s not making something that is completely unhackable. It’s making something that is difficult to maintain that compromise,
Brian Seguin
Right, the hackers want to go mostly the path of least resistance.
James Hunt
Right. When you find like with the SolarWinds hack that hit, you know, a couple months, a couple months ago now, the most egregious part of the SolarWinds hack was how long they had access. And I believe that when they finally came out that it was through a CI/CD server that had a secret that nobody wanted to bother trying to change.
Brian Seguin
So can you just, on a high level, just real quickly review the different tiers or the different levels?
James Hunt
Sure. So at the very beginning, the lowest level of the tower, you’re not doing anything, you’re stuffing the passwords and the keys into the application code and just deploying it. In the second tier, we move those secrets into infrastructure repositories that have locked down access, and the applications just pull stuff from config files and environment variables. Moving it a step further, we take the secrets out of the infrastructure-as-code and put them into the deployment system, Kubernetes, or Cloud Foundry, or whatever, and it injects them late in the deployment stage without having to manage them in Git. And then the top of the Tower is where the application itself, rather than being a passive consumer of credentials, actively engages with the secrets management solution, like Vault. And that’s really what we’re talking about today is: if we want to get to the fourth stage, what are we looking at from a backing system? What types of platforms can we use? And which ones do we rent, buy, and build?
Brian Seguin
so from a rent scenario, I would assume this has to be only something that your IaaS provider provides, right? This is your Amazon Secrets Manager, your Azure Key Vault, your Google Secret Manager, right? Is there any other type of rent-type solutions for this?
James Hunt
I have not seen any standalone secrets managements that are SaaS offerings,
Brian Seguin
I’d imagine that’d be extremely difficult to do.
James Hunt
I don’t generally go looking for those because the IaaSes provide them. They’re integrated in with the billing and the cloud spend that I’m already spending money on and I already have building relationships with. And it’s all integrated into the primitives that the IaaSes provide.
Brian Seguin
So from a rent scenario, is rent really just limited to using the secrets manager if you’re on an IaaS?
James Hunt
I think so; unless somebody comes out with a 1password for your servers, or a LastPass for your cloud. And I just– I don’t know, from a market perspective, if that makes any sense. It’s a lot of liability. And it’s not a lot of stickiness. Because when we talk about secrets, people get really — like if you say, “hey, I want your all of your app code, like I’m GitHub, I want you to push all of your application code to me.” Fine, that proposition makes sense to a lot of people, GitHub is not going to steal your code. Whereas if you’ve if you’re trying to bring to market a SaaS-based infrastructural Password Manager, essentially, you’ve got a ton of liability. It’s just a scary prospect for a lot of your customers. So you have a lot of convincing to do, it’s very much an uphill battle. With the IaaSes, they already have the customer base. And what they’re doing is really patching a hole in their product offering. Because once I have servers, I have secrets. And to get more servers, I’m going to need more secrets for the the integration and clustering and all that other stuff. So yeah, you’re talking Amazon Secret Manager, Google Secret Manager and Azure’s KeyVault.
Brian Seguin
So none of these provide a solution for the on-prem scenario. So this is only if you’re operating in the cloud, you can, you can rent, right? In this scenario. The advantages are that it’s easy to implement; these IaaSes run this secrets manager for you. The disadvantages is the pricing, I guess, right? It’s the pays-you-go model.
James Hunt
Right
Brian Seguin
For Amazon, you’re paying $0.40 per secret, per month, plus, how many times you access them? Right? And I think Microsoft and Google have similar patterns. Although Google I don’t think that you pay per secret per month, or how does that work?
James Hunt
Google is a little different. Because what Google does is version secrets. So you’re right. Amazon’s about $0.40 a secret a month Google is $0.06 cents per secret per version per location per–
Brian Seguin
Yeah, that was very convoluted. I have no idea what you just said.
James Hunt
It goes back to the rotation of secrets. When we talk about things like database passwords, rotating and deploying the rotated versions is a delicate dance, right? Because at any point in time you have customers or clients, rather, I won’t call them customers, clients, you know, other systems and applications that have the old password and that password needs to continue to work. But over time, you want that to work less and less. Right? So if you have a database password to your MySQL instance, and you have a policy that, for security reasons, you’re going to rotate that every seven days, you’re still going to need to keep versions pre- that you had rotated out from. Right. So the, the beginning-of-the month password, you still need to have a copy of that, in case you need to do some, you still need to accept that at the MySQL instance. So Google just charges you per secret perversion. So if you have only one credential, but you change it every hour, you’re going to pay $0.06 times 24 hours for that day. Whereas Amazon’s gonna charge you $0.40 per month.
Brian Seguin
I think I just want to go with Azure’s model because it’s it’s a lot. It’s a lot more simpler stuff
James Hunt
Microsoft doesn’t charge you-
Brian Seguin
They charge you $0.03 per 10,000 times you access it.
James Hunt
So they charge you for access.
Brian Seguin
Right, exactly. I like that model. It’s easier for me to comprehend. But I guess the Google and Amazon models might have other stuff that I don’t understand in it.
James Hunt
And it’s interesting to note, looking at the pricing for this, Amazon — they all charge you per 10,000 API instances, or API accesses. Google and Microsoft are both $0.03. And Amazon is $0.05. So unless I’m missing something, and I fear that I am, Microsoft is going to be the cheaper deal. Hands down. I believe there’s some sort of holding costs, though, for the vault that Microsoft spins up for you. And that’s one of the other nice advantages of the Amazon, Microsoft Google secrets-in-the-cloud thing. Two advantages really, one, they’re managing HSM vendor relationships. So they’re on the hook for the encryption, and they’re taking care to, to pay their licensing and do the upgrades and all that. But the other real advantage, and it kind of goes in with the “it’s just easy” advantage. You don’t have to know anything about that. They are doing it and it’s completely transferred, you have no decisions to make on, which HSM to use how often to rotate the, the master password that encrypts your vault, none of that stuff, it’s all more or less handled. And in theory, if there’s ever a breach at Amazon, or at Microsoft, or Google, (a) a whole bunch more stuff on the internet is broken, and (b), in theory, there’s a legal avenue for your business or your enterprise to pursue against them.
Brian Seguin
Interesting. So kind of taking a different take on this, it’s, it’s also not like, we can just pick up all of our applications and move it over to a different infrastructure tomorrow, or as we see fit in order to plug in these different secrets managers, especially after you’re consuming, you know, the secrets from the secret managers, you really get locked into that vendor as you start to consume that, right.
James Hunt
Yes, 100%. For some people, though, I think that is a good thing, right? I mean, that’s what you’re paying for is for them to handle it. But the flip side of that coin is, you now are dependent on them doing that, and their way of doing it.
Brian Seguin
So I guess that kind of leads to the buy scenario here. So that’s where you want to kind of purchase a secrets manager or license a secrets manager, deploy it either on-prem or in your cloud. I think the big advantage of this is if you have a hybrid environment where you have on prem and and one of the infrastructure providers, you can use the same secrets manager for both. I think the best example for this that we use all the time is Vault. Vault really does a great job. I think there’s some other ones I wasn’t there.
James Hunt
Yeah, there’s a few. CyberArk, who’s been in the secrets management. business for a very long time, has a product called Conjur, that is similar to Vault in that it is a standalone secrets API driven engine. You can use BitWarden most people think of bit Warden as
Brian Seguin
As the password manager.
James Hunt
I’m tired of paying for 1pass teams, I’m going to go get BitWarden, but BitWarden’s Open Source and because of that, it gets a lot more, it’s a lot more flexible, and people have adapted it to different things. As we were doing the background research for this podcast, I did what I always do, and I looked at the CN/CF landscape diagram. 17 hours later–
James Hunt
(laughter)
James Hunt
I noticed, on the diagram — It’s a very large diagram — that square has a product that they open source called KeyWhiz which, as a father of two I am enamored with this name — that’s K-E-Y W-H-I-Z, like Cheez Whiz, but for keys, I guess. Full disclaimer: I had not heard of it till we started doing background research. Square of course is the point of sales behemoth that if you’ve been shopping in person in the last five years, you’ve seen someone without a phone with a big, white square thing on the end that reads credit cards — that’s square. They apparently built a thing and like most large enterprises in the tech space, they realized, “hey, we should just open source this and let the world have it, and then maybe it’s not such a burden on us.” It’s worth a try. I don’t think it has the track record (at least not publicly) that Vault and Conjur have, and their pedigree as I mean, really, for my mind Vault, kind of built this whole market when they launched the open source thing,
Brian Seguin
because there’s no there’s there’s Enterprise Vault, which you license, and it has a bunch of more features in it. And then there’s also the Open Source Vault, which is really good for a lot of the very basic secrets management needs.
James Hunt
And as we often say, buy in this in all of cloud native sometimes is buy with your time, not buy with your pocketbook,
Brian Seguin
Yes. And so I guess the disadvantage is you’re still responsible for running the secrets manager, you’re still responsible for your secrets. The advantages are you have that uniform secrets manager across your infrastructures and across your team.
James Hunt
You can deploy Vault anywhere, and not just on vSphere.
Brian Seguin
The other advantage is that there’s a lot of Open Source secrets managers out there that you can implement. So you may not even have to pay licensing, but you’re buying it, like you said, with with your time,
James Hunt
And the lock-in aspect is different as well, as long as you stick to the dumb key-value encrypted store, then you have portability. Vault, which we’ll talk about as we get into build and the when to use, Vault has a whole bunch of bells and whistles. Like I said, it can integrate with Postgres, it can integrate with Amazon, and generate IAM access keys for buckets and other things inside of the API for AWS. As long as you stay away from those things, you can eminently pick up all of your secrets and move them somewhere else. Because all it is is a key value store. Which is nice, if you’re not you know sure that vault is right for you or that Conjur is right for you.
Brian Seguin
So build in the scenario. Are people building their own secrets managers? Or-
James Hunt
if they are I fear for them. This I think this; if you remember back to our containers episode, where I think I said nobody should be building their own containerization platform. I would argue that nobody should be building. You should build your container platform before you build your crypto secrets manager.
Brian Seguin
So i don’t want to just hire a team of cryptographers, and bring them in, I create my new encryption keys and everything like that.
James Hunt
I tell you what, if you do hire a team of cryptographers, you’re doing it the right way. But what they’re gonna do is come in and say, yeah, you should use Vault or Conjur, or BitWarden, because security is hard enough when you’re not writing everything yourself. Vault in particular, we went through we– there was another product called CredHub that Cloud Foundry loves to use. And when they when Pivotal first presented CredHub at one of the CF Summits, they had this whole presentation about what it can do and how they built their own creds store, and it’s encrypted and rotating keys and all this other stuff. And then the guy finished his talk, and said, “are there any questions?” And one hand went up. They said, “has the source code for this been audited by a third party security firm?”
Brian Seguin
(laughter)
James Hunt
Which, you know, I mean, that is the most important and pressing– I don’t care if you wrote it in Java, in Rust, in C, In freakin’ Assembly, has somebody who knows what they’re doing looked at this and said, yeah, you did this right? Or have they found bugs? because the the worst thing to do with a piece of security software is use it with blind faith. If you hire a bunch of cryptographers, they’re gonna come in and tell you don’t reinvent Vault. And that’s why I say for the most part, you’re not going to build your own base layer. What you are going to do though, is build your applications into whatever-
Brian Seguin
You’re refactoring your applications to be able to consume one of these other secrets solutions. I guess the other way that build is which we weren’t talking about in context of this episode was the tier one and two where you you don’t store your keys or you you know you started keys in a private repo.
James Hunt
Yeah, I guess but I don’t really count those.
Brian Seguin
No, those aren’t really secrets managers so those shouldn’t be counted which probably edit that out but
James Hunt
I know we won’t. So yeah, so rent is: use one of the IaaS secrets managers buy is run your own third party secrets management and build is for the love of God: don’t
Brian Seguin
Yeah, and I think for the recommend here, if you have a hybrid environment using on prem and cloud, you really want to buy in this case. You really want to have that uniform secrets management solution. Buy is even really good if you’re just on an IaaS, but it’s also really easy to just plug into one of the IaaS provider’s secrets managers,
James Hunt
I usually recommend people rent if they have nothing, and they’re already in the cloud. If you’re running stuff on EC2, and you’re at level one of the secrets tower, right, you’re hard coding secrets in your app, or putting them on a file on the disk of the EC two instance of the Google Cloud compute VM, just chuck them into the SDK. You’re probably going to be small enough that the holding costs aren’t going to be astronomical, you’re probably not doing enough churn where the API access is going to stack up to anything more than like a couple of bucks. As you grow, however, and your application gets more complicated, it grows new features with more secrets and more interaction with other systems, then I would start looking at a buy scenario using one of the off-the-shelf pieces, and again, as you climb the tower, you have to make the decision, do I want to lock myself into the cloud provider? Or do I want to kind of make this decision, strategically, to be portable on the secret stuff? But I do have one specific use case for build that I want to talk about, because it’s it’s something actually we did.
Brian Seguin
Do you still need cryptographers for this?
James Hunt
No. Because what we did is we used Vault. When we built the SHIELD Cloud, SaaS Data Protection solution, one of the key parts of SHIELD was the ability to encrypt archives, the snapshots of the data systems we were backing up because SHIELD as a SaaS offering is: hey, everybody, give me your data, I swear, I’ll keep it safe for you. And that means safe from deletion. Right? We won’t accidentally drop the S3 bucket or the disks that these things sit on. A server outage won’t cause you to lose data, because that’s the whole point of SHIELD is to protect the data so that you can restore it when you screw up. But also to make sure that, as we consolidate and aggregate all of those snapshots into one place. We don’t paint a gigantic red target on our backs.
Brian Seguin
You need a secrets manager to plug into this right?
James Hunt
Well, what we needed to do from an architectural perspective for SHIELD is every snapshot gets encrypted with its own unique and randomized encryption key. And in order for that to work, we have we have a secret problem, right now, we have not just one secret, we have hundreds of thousands, potentially millions of secrets that we need to manage. And when faced with that scope, and that scale, we decided we need a secrets manager. So we decided Vault. Vault is good enough, were stuffing the keys and the initialization vectors into the Vault key-value store. But the application actually talks to Vault. And generates the keys and stuff — it is very much aware of Vault, if you took SHIELD Cloud, and you’ve removed Vault from it, it would cease to operate. And and that is a specific — we had a business case need that was revenue impactful. We can’t sell a data protection solution if the data isn’t safe. And we made a conscious decision to integrate an Open Source platform into that SaaS offering.
Brian Seguin
Okay, so you’re taking–
James Hunt
and it’s worked out very well.
Brian Seguin
You’ve taken the Open Source code and you’re just building on top of it to meet your needs.
James Hunt
Well, where we’re running Vault, we didn’t make any changes to Vault. We’re just, we’re running a vanilla, Open Source Vault inside the the SaaS offering. And we integrated the application via the Vault HTTP APIs. But the point I want to make there is we had a very specific and well-defined security need, we had the ability to change the application to match what we needed it to do, right? This was our code that we had written. And because of that, we went with what I would kind of classify as a hybrid build / buy because we did expend a lot more engineering effort in getting the security of the whole system together. But we didn’t build our own secrets management.
Brian Seguin
Interesting. Okay, I think that’s it. Thank you for listening to rent, buy build. Next time we are talking about
James Hunt
Distributed monitoring. We’re gonna look at all the different ways you can look at your systems from all over the globe. From latency reliability and availability. Join us next time on Rent / Buy / Build. You can find all episodes of rent by build on our website, https://rentbuybuild.cloud.