Episode 12: My Friend, the Twitter Bot

I'd like to introduce you to a friend of mine, the Birthday NOC bot on Twitter. In the interest of not insulting the intelligence of our listeners, we're going to build a semi-complicated application that does something tangible, and then deploy that on all the different clouds.

I’d like to introduce you to a friend of mine, the Birthday NOC bot on Twitter. In the interest of not insulting the intelligence of our listeners, we’re going to build a semi-complicated application that does something tangible, and then deploy that on all the different clouds. What’s the something tangible? Why tweeting out birthday wishes to parts of the IPv4 address space, of course.

James Hunt
Hello loyal listeners, James here. Brian’s out today doing business-y things in business-y places. So I wanted to take a moment and just kind of step back before we dive deeply into this AWS thing that we talked about. As you recall from last episode, we’re gonna start focusing more on individual IaaS and cloud providers, dive deep into their service catalogs, kind of go through what they offer, what each offering does, how it does it, and to some extent, what it would take for you to build a replacement.
James Hunt
And the reason I want to do this is to fully explore some of the things that you may recognize from either your engineers talking about, or reading blogs, or watching talks — knowing and recognizing names like Fargate, and EKS, and Lambda, or CloudRun, or Anthos. All these technologies you might be passingly familiar with, but maybe never had the time or inclination, really to dig deeply. And that’s kind of what we do. Right? That’s, that’s what I do as the tech side of this podcast; I like to build and explore.
James Hunt
So we’re going to kick off the AWS dive with a bit of a survey of the many different ways that you can run application workloads inside of Amazon. And we’re going to start by focusing primarily on containers. This will be the middle ground between straight up virtual machines, and all the way to the Functions as a Service, also called Serverless, or FaaS offerings, so we’re not talking EC2, we’re also not talking Lambda. But we are going to talk about ECS, ECS plus Fargate, and EKS. And the differences between these platforms are important and maybe not as well defined as we would like in the general consciousness.
James Hunt
I spoke at length, last week, with Brian while we were doing research for the podcast, on a strategy or tool or mechanism or methodology of how properly to illustrate what each of these service offerings is capable of, where its strengths lie, what its weaknesses are. And he agreed with my assessment that we probably need to build something. And I know we always recommend renting, haha, very funny. Cue Brian’s laugh track. But I want to build an application. So we’re going to build a non-trivial application, we’re actually going to lift an architecture that I did an implementation for on the Linode Cloud — Linode is a proud sponsor of rent by build, please go sign up for Linode VPS. It’s an amazing technology. Blurbs aside, we built for — I think it was some WebSummit. We did a masterclass type thing. Me and Mike Quatrani from Linode.
James Hunt
And we built a Twitter bot, right, a fun little project that was surprisingly complicated for what we want to do the explanation. So here’s the explanation. A while back, I came across a listing of nation states and geographical boundaries, and the IP addresses and ranges, the CIDR ranges that they contain. I found it useful when I was trying to block a large scale attack from the Middle East against some web server, someone was running a botnet, primarily Palestinian IP addresses. And we just wanted to block most of the non-continental US because that’s what the customer wanted to do.
James Hunt
But I saved that data set. I kept that data set in my back pocket because I thought it would be interesting to do something quirky with it. And “quirk” is one of the hallmarks of my personal brand, I like to think so I built for this Linode master class, I built the birthday NOC Twitter bot. And the idea being that this data set includes where the CIDR range was, was granted, who owns the range, what the range is, and when it was granted, and that that last data point that when the birthday, as it were of that ipv4 range is the whole underpinning of this, this. I call it a tool, but it’s really a toy. It’s a complicated toy. But so what Birthday NOC does is it consumes this data set. And on the anniversary of the grant of each IPv4 range, it tweets out birthday wishes and uses things like the number of hosts in the network. So a /24 has 256 addresses. A /8 has many, many more. It uses the relative geographic area, the seasons etc. to kind of personalize these greetings to a set of numbers that are contiguous and addressable on the Internet.
James Hunt
And this actually runs or we will be running this on on Twitter dot com itself on the actual Twitter application. So the fun thing about Birthday NOC is that instead of just building a single application, in say Go or Ruby or Node.js, we built or designed an architecture,that’s slightly distributed, I say it’s a distributed system in that it has something akin to microservices. But we separated the concerns.
James Hunt
So there’s an actor that does the tweeting. But the actor that does the tweeting, the process that actually submits tweets, doesn’t write the tweets, it’s just — let’s consider it the Social Media Manager of the Birthday NOC account. There’s also a data manager who takes in this dataset and properly parses out additional parameters like how many hosts, the month, year and day, date processing, geographic seasons, and all that stuff. And then there’s a set of one or more writer processes. These writers take the data, the set of information, and they pre-write multiple tweets indexed by date for any number of IPv4 ranges
James Hunt
Now, interestingly enough, you find some fun patterns in data when you start to do this type of work on it. Certain days have more grants than others, as it turns out, but we’ve found through a little bit of analysis that there’s always at least, I think, six — five to six grants per day. So then the front end Twitter bot, that actually posts to Twitter consumes a Redis feed, a list that it regularly cycles through. And the writers take the data from another Redis key value store and build via a natural language pattern, I think is written in Perl, and puts those back into Redis.
James Hunt
The Perl bit brings me to the next thing. All of the pieces and parts are a combination in this architecture are a combination of off-the-shelf components, off-the-shelf libraries, bespoke libraries and bits of code, and a whole different mishmash of languages.
James Hunt
The point here is not to wax poetic about how containers allow us to choose the right tool for the right job. The point is more to discuss that this example — this Birthday NOC — this quite funny, quirky, laughable little implementation is trying to be a decent representation of real-world applications that have dozens or hundreds of different components that you’re going to need to be building and running on these platforms.
James Hunt
Which gets me to the platforms. We’re going to take birthday knock, and we’re gonna put it on straight ECS, then we’re gonna put it on ECS plus Fargate, which is a little bit of a scale-to-zero story for ECS. And then we’re gonna put it on EKS, which is kind of the the lowest common denominator, because EKS is really just a Kubernetes.
James Hunt
But once we take those three, we’ll also be looking at Lambda-fying this in a future set of episodes, and putting it on EC2. But for the purposes of the podcast, so that you know what to expect. And whether you’re going to keep subscribing – which we hope you do. The episodes aren’t going to focus on the technical. Instead, what we’re going to do is produce an additional set of content, blog posts, videos, etc. Twitter, tweet storms, all kinds of stuff, to get to the technical nitty-gritty of how to spin these things up how the application is put together, how it’s mapped onto the concepts of ECS and EKS, where Fargate fits in how we lambda fight all that.
James Hunt
And then we’re going to take that content those bits of information as auxilary reference material for us to talk about on the podcast proper. So this is where, because Brian will fall asleep. If I spend the entire time talking about how we got Rust and Go and Ruby and Perl and Redis and MySQL and Postgres all the talk nicely together.
James Hunt
We’re going to instead focus on what that means from the business case. If you’re looking at ECS, what are we looking at from a vendor lock in? What other parts of the service catalog are we using to empower the ECS strategy, et cetera, and then the same for EKS, Lambda, etc.
James Hunt
So it sounds like a lot of fun. I’m looking forward to it. The https://rentbuybuild.cloud website we’ll have all of the blog posts and links to the videos as we post them up as we go out and try and build Birthday NOC. If you’d like you can follow us on Twitter at @rentbuybuild. You can also follow the Birthday NOC bot, @birthdaynoc on Twitter.
James Hunt
I’m your host James Hunt and Brian we’ll be back next week when we’ve dropped our first episode where we talk about EKS and deploying the Birthday NOC bot on top of Kubernetes
James Hunt
You can find all episodes of Rent, Buy, Build on our website https://rentbuybuild.cloud. Rent, Buy, Build is proudly sponsored by Linode and Hunt Productions, Inc.