End-to-End Django on Kubernetes by Frank Wiles

Speaker 1: Good afternoon, everybody. Thanks for having me. This may come as a surprise to you, but I am not Josh Burkis. But the more I thought about it, I realized that you could be easily confused. Because Josh's name's on the program still. We both have beards. We both like Django and Postgres and Kubernetes. We even both have the same damn glasses. But there's some differences. Over here on the left, we've got kind of the average user of Postgres 's level of knowledge, and then I know a little bit more, and then Josh knows a whole lot more. But about now you're probably asking yourself what the hell this has to do with anything about the talk. And there's one final important difference that that does pertain to this talk.

Speaker 1: My back works pretty okay. Josh is not so much. Um, which is why I'm up here to talk to you about Django and Kubernetes. See, Josh managed to hurt his back making these lovely speaker gifts for all of the DjangoCon attendees. Uh Josh is an excellent potter as well as open open source geek and last week he hurt his back finishing these up so he wasn't able to make it and uh the Django Tom team was like hey uh any chance you want to give a talk on Tuesday. So this one would probably not be my most polished talk ever. But uh on to the topic that you are all here for Kubernetes. Kubernetes is arguably the best and most popular container orchestration system out there today. That doesn't mean that it won't get replaced in a year by something cooler and better, but right now this is kind of what everybody is playing with for the most part.

Speaker 1: But before we dive too deeply into Kubernetes, we have to get into some terminology. So the name, Kubernetes, what does it mean? Is it Greek for ship captain? Or is it that Google learned its lesson naming things after Go? Or is it three all of the above? If you picked three, uh you are correct. It is Greek for ship captain, and I do think that they named it because they they had so much trouble with Go. Today, most everybody is looking at or moving toward containers in some form or fashion. And containers are great about being able to depend To package up all of our dependencies for an app into a thing that we can then move around and share and use. But working with them on their own

Speaker 1: Is is not exactly the most user-friendly thing. If you had to remember which ports am I using, what volumes do I mount, where, that stuff gets complicated over time. And that's why c container orchestration systems exist. Some examples of those, if you're not familiar, are like Docker Compose is really a container orchestration system. Docker Swarm, Mesos, AWS. container service and Kubernetes are all handled orchestrating these containers that you want to use together. So what's container orchestration? Well event loops are used by Kubernetes components to reconcile things between local machines and the desired cluster state. So what does that mean to us? We basically tell Kubernetes, this is how I would like the world to look.

Speaker 1: And then Kubernetes sits there and spins in a loop and tries to make that happen for you. It can't always do it, but it will continually keep trying to make it happen. So I'm not gonna so it it the the other term they use from the documentation here is is a control loop, which in robotics is kind of the main process that's just sitting there going , am I standing up? Am I if I hit a wall, do I need to back up? What's happening right now? And that's basically what Kubernetes is constantly doing. Is everything that's supposed to be running running? Can everything talk to each other that's supposed to be talking to each other, right? And I'm not going to lie to you and say that Kubernetes is super easy to learn. We are definitely not going to learn it all in 40 minutes today, but it it's a big complicated system.

Speaker 1: It's a bear. It's a big scary bear. But my goal is to change your impression of it from this to more of this. Um when I started playing with Kubernetes, the terminology is what tripped me up. There's a bunch of new terms, there's a whole bunch of new concepts that you just don't tend to think about, but we we've all done before, but the way they talk about them sometimes is confusing. So most of today is going to be coming up and getting comfortable with what these different terms mean, and then we'll piece it all together and have a working Django app towards the end. Oops. Sorry, these slides move slow when there's big images. So Kubernetes has a concept of masters and nodes, worker nodes.

Speaker 1: The masters are where all the Kubernetes magic happens of what should be running where, and the nodes are where your containers actually run. You can have differing numbers, they're not a one-to-one relationship. So this would be a simple three. Master cluster with three worker nodes. A more real-world scenario would be something like this. So inside of an AWS region, you have a master per availability Zone, those three are clustered together. And then you have some number of worker nodes also in each availability zone. This is so that underneath Kubernetes is really just an etcd cluster. that has a nice API over top of it. So when you say, I want you to run these sets of containers, it puts it that data into the etcd cluster, which gets then clustered between all of the masters.

Speaker 1: And then the API and the scheduler little daemons that run on these nodes constantly are looking at that and saying, am I running what I should be running? Is there something out there that needs to be running that's not running? Where should we run it? One of the things that trips people up is authenticating to your Kubernetes cluster. Almost everything happens through Kubernetes. This config file in your home directory, effectively known as kubeconfig. There are multiple authentication with Kubernetes is fairly pluggable, but not as pluggable and easy to use as, say, like Django's authentication backends. But out of the box, you tend to have one user with a password and one set of SSL keys to talk to it, and you share that amongst multiple

Speaker 1: sysads. That's kind of the default configuration. It feels wrong and messy and sick, and it is kind of wrong and messy and sick, but it works. There are other systems you can authenticate communicate against Google if you use Google Apps so you can give just certain people access to the cluster. There are other schemes that you can implement I'm not going to get too far into that, but it is important that you have a kube config and that it is properly set up pointing at your cluster. And you can have multiple of them so that you can switch between multiple clusters of which one you're talking to. at any given time. And so you can access your cluster by proxy. So if you run kubectl proxy, it looks for that default kubeconfig, it looks for the cluster that you're currently pointing at.

Speaker 1: So like I have five or six clusters in my config, so I pick which one I'm going to be playing with at any given moment. And then if I run kubectl proxy, I am then able to access The API of the actual masters. If you go if your cluster is running the Kubernetes dash dashboard you can then access it with this and I need to move this to a different window Oh, come on. No, that's not going to let me play with my browser, is it?

Speaker 1: Well, I guess we are going to skip the dashboard portion of the evening. Um the dashboard is a really decent web interface to the cluster as a whole. You can see all of the various components, you can see all of the config, you can make edits to the config, you can see resource utilization across your nodes, which ones have high CPU or high memory usage. And you can kind of just get a nice kind of dashboard state of your cluster. All of that same information, all of the information for the dashboard comes from the API, which is also all the same exact information that you get on command. That you can also then access from Python and Go and other languages. One of the ways that you keep things sane in a Kubernetes cluster is by using namespaces.

Speaker 1: A namespace in Kubernetes is a fence between other containers. So pods in a namespace containers containers in a namespace can only easily access other containers in that same namespace. So you can use it as kind of a light mechanism to for multiple multi-tenancy. You can also use those kinds of namespaces as a light way to kind of separate dev from stage from prod and all inside the same cluster. But it's not as hard and fast of a wall as true multi-tenant separation. Like your containers could end up talking to each other. It's a balsa wood fence, not a brick wall. In Kubernetes, you define resources using YAML.

Speaker 1: So this little bit of snippet at the top there is all the YAML you need. to create a namespace called revsys-rocks. You create it in the cluster by just running kubectl apply -f. Pointed to the file you created and that will come back and say I've created this namespace. Um if it's already created, it'll say uh I configured this namespace because it was already configured and and it you can reapply these same files because all you're doing is adding state into the system and so if the state is the same nothing changes but if there's a new state action get taken. Deployments. Deployments are a template of how you would like the world to look. So you say, I want to have this Django app.

Speaker 1: And I want to run this particular container, and I want to use these environment variables, and I want to run three copies of them. This is kind of hard to read size-wise, but you'll see we just call it a kind of deployment, and then we say we want to have two replicas, and the template is this particular container and we want to open that container port 80 to inside the cluster. And just like with the namespace, we do the exact same apply command to actually put that into the cluster cluster. Services are in Kubernetes are what we think of as a service. I I've just created a Django app running in

Speaker 1: those couple of containers. Now I need to tell the rest of the cluster that there is this web service out there. And I define that like this. We create a service. Notice all of these are in the same namespace, RevSys Rocks. I tend to use the same name for the namespace and the app and the service and everything to just keep myself sane. You could call them things differently. I know that one of my coworkers, Steven, would probably call this service HTTP or WISGI, uh, where I would call it the name of the actual service that I'm thinking of it as the website, right? Um there's no hard or fast rule here. But all we're doing is saying, hey, for this service, I'm gonna open up port

Speaker 1: 80 and it needs to go to the container port 80. Kubernetes has a concept called an ingress controller. So so far everything that we've done is available inside of our cluster to other things running in the cluster, but it is not available to the outside world. in any way, shape, or form. Opening that port 80 did not open port 80 on an external IP address anywhere. So the inter ingress controllers map outside world things to inside the cluster. Now, depending upon where you host this , changes what kind of ingress controller you can use. So if you're on AWS, it would use an ELB or an ALB as your ingress controller and it manages what points where when. So you just say I want one of these and it'll go out and create one and then you start putting

Speaker 1: pointing DNS at the C name, right? But you don't have to actually go configure it at all. Which leads to a quick aside here. We use a controller called KubeLego, which handles everything about Let's Encrypt and certificates. You install this into your cluster, and then in these YAML definitions, you can use what are called annotations. And they're basically just keys in the YAML that Kubernetes itself is not particularly looking for, right? So a controller is just something that is listening to this API, looking for state changes, and taking some action. The default Kubernetes ones handle things like, ah, I need to be running this container over on this node, but you can create your own annotations and take other actions. So in this case

Speaker 1: case somebody created a system to handle Let's Encrypt certificates. So you say, hey, I want a Let's Encrypt certificate for this particular host name. It'll go out and register it. It hijacks the dot well-known location. handles all the key management, stores it inside the cluster, and presents that to the world as let's encrypted SSL connection from then on. And you literally have to do just a couple of lines of config. So this is an ingress controller definition and you'll see we have the similar kind of pattern name namespace right um then there's the rules there and it's a host this is the domain is actually revsys. rocks, one of the new top-level domains, so don't get confused. That could be.

Speaker 1: com or. org or whatever. But then we have a a little part that I want to highlight, and this is the kubego part. So we just have an annotation there saying, hey, I want TLS Acme and I want to use an Nginx ingress controller and its hosts should be revsys. rocks, and I want you to store that as a secret named revsys-rocks-tls. And I don't have to do anything else. When it comes up, it gets the cert. If it needs renewal, it'll renew. And I don't have to deal with that. And I don't have to deal with that on a per application basis or even per container basis. I could throw a couple of Rails projects in a Go project. into this cluster and they have let's encrypt and it's totally independent of whatever I'm doing in my containers.

Speaker 1: So One of the last pieces of terminology is a pod. When we create deployments, all the containers in a deployment form a pod. Pod. Pods are sets of containers that are deployed together on a host. So if you have things in a pod and it has four containers to it, all four of those is gonna are going to be deployed on host A. If for some reason it can't deploy on host A or host A dies, they will all be picked up and run together on host B. So they are always a set together. This could be useful for lots of different scenarios. In all the examples I have today, we are only using one single container. So the the uh difference there does not become particularly apparent. But if you needed additional containers that only talk to each other and not necessarily the outside world, this can be very efficient.

Speaker 1: You can do things like share a Unix socket on a host that you wouldn't be able to do because you couldn't guarantee that they would be running on the same host. So if you have like a one container Django app with one memcache instance, you could have that talk over a Unix domain socket instead of a TCP socket and get a little bit better performance. performance and you can only do that because you know they're always running on the same hosts together. So at a high-level view, Kubernetes is the masters run this API and store this cluster state And the nodes run pods, which provide services inside the cluster, and ingress controllers map the outside world to the inside world. So if you have

Speaker 1: a host, let's say we have three worker nodes, and we have some stuff running on host A and some stuff running on host B. If you shoot host A in the head, AWS just terminates the instance, the other masters are going to go, wait a second, the pods that were scheduled on host A are no longer running on a host because we can no longer see host A, I need to schedule them somewhere. Where do they fit? Okay, host C is pretty empty. I'm going to run them over here. Change all of the pointers, all the different proxy ports, the ingress controller, all that stuff gets changed. changed over and you're back up and running. So you can do things like upgrade your worker nodes from when one AWS instance size to another and never have any of your stuff go down.

Speaker 1: And not have to change any of your configurations or IP addresses. It gets you away from pointing at IP addresses and and and temporary host names and and and makes things move move a little more smoothly. So how do you run Kubernetes in the real world? There are kind of three different things you might interact with. One is called COPS, K-O-P-S. It's a utility for spinning up kubeclusters in AWS. It works really well. It handles all of the AWS specific nature of Kubernetes clusters. So one of the hardest things about Kubernetes is getting a cluster to start. Uh it is not easy to turn on. It is really hard to kill once you've turned them on and that's kind of its job, but getting them turned on is kind of involved

Speaker 1: and prone to error. So people have created these wrapper utilities. to make the process a little more turnkey for us mere mortals. The other option, and this is the option I would encourage you to play with first. If you have an interest in Kubernetes, play with it on Google Container Engine. Which is a hosted version of Kubernetes with Google. The reason I suggest it is then you know that you have a well-working, good-to-go Kubernetes cluster to play with. Any problems you're having are your misunderstanding of Kubernetes. how Kubernetes works or a configuration mistake and not perhaps you set up the cluster poorly. And then there's also Minikube. which runs a single Kubernet single node Kubernetes cluster on your laptop using Vagrant or VirtualBox

Speaker 1: and and and and uh Linux systems on your laptop. And that's a great way to play with Kubernetes in the small for developer environments. You can use those same definitions to define which containers to run at which services to run. to expose on your laptop and then just move and use them then in your in your production clusters. So one of the things you've got to be able to do with containers is configure them, right? We're all 12-factor apps. now right so we've got to be able to push this uh configuration into these containers and Kubernetes provides several different ways. Environment variables of course we can just just define environment into the YAML there towards the bottom. I'll highlight this a little bit. You can see we've just defined an environmental name and we've put in

Speaker 1: a value and that gets injected into the container. environment. That's great and all, but a lot of times we don't want to expose all of that into our configuration. We can also use what's called config maps. And this lets us map sets of variable-like things, uh whole files or entire directories of files into our pods. So maybe we don't want to have to list every single environment variable in that deployment YAML. can say here's a config map of 25 environment variables, take these and apply them into this pod. And you can pick which map goes to which container. And it just does them all for you. You can also do things like I want to use this Nginx configuration file, put it here on disk. And it will grab it from Kubernetes

Speaker 1: configuration secret store and plop it into the pod at runtime Kubernetes also has a concept of secrets. Um these are great. We could obviously put our database password and our API keys into those environment variables in our deployment YAML, but that means that everybody gets to see them. Perhaps we don't want our developers to know those and just the ops people should hold on to those. So Kubernetes lets you define secrets. Secrets are available like most things only inside the new namespace that they're defined in, so you can't share secrets across that fence. But unfortunately for secrets, they're not particularly secure. Right now Kubernetes stores them as base 65

Speaker 1: So they're not as secret as you might want. Now, to be fair, they are working towards real secrets encrypted on the master secrets, and this was Just a stepping stone to to to getting there. But it does keep secrets that should not be on a node from getting to that node. So until a pod needs to run there that needs access to that secret. That secret won't exist on that node. So it does keep them off places where they have absolutely no business. It's just that once they're there, they're not particularly secret. And so this is how you use secrets in a in an environment variable. So we say, hey, I want to have this database password environment variable, uh, get its value from

Speaker 1: the secret named RevSysprojects DB password and the key in there of password. You can also use, I mean this is just a set of hosts, right? So you could run a vault cluster. in your Kubernetes cluster and get your secrets from Vault or some other kind of really truly secure secret storage. So, because we don't know where our containers are going to be running, centralized logging becomes terribly important for being able to figure out what's going on. So if you use Google Container Engine, the logs from your cluster go straight into Google's logging tools, their Stack Driver logging system. that works fine. We've had good luck with the EFK stack, which is Elasticsearch, FluentD, or FluentBit, which is a smaller C version of

Speaker 1: EK. of the fluent daemon uh and cabana uh but you gotta have this or you're not gonna be able to tell what's going on right i i i don't even know where revsys rocks which host it's running on right I'd have to to go and dig and find out where it's running to even get onto the host to then look at logs. So having centralized logging is important. And so for part of that, we've uh lightly open sourced this. Uh it we use it, uh I don't know how useful it will be for for you all, but um it's uh jslog for kube and it configures Gunacorn and your Python apps to use JSON uh logging to standard out. and includes information that is Kubernetes specific. So like what was the pod's IP address inside the cluster, what host was it running on, what was the name of the pod?

Speaker 1: uh uh what app is it in those kinds of uh metadata that's Kubernetes specific gets added into the JSON that's emitted in your logging So data persistence is pretty important. Uh and there's a couple of ways to handle it with Kubernetes. The hard way is with persistent volumes. This works, but it's kind of hard to manage and it's kind of hard to wrap your brain around. This would be this is advanced Kubernetes here. What you're doing is you're saying I have this volume and I want to uh I provide a certain amount of space and then your apps claim They make a persistent volume claim of how much space they need, and Kubernetes tries to match up the claims with the volumes

Speaker 1: as efficiently as they as it can and then we'll mount those volumes on the hosts where those pods with the claims run. And then if those pods get evicted for some reason or the host dies, it then remounts that volume to the new host where those these things run. And in a perfect world that's exactly how it works and it and it and it works that smoothly. I have yet to experience that perfect world. So the easier solution is to do off-cluster storage. Um and this is where I encourage people to start. And all this is, is the existing way you are doing storage. You have a database server somewhere that all of your containers then connect to. And you manage your database server as bare metal, or you use Amazon RDS or something like that for those kinds of persistent data stores.

Speaker 1: One of the things that Joshua's going to talk about is Petroni. Petroni is a templating system for highly available Postgres. The idea is that you could keep a master running in the cluster and slaves running in the cluster and replicate your data from one to another, and as containers were killed off or nodes died, you could You can keep that replication chain working between the nodes to the point where you didn't lose data. I've heard good things about it. I've never actually played with it. And so I wasn't comfortable showing you how to do it, having never done it myself. But I do want to mention it in case you're interested in playing a little fast and loose with your data. Um oh and this was out of

Speaker 1: this slide is out of order, I'm sorry. Um the idea here is your persistent data instance is just an instance there outside of your actual blue Kubernetes cluster just inside the same VPC so that the cluster can access it, but it's not actually running on Kubernetes. It's just a bare metal node. Using Ansible or Puppet or whatever you want to use, or do it by hand, right? So one thing that I do not have a ton of experience with, um, but I know is useful. is Helm. Helm is a package management system for Kubernetes. You can think of it as templating those YAML blocks. But it's useful in more complicated scenarios. So you can say, run

Speaker 1: me a console cluster. And I want to have this many nodes, and it will figure out what all needs to be applied to the Kubernetes API to get you an up and working console cluster with the federation and the leader election. and handle all that stuff for you. So you can build these templateable systems to the point where I should be able to take your system and helm install it and I just have that then running and working on my cloud. And I shouldn't really have to do anything else other than maybe a little bit of secrets management. So One of the things, because Kubernetes is just an API, really, we can use the API from Python. So this is all you have to write.

Speaker 1: If you've got kubectl proxy running on your local host or you have a well-formed kubeconfig file in your home directory, all you have to write to get a list of all the pods running in your cluster. And you can s and I'm just printing out their pod's IP address, the namespace, and the name of the pod. This is uh a generated API off the swagger docs from Kubernetes. and it's kept up to date with releases. So this you should have always have full access of the API from Python. So you do not have to build your tooling in Go unless you want to. So what does that look like? Here's the output for that is just I ran this on our RevSys production cluster and you can see various

Speaker 1: Namespaces we've created there in the middle and the various pod IP addresses and then the names of the actual pods. And you'll see that it takes the name that I gave them, like RevsysRocks, and then append a uniqueness to it and that's that particular instance of that pod. So every time a new pod comes up it gets its own unique name and then if it gets killed a new one comes up so you can see a differentiation in the logs. Even if it's the same container, one got evicted and a new one got created, you'll see that name change happen. So everything in Kubernetes works with a operator or a controller. And why would you want to create your own? Well, like KubeLego, you can create

Speaker 1: your own tooling that takes action when these things happen. Right? So you add a little bit of annotation of your own, and you can watch the cluster using that little bit of Python and say, ah, I'm seeing a new pod come up that's annotated. Frank needs to do something. something to it. And I see that and I can go take action either inside the cluster or outside the cluster, however I need to see whatever I want to have happen when that annotation comes up, I can make happen. So here's some examples of operators you could build. Pipe a message into Slack anytime somebody creates a new deployment. When somebody's launching a whole new thing, pop a note into Slack so we know that that happened. Or maybe we want to check anytime pods come up and down for whatever reason.

Speaker 1: We want to get that message in Slack. That'd be like 10 or 15 lines of Python. Nothing particularly hard. Package that up in a container, tell Kubernetes to run it. You could watch your Django apps. and look for the database connection information and automatically back up any databases that are running and being used by your cluster without having to go in and define each one. You can just say, oh look, here comes a new Frank's test system 47 uh just came up. It's annotated as backup equals true, so I'm gonna go back it up and put it to S3, and I have one centralized system for dealing with that. Just like we have centralized logging, we can have centralized control because we've abstracted out the kind of the whole concept of ops to this API we can watch.

Speaker 1: Maybe you have really complicated rollout scenarios where you have six hours of collect static runs before finally things come up. Well maybe you want to handle low amounts of downtime. by spinning up an entirely new service, once it's all ready, spin down the old one and move traffic over to the new one. You can orchestrate that with just a little bit of Python, even if it's something Kubernetes itself doesn't really support. Hopefully that is enough information to make you interested in Kubernetes, but I'm sure you probably have questions.

Speaker 2: Uh so how do you handle um like CPU? Like I have some service that's gonna need a lot of CPU and I don't want to run you know, these two services on the same node because they're going to conflict with each other and that kind of stuff.

Speaker 1: So in the effort of fitting things on the slide and not melting your brain too much, um I left out um resource quotas which are just items that are in that same YAML and you say this takes this much memory. It has a soft limit of this and a hard limit of this and you can have that for CPU memory and storage and Kubernetes will handle it much like any other kind of quota system. So if it reaches its soft limit. information goes into the API of I'm at my soft limit. At my hard limit, it actually kills the pod and then recreates it. So you can tag pods by how much resource they should use and then where you want to stop them if they grow beyond that. You can also then target nodes. So you may want to have a cluster with some memory-heavy AWS instances and some

Speaker 1: Some CPU-heavy AWS instances, and you can say this pod needs to run on one of my memory-heavy instances, and these should run on my CPU-heavy instances only. And that's how you can do that. And so it will stack as best it can into those nodes based on the values you've given it. It will overflow if you don't put any resources. So like in my examples, it will just keep packing them into nodes and you will eventually hit swap and and things like that. But if it can't then find a spot to put it, because there's not enough resources to put a pod, it will continue. try to find a spot for it and you will see information in the dashboard and the logs that I can't run this pod, I do not have resources to do it. You add another node to your cluster and it immediately puts it

Speaker 1: right there.

Speaker 3: Sorry. I'm totally new to Kubernetes. Uh on the ingress point, does it come to the nodes or to the pods? And

Speaker 1: so it comes to the ingress controller. Right. And then it's proxied inside the the cluster from there. But you would think, oh, this is going to add this extra hop and it's a pain. It really in practice, at a really, really huge scale, it it matters, but for your day-to-day use, no one's ever going to notice that that extra hop is in there. It's a little Go proxy and it's super efficient.

Speaker 3: So considering the the pods auto-replicate, I guess. Yes, the load. Do I still need a load balancer in front of it or no?

Speaker 1: You need uh so the ingress controller from the outside will be the load balancer, right? And so then it comes to the con the cluster and then it load balances from there and then handles all that. Where is this pod running? And it just shoots it to where it needs to go inside the cluster. And you don't have to think about it.

Speaker 4: So in thinking about your application, what what are some ways to make a decision on does this solve more problems than it creates in terms of how do you decide at what point this is gonna do that for me. It's gonna solve more problems than it creates creates

Speaker 1: I mean it that's it's a tough call and that's like with any other tooling right like on on one level I think it's easier sometimes to SSH into a box and I've got to install the thing and and And do it by hand, but that's not reproducible in any way. And so, like, I mean, every tool has its pros and cons. The thing I like about this one is that I'm gonna be doing stuff with containers, so I need some sort of container-based system for the most part. And most other non, you know, uh Ansible and Puppeted Chef and things like that are not kind of container focused. So I don't see them as good tools for solving things around containers that much. Where you pick like which one you use or if you use use one at all um is is hard. The thing I like about this is it really does free me up to not think about the mundane things

Speaker 1: like where is this going to run and what port is available to open on it and how do I proxy from this port to that port? I don't have to think about any of those details. Um and but does give me the power to listen on the API for when things change and take some sort of action. So I like it for that. I don't know that you know if you have one app and you run it uh and you do a deploy once a week, this is probably overkill. If you're managing managing 50 microservices and you deploy 10 times a day, you probably already are building something like this or using something like this to manage all that

Speaker 5: thanks very much for the introduction. Um I was wondering what the next step might be. How did you go about learning this? Do you have resources that you thought were particularly helpful and And could you recommend them?

Speaker 1: So Kubernetes is a very fast-moving beast. and they come out about every six months. And what they they have actually a fairly nice process of things come out and they are marked as alpha and then they are marked as beta and and then they eventually become stable and once they get to beta the YAML configuration for the most part doesn't change and you can pretty much just pick it up and move it over. The alpha stuff is pretty alpha and Good luck if it works. But so the documentation often lags behind the version just a bit on the newer stuff or the stuff that just recently changed. So the documentation should be an amazingly great resource, and it is, as long as you keep in mind that if this thing came out in the last

Speaker 1: version the docs may be wrong or if it just moved from alpha to beta the docs may be slightly off right and so that that's but there's really it's mostly blog posts and tutorials where you're like how did somebody else go about this let me go look at their Kubernetes configuration and then some playing around. There is no really great, here's the book on Kubernetes and and and it's and it solves all

Speaker 6: Right. Thank you, Frank.