## Intro [00:00] What is Kubernetes? Kubernetes is a tool that makes it easier to manage and scale an application that exists as a swarm of containers. So it will handle things for you like automatically scaling up resources whether that's networking or storage or whatever based on demand and it handles that for you without you having to think about it. You define upper limits for a particular type of resource and it'll go up to that as needed. It scales automatically. This is particularly useful if you're trying to keep costs down and keep availability high. The rise in complexity of applications built as microservices is what necessitated something like Kubernetes. You could do all of this before. The scaling of applications running as microservices in clusters or swarms. Before we had Kubernetes, it was just very difficult and it was very manual. Kubernetes is just an abstraction on something that we used to do manually. Kubernetes is the second largest open source project in the world after Linux. So what is it? Why is it so popular? And how is it built? Today I sit down with Cass Cosgrove who is the leader of the Kubernetes release team sub project. Cat started off as a software engineer specializing in embedded Linux and got involved with Kubernetes by learning K3S, a minimized version of Kubernetes that is designed for smaller memory limited embedded devices. Kat was a contributor to Kubernetes for years before becoming a lead on different parts of the project. In our conversation, we cover the architecture of Kubernetes and what Kubernetes pods are. Why CAT thinks Kubernetes won and became the de facto infrastructure management solution in part because of its strong focus on documentation. How to propose changes to Kubernetes via the K process and how Kubernetes release happens in the span of 12 to 14 weeks and many more topics. If you're interested in backend infrastructure or how wellorganized large open source project operates, then this episode is for you. If you enjoy the show, please subscribe to the podcast on any podcast platform and on YouTube. K, first of all, just welcome to the podcast. Thank you for having me. Unless you're a Kubernetes well an ## An overview of Kubernetes and who it’s for [02:02] infrastructure uh person who actually deals with Kubernetes day in day out, most software developers, including myself, I've never dealt Kubernetes directly. However, so many of the systems that I deploy to, a lot of the APIs that run today are probably on Kubernetes. I'm I'm pretty sure for example use render. I'm assuming they're going to use Kubernetes. At Uber we use Kubernetes and apparently they're they're one of the the biggest deployments. What is Kubernetes? How did it what is the relationship with Docker and or or if if if there is any and as an average developer when would you be exposed to it if ever? So I think the average developer especially if you're like a web developer or an application developer you don't need to worry about what Kubernetes is. you're never going to have to touch it directly. And um it it's a specialization being like a Kubernetes cluster admin or uh an S sur or something. That's that's a specialization. And I don't think that the average developer needs to know or care what Kubernetes is. You certainly don't need to know how to use it. You can know what it is, but you don't need to know how to use it. There's so much going on. Um it's it's complex. It's notoriously difficult to learn. I don't think that there is a reason to dive into it uh super deep as an average developer unless you're trying to make a career switch into Kubernetes. Um which I couldn't blame you for because it's very lucrative. But um Kubernetes is a tool that makes it easier to manage and scale like an application that exists as a swarm of containers. Um, so it will handle things for you like automatically scaling up resources whether that's networking or storage or whatever based on demand and it it handles that for you without you having to think about it. You define um upper limits for a particular type of resource and it'll go up to that as needed. It scales automatically. This is particularly useful if you're trying to like keep costs down and keep availability high for instance and it it looks so this is direct from the Kubernetes documentation and this is just this is what Kubernetes components look like. This is a very very simple cluster. Um, this box over ## A quick glimpse at the architecture: Kubernetes components, pods, and cubelets [04:27] here on the left is your control plane. And you aren't going to interact directly with this a ton. This is mostly automated. Um, over here on the right, these where it says node, node, node, those are all pods. And within these pods run the cublet, which is another like little little control thing. It's interacting with the um Kubernetes API to stand up and down all of your containers. And all of your containers exist within here too. So if um one goes down like a a pod fails for whatever reason, whether it the code itself, the application crashes or it runs out of resources or something else weird happens, um your deployment will stand it up again uh automatically for you. You won't notice it. Your users won't notice it probably. This episode is brought to you by work OS. If you're building a SAS app, at some point your customers will start asking for enterprise features like SL authentication, skin provisioning, and fine grain authorization. That's where work OS comes in, making it fast and painless to add enterprise features to your app. Their APIs are easy to understand and you can ship quickly and get back to building other features. Work OS also provides a free user management solution called Offkit for up to 1 million monthly active users. It's a drop in replacement for Alt Zero and comes standard with useful features like domain verification, rulebased access control, bot protection, and MFA. It's powered by Radics components, which means zero compromises in design. You get limitless customizations as well as modular templates designed for quick integrations. Today, hundreds of fast growing startups are powered by work OS, including ones you probably know like Cursor, Verscell, and Perplexity. Check it out at work.com to learn more. That is workos.com. This episode is brought to you by Modal, the cloud platform that makes AI development simple. Need GPUs without the headache. With Modal, just add one line of code to any Python function and boom, it's running in the cloud on your choice of CPU or GPU. And the best part, you only pay for what you use. With sub-second container start and instant scaling to thousands of GPUs, it's no wonder companies like Sunno, Ramp, and Substack already trust Modal for their AI applications. Getting an H100 is just a pip install away. Go to modal.com/pragmatic to get $30 in free credits every month. That is m o d.com/pragmatic. And then so how does a ## Containers vs. virtual machines [07:00] pod and let's say virtual machine connect or or like in this case when we say container, it's probably a virtual machine, right? Nope. There is a difference between a container and a virtual machine. Um, all right. Containers are virtualizing um the operating system and applications running within it. A virtual machine is larger and beefier and it is uh likely um also virtualizing hardware. So um they are like they're different types of virtualization and Kubernetes relies on many types of virtualization. Um, virtual memory is probably the is the most recent like I don't know breakthrough in virtualization I guess and we cracked that in like uh the 60s but that's that's what allows things like containers to exist virtualized memory. Um but yeah a container is a type of virtualization a virtual machine is a type of virtualization it's just much much beefier. um you you can run Kubernetes inside of a VM. Like there's nothing stopping you from doing that, but uh containers are not VMs themselves. They're they're smaller, they're more lightweight, they're easier to share around. Um they're they're much they're more configurable as well, right? Like Yeah, they're they're you can just chuck one. uh like the popularity of Docker is largely because of the sharability of Docker files and things like the Docker registry and now there's you know a million in one container registries but sharability the small compact size of containers is what makes them so attractive. It's why we generally not always develop applications as microservices now rather than monoliths. That's the containers is what allow us to do that. And the like rise in complexity of applications built as microservices is what necessitated something like Kubernetes. Um you could do all of this before the the scaling of applications um running as microservices in in clusters or swarms. Before we had Kubernetes, it was just very difficult and it was very manual. Kubernetes is just an abstraction on something that we used to do manually. And now since we are automating a lot of it with a very very complex um overlord like Kubernetes uh we can scale our applications much larger much faster and we we can build much more complex applications as much more complex chunks of microservices instead of having to throw a monolith in here or there. So um this is like most advances in technology since the NIAC machine in the 1950s. Kubernetes is just an abstraction layer on something that we used to do by hand that sucked that was hard to do by hand and now we've we've automated it. And the trade-off there is it's when you dig into it it's it's confusing. It's it is it is confusing. Can can can we talk about the history? What happened? What was before ## The origins of Kubernetes [10:02] Kubernetes? Then as I understand the roots are somewhere with Google right cuz I think it's good to understand what is it that you know either CIS admins or or other people were doing and then you know why why why we decided or why some engineers decided like all right let's actually build a better tool and now the whole world just seems to have caught on saying actually yeah we actually want more of this. Yeah. So Google uh has has a history and a reputation of overengineering things, right? They they they do. Um so internal to Google a long long time ago, there was a tool called Borg. Uh that is that is a Star Trek reference, right? You will be assimilated. Still around. Still around. And um Borg was used internal to Google to manage like their their own clusters, their own their own microservices. And somebody decided one day, oo, this is really good. I think we're on to something. We should open source this. And so they did. And they talked to the Linux Foundation who stood up the cloudnative computing foundation as a directed fund of the LF. And Kubernetes was the first project donated to the cloudnative computing foundation. That was in um July almost 11 years ago. And the first commit was by Joe Beta. um a fun fun thing on the name Kubernetes um it means helmsman in Greek and a typical uh ship's wheel. Our our logo has eight spokes, but ours has seven uh as a reference to seven of nine from from Star Trek. It's like a nod to to Bourc, the original name. Yeah. Yeah. Uh but it uh it took off pretty quickly. Before that there were other there were other ways to manage clusters right there's um mthosphere and there was docker swarm and a couple of of others like this was very clearly something that the market wanted that that developers needed that uh s sur needed though we didn't really have s sur as a term back then back then we we certainly do now s sur is just like a super fancy cis admin now but uh yeah it it was born out of Google um solving their own problems and going you know what no people people need this and they they decided to ship it for free and now we're the second largest open source project in the world behind Linux which ## Why Google built Borg, and why they made it an open source project [12:30] we rely on so and and as as as a fun history I talked with Dave Okconor who was one of the first SRRES inside of Google when it was internally called like that but externally the term wasn't popular until much later and he he said this interesting story of how at the time when he joined Google, they already were managing thousands or tens of thousands of servers and the problems that they were facing is they went to um like a hosting provider where they wanted to host their their servers and they told them how how many they need to manage and the hosting provider was used to managing maybe a hundred at max and it was very manual and then he he said that they just needed to create the the processes the tools etc. So my is is it is it fair that the understanding is that we had this explosion in in both physical servers then we had virtualization virtual servers. Google figured out how to do it and then they just turned around and said like okay let's yeah open it up pretty much. Yeah that's that's pretty much what happened. What I mean right now I think it's it's a really positive thing on Google but why do you think the motivation might have been for for Google to it is amazing that they did it and the industry has has greatly benefited from it but they could have just said like okay let's just keep doing what we're doing with Borg and not bother too much because they still have Borg right they do still have Borg um as far as I know they still have Borg I haven't asked a Google engineer about the state of Borg in a minute they have I recently asked as far as I know it's still a thing um do you mean like why didn't they just like productize Borg and sell it. Uh either that or or just why did they think this would benefit uh the the industry and and maybe also their their business? Sure. I mean now that we have Google Cloud, like I I can see how for example, you know, Kubernetes might have indirectly helped that, but I'm just thinking like just from a selfish perspective, this is a really unselfish thing from a corporation to do. I'm trying to understand there's usually a longer term motivation potentially or or or maybe it was just engineers being geeky and and thinking this is really really helpful. Um I know those engineers quite well and they are u massive dweebs and I love them but um awesome. So like, yeah, I totally get how people uh would think that open sourcing something as successful as Kubernetes um may not be like a smart business decision, but look at the sheer amount of influence Google has gained over the cloudnative space over that. Um they still have people, they still have employees sitting on the technical oversight committee for the CNCF. They have Linux Foundation governing board members. They have CNCF governing board members. They have people I don't think they have somebody sitting on Oh, no. They do have somebody currently sitting on the Kubernetes steering committee. They still have a degree of control. Um the way the way we govern ourselves as a project um no one company can have more than two representatives on the steering committee. So they can't like rest control of the project from the people anymore like any any more than AWS or Microsoft or Red Hat can. But it has given them a a pretty extreme degree of of influence. Uh I mean we're we're here on this podcast talking about the fact that Google gave us Kubernetes, Google is directly responsible. That's fair. I'm I'm trying to understand because I think it's great that this is happening and I think the more we understand what the benefits are and why other companies potentially should follow. For example, ## The benefits of open source projects [15:51] Spotify did with was with backstage their their developer portal and what they told me is they think it will make it cheaper on the long term for them to actually maintain themselves because now they're going to get other developers from other companies or or independent engineers bringing the ideas and also standardization is is is another thing. So yeah, if your project is good, people will come contribute to it. Um I we open source maintainers have mixed feelings about this but um it is a way to get free labor right nobody pays me to maintain Kubernetes it's it's not my job I don't have a cloud provider writing me a paycheck for running the Kubernetes releases um so we we do do all of this work um for free another benefit is that open source software is inherently more secure than closed source proprietary software Right. Um that that security angle is real hard to argue with. Uh Kubernetes just addressed a like a nasty CVE very recently in ingress engine X and that would be much slower to address on proprietary software with a smaller team that can't work around the clock. Right? You can't force your employees to work around the clock if it's a small group of people that are all in one time zone. an open source project, especially one as large as Kubernetes, we can respond around the clock, you know, and and how how large is Kubernetes both in terms of the uh the widespread in in the industry, the complexity of the software ## The size of Kubernetes [17:25] and the the team as as you uh referred to um Kubernetes first I I first I think I need to explain the different types of people involved in uh an open source project. Um generally you have users who are people who just consume your tool. You have contributors who are writing code or writing documentation or um in the case of the release team doing some like project management type of role but they don't have any control over the future or direction of the project um and have like maybe limited permissions within repositories or orcs. Uh maintainers have titles within a project. So, SIGDOC's technical lead, release team sub project lead are both titles. Um, you exist in an owner's file and you have control over governance and policy decisions about the project that affect everybody. Um, there are a couple dozen SIGs within Kubernetes all with um two to three chairs and two to four technical leads. Uh, those people would all be considered maintainers. then sub project owners um exist below that before you get down to regular contributors. So there's there's probably it's it's a large project and and everybody in SIG leadership is a project manager functionally like we're all doing administrative work. We're all managing people. um were all affecting the direction of our various areas of the of the Kubernetes project and there's probably 150 200 people who would be classed as maintainers and then uh more than a thousand contributors a month. It is it is a month a month. Wow. It's it's massive. So, so about like a few dozen, let's say like 20 to 40 SIG uh leaders, about 150 200 maintainers and like a little pyramid almost and then the the contributors who are I'm assuming contributors, some of them will be regular, some of them will be just one off. Some of them are one-off. Sometimes we, you know, we get drive by contributions like any other big project and some people just stick around as contributors. Um, and you you go through what we call a contributor ladder. So you show up as a contributor at first. You work as a contributor for a while in whatever area interests you most and if you stick around long enough and you want it, you'll eventually become a maintainer. You'll end up leading a sub project or you'll end up as a technical lead or a SIG chair. That's that's what happened to me. I was on the Kubernetes release team for 3 years and then I led a release myself. um Kubernetes version 130 was my release and after that the SIG release chairs offered me the sub project and now I just keep running them and now I'm a maintainer and because I I had run the um docs sub team for the Kubernetes release a couple of times SIG docs asked me to be a technical lead and that's just that's how it works anybody can show up anybody can open an issue anybody can um open a pull request and dive in and and help out. You don't even really have to be a Kubernetes expert to be useful. You just you just have to know how software works or know how to write documentation or um know how to manage a team. And then so Kubernetes widespread wise, as I understand, most of the large ## Cluster management solutions, including different Kubernetes services [20:55] infrastructure runs on on Kubernetes. If if it's compute, if if it's large, it's it's the one that makes most sense. I'm not even sure there's a there's a competitor that that is open source that that is equally capable is there? Well, not not that I would say is equally capable. Um there are certainly other ways to manage clusters. Um Docker Swarm is still a thing. Uh MSOS is still a thing. But yeah, like your people are most commonly using some kind of managed Kubernetes service like um anything on AWS or GKE, right? or like Azure has their own service too. Um pretty sure Oracle does, Red Hat certainly does. And then there are um flavors of Kubernetes like uh VMware's Tanzoo, right? Um and so G given that this project is so uh successful, how does the the the funding ## Why people contribute to Kubernetes [21:48] work? I'm going to assume that some of the contributors as you mentioned are actually paid full-time working at certain companies who have critical infrastructure running on Kubernetes and these companies figure well it's cheaper for us to pay a contributor to ensure that our use cases are are met but for example in your case and some of the other contributors how how does it make sense to spend a considerable amount of of your time which you know like we're talking about software engineers who are highly skilled uh spending here and there will be some goodwill. But what what are other uh ways that the the project remains as sustainable as it is? Uh unfortunately for a lot of us um this is both our this is a hobby, right? Like I I I enjoy it. Um many of my friends are other maintainers or other contributors. Like some of some of my closest friends are also maintainers. I flew to New York just to hang out with uh two of them a couple weeks ago. No reason. not work, just hanging out with them. And uh that's that's not too terribly uncommon. Like people just think it's fun. Um some people also do it as a career bump. This is something that like the Kubernetes project doesn't really love to admit, but we are a very very large and very very influential project and having it on your resume can be a career bump especially for people who are fresh out of college or very early in their career. which is why shadowing on the Kubernetes release team is such a popular entry point for contributors. We get hundreds of applications for the Kubernetes release team every cycle and about half of them are students. Um and that's that's pretty common too. Just just they want the they want the career experience. That is that is encouraging to hear by the way. Yeah. It's a lot of students. It is so many students and uh we do we do get students onto the team. Um, obviously I can't overload the release team with people who have no experience, but we we try to fit um a handful of them in every single cycle. Um, which which feels good. It's it's a good entry point for people. So, we get to watch them grow up through the project until ultimately they go get a job and leave and we never hear from them again, which is fine and doesn't make us feel bad because we we really have more contributors on the release team side than we know what to do with. Um, we we cannot have them all on the team every cycle. But um as for people who are paid for it, that's usually cloud vendors actually. Um like the the big clouds pay a lot of people just to be contributors or just to be maintainers either because you know they need something done or they want the influence or because they want the clout. Um some people get hired by some company just because they have a maintainer title within the project and that's that's it. It's it's purely clout and I don't object to that either as long as somebody's, you know, getting paid to do this instead of doing it for free. But the CNCF doesn't pay us. The Linux Foundation doesn't pay us. Um, the CNCF does pay our infrastructure costs. So like we're not starting a GoFundMe or putting money out of our own pockets to pay for our like testing or whatever. The CNCF pays that. That's part of what you get when you donate a project to the CNCF. they they take on some of your infrastructure costs um depending on where you are in the like sandbox incubating graduated scale but um yeah most of us do this for fun which is not the most healthy thing in the world and you have to set like really really really firm boundaries well well yeah there there there's that thing that comes into mind and also there's this famous XKCD comic where where there's this big pile of rocks and there's this small thing which says someone's side project so given how how critical Kubernetes is is becoming globally. It's it's a I think it's a bit of a mixed feelings. On on one end, I think it is cool to hear that it sounds like it is a very independent project that doesn't ## The anti-burnout policies Kubernetes has in place [25:47] really have any one vendor uh going over it. On the other, there is this downside of like, well, you know, like yeah, it just just makes feelings. It it makes people um it makes me wins for small projects. Um but everybody starts as a small project and however like Kubernetes is so large and so influential and we have so much momentum that XKCD comic is not going to become a problem. U there there is a there is a queue of people waiting in the wings who would love to replace me and part of our we as a project our philosophy is that part of our job as maintainers is to be looking for somebody to mentor into replacing us. Uh we have a lot of anti-burnout policies in place. Um leading a release like being being the actual point person for a release, the the person who ultimately reports reports to me is a very stressful, very public, miserable job to do. It is 14 weeks of having to manage and un 14 weeks. That's the release cycle. That's the release cycles. Yeah. Like 14 to 16 weeks. It depends on when it is in the year, but 14 to 16 weeks is about how long it takes us. And um it's 14 weeks of directly managing an unruly group of like 30 people who are trying to manage interactions with potentially hundreds of contributors between them and it sucks and by the end of it you are so tired. So we don't allow you to be involved in the release cycle after you lead one. You've got to take got to take a cycle off. Um, also SIG releases charter dictates that we will delay a release. We will push it back before we make our team work nights or weekends. We're we're never going to ask you to work more than you are comfortable working. Um, and we're we're very supportive of stepping back if you can't handle it. It doesn't affect your place in the project. You can come back the next cycle or whenever you're ready. Um, we're very encouraging of of taking breaks. So, we don't we don't really like to allow the the burnout to build up to a point where we might topple over like that XKCD comic. Um, we're always replacing each other. So, like eventually I this part is is just very encouraging to hear and this is something that I would I would assume that a lot of commercial companies like a commercial setup would uh let me just say hesitate in in in many cases again because of that thing. So, so I guess maybe we're seeing two parts of it, but this is just it it's the first time I'm actually hearing for such a large such a critical project to actually have it in place, which is just music to my ears. We're unusually well structured for an open source project. Um, we're we're unusually well structured for like an actual product within a company. Actually, we are are very very well organized by necessity because we're massive and we deal with so many people coming in and out constantly. we have no choice but to be super super well organized. So why do you think Kubernetes you you you you've you've seen the story mostly from the the inside and starting from the side. Why do you think Kubernetes won so big? Why has it become the de facto choice for any from from any mediumsiz sometimes even small size ## Why Kubernetes is so popular [29:07] infrastructure? By the way, I' I've talked with Linear and they they chose Kubernetes very early on. Apparently, uh they they thought, you know, some people thought this would be like overkill, but it actually turned out to be a great decision for them. Yeah. H why do you think this project specifically won? What did Kubernetes do so well? Um I think we we caught on early because of because of hype, right? uh because of the Google name brand behind us and the association with Docker which was already very popular. um it was a familiar company donating a project that relied on a tool many were already familiar with and that got us initial like hype cycle right uh but then I'm going to toot my own horn and say that the continued popularity of Kubernetes is at least in part due to our truly exceptional documentation. If Kubernetes does something that you can touch as a user um as a cluster admin, we have documented it. Um Kubernetes uses something in the release cycle called um a Kubernetes enhancement proposal, a ke. Uh this was inspired by Python's PEPs, right? And one of the things we require for a kept to be considered complete and thus uh includable in a particular release is that if it has a userfacing change at all even if it's just a feature flag it must be documented or we do not allow it in the release. Uh today as we're recording this is actually the docs freeze for Kubernetes version 1.33. So today a lot of cap owners will either merge documentation or I will revert their PRs. Wow. That that that that that sounds like tyranny. It's does it does it you're never you're never left digging through Kubernetes code to figure out like what what this is. It sounds amazing as a user and as a developer. I'm cringing you know as a dev like like you really just don't want there's so many excuses we make up for not writing documents. the code will document itself if it's in documentation. It's not written, etc. No, I I'm I'm really impressed. Yeah, I I know the devs hate it. I know. But um our priority isn't making the devs happy. It's building a usable, sustainable project that people want to keep using. That means we got to document that. Um I hated writing docs when I was an engineer, too. I did. Um, I've I've slung some truly horrendous code that was entirely undocumented and I myself looking at it probably could not tell you what on earth it's doing. Um, so we we can't have that in a project this large, you know. Um, yeah. And and the project this complex, right? Like so so it it sounds like you're managing to balance because what I understand is Kubernetes is just become more and more complex as there's it does. you know, you you can tell me more about it. But there's going to be more more edge cases, more hardware, more setup, more more exotic this and that, but when you when you have proper documentation, at least that helps with it. If if you don't have it, there's no chance. Yeah, there's no there's absolutely no chance. And so that has gotten us over the hump of Kubernetes is hard to use. It is hard to use. Um it's hard to learn, but it could be so much worse. and the the availability of good documentation is I give a lot of of credit to that. Um also like GKE has really great tutorials for learning Kubernetes. I GK is Google Kubernetes Engine is their hosted solution. Yeah, it's their hosted solution and they they absolutely have the most like featurerich Kubernetes hosted um option by virtue of having had like several years of a head start on that. Um, but yeah, their their tutorials, they're just like how to learn Kubernetes are are actually quite good for like just getting your dipping your toes in the water. Um, and our own documentation, our own quick start guides. And that's another benefit also of being an open- source project. Like just like an open source project, the software is more secure. Our documentation can be better. We have a small army of people contributing to it and improving our documentation on a constant basis. So we we don't really um Yeah. And I guess I I'm not sure if this applies. Correct me if I'm not, but a lot of open source projects say like look, if you want to start contributing and you're you're not sure just yet, you can always just start with the documentation, right? Like it's ## Why documentation is a good place to get started contributing to an open-source project [33:34] it's an incredible which is win-win. Yeah. Docs are like a t pretty typical entry point for people who are like nervous about contributing to open source for the first time, which is a fear I absolutely get. I was terrified the first time I opened a PR in a Kubernetes repo. Um, it's it's nerve-wracking, especially if you're super early in your career. You know that you've got all of these people with way more experience than you like laser focused on what you just did and like did you apply a label incorrectly. Are you going to get chewed out for that? Um, but yeah, docs are an easier entry point for that. It feels like a little bit less pressure and that's pretty common across open source projects, not just Kubernetes. Um the unique thing about Kubernetes and getting started is the release team. We will let absolutely anybody apply to be on the release team and it's you know 20 25 spots uh every cycle three times a year. So it's it's pretty competitive but we will we will let absolutely anybody apply. You don't even you don't have to have contributed to Kubernetes at all before. Every cycle I've got to add two or three people to the org because they've never contributed before. Like this is their first and but but just just to emphasize, do I understand correctly? For example, we we recently had an episode with with Linux where in order for you to become a well to to lead or release it, you will you will need to put in many many years outstanding contributions and so on and there's no guarantee of of doing so. Correct. But for for other open source project it is also common to have this kind of tenure based thing because what you've said I've not heard it and any other projects say like we will actually open up the spots and and you can apply ## The structure of the Kubernetes release team [35:15] and taking on people with little tonal experience sounds like an approach I've just not heard before. Yeah the uh the application process we have and the way the release team works is unusual for an open source project. Um no tell me more. So so we we talk about that a lot. Um I will absolutely talk about the structure of the release team. So um the Kubernetes release team has two halves. There's the release team which I manage and release engineering. Um release engineering is doing the actual cutting of the releases. But the part of the team I manage is made up of several sub teams and a release lead. Um the sub teams are communications which is responsible for handling things like feature blogs. So that's working with uh cap owners or SIG leads to make sure that a particularly interesting feature that will make it into that release has a blog that goes out on the Kubernetes blog on a certain schedule. Oh, nice. Yeah, it's very cool. Um we we usually have like between 8 to 12 every cycle that end up um and release coms does not write that blog. They're just responsible for managing that it happens. Uh they're also responsible for managing interactions with the cloudnative computing foundation to schedule the release webinar and with the media to handle the webinar. Yeah, we have a we have a release webinar. Um we also have to manage uh the media to deal with press embargos and schedule media interviews for the release lead and anybody else on the release. This sounds like a publicly traded company doing its earnings call. It does, doesn't it? Um we also have release I'm so impressed. Yeah, it's it's a lot. Um, release docs manages making sure that every one of those kepts that has userfacing changes has documentation by its deadline. Uh, enhancements manages all of those kepts and makes sure that they uh have met all of the requirements, the the check boxes in the ke, which is like has the lead opted in? Um, is your code complete? Is it tested? Uh, did you get a production readiness review? That kind of thing. and um release signal which is responsible for watching our CI signal boards and chasing down SIGs and cap owners to squash bugs to make sure that we don't have like a master blocking test failing right before we cut like beta 2, right? Um they're also responsible for the go no-go signal on whether or not we can cut a release. And this requires 20 to 30 people every single cycle. five of them are leads and are selected. Um then the rest are shadows that are go through the application process and and get pulled in. That is unusual. Most open source projects do not operate that way. Um most open source projects I think don't need to. That's a consequence of the fact that the the project is so large and we have so many contributors. We couldn't manage this with a consistent group of like 5 to 10 people without catastrophically burning them all out. um we we just simply couldn't do it. Um and even with the rotating cast, the release team can be a burnout factory because it's so much people management, it's so much project management, it's so much arguing with people and juggling like maybe conflicting priorities between teams. Um so we don't we don't like people to stay on the release team for too long. We want to see you move up through the ranks a little, maybe lead a sub team and then leave, find somewhere else in the project to go because if you stick around, you you will burn out, right? I did not obviously, which is why they gave me the the bigger hat. It it it is very interesting for me to reflect on how much project management and just how much like work management is happening within Kubernetes where whereas when we did the deep dive with the Linux kernel, it was a bit of the opposite. But again, it was pretty clear there that in the in case of Linux kernel, a lot of this work that that you're you're mentioning is happening at the actual releases. Let's say, you know, like Red Hat and whoever picks picks it up. But I think it's just a good reminder of like this this work for for any large project. It's there. You could kind of shift it here and there and and you you can you can play with it, but it has to happen. like it would never no project with more than like two people contributing to it is getting by without some kind of project management work. It's just that we don't have the title project manager. All maintainers are project managers to some degree like we we all have some um administrative or like um governance policy uh position within the project. It's it is project management if you are a maintainer. We just we just don't call them that. Yeah. Um, so like the Linux kernel has them too. Like somebody is doing that work. They're just not they're just not they don't like they don't like that word. I think that was my sense. Yeah, they don't I think a lot of um a lot of engineers are resistant to doing work that uh they don't they don't consider real engineering, you know. Um yeah but we're kind of seeing this by the way like now you know like software engineers like we don't like to say that we do product work but now product engineer is becoming a thing which is a software engineer who who will talk with customers as well and who and you know same same thing I think there used to be a time not not these days but maybe 10 or 15 years ago testing was we had dedicated you know testers or tests or whoever and then engineers wouldn't engineers wouldn't or would object to do testing but but you know, we would write unit tests and then integration and auto and and there this merge. So I think there's clearly a play course and what I'm personally seeing is is a a competent tech tech professional ## How responsibilities shift as engineers grow into senior positions [40:55] may that be a software engineer or or someone who is who's elsewhere will just take on more and more responsibilities. A great product manager these days will know how or will understand code. they they might not code by by choice and not same same thing but even just talking with you you you've been a software engineer you've actually built like critical infrastructure and now you're you you're doing something where we're doing less of that but it yeah I haven't written a line of code in months I don't have a reason to you could if if you wanted to or if you needed to if I wanted to yeah like for fun sometimes like I'll do some hobby projects at home or whatever but I don't have a reason to write code I used to be an engineer I can read and understand it just fine can write it if I need to. Um, but it's not it's not part of my job anymore. And like we you see this within engineering at real companies, right? The more senior your engineering title is, the less code writing you're actually doing. A principal engineer should not have their fingers in the code nearly as much as somebody who's like an an SD2, right? like they they don't they're making architectural decisions about the product or they're mentoring juniors. So like same deal within an open source project. I'm the open- source project equivalent of a principal engineer. I don't write code anymore. I make decisions about how we get this thing out the door. This episode is brought to you by cortex.io. Still tracking your services and production readiness in a spreadsheet. Real Microser is named after TV show characters. You aren't alone. Being woken up at 3:00 a.m. for an incident and trying to figure out who owns what service, that's no fun. Cortex is the internal developer portal that solves service ownership and accelerates the path to engineering excellence within minutes. Determine who owns each service with Cortex's AI service ownership model, even across thousands of repositories. Clear ownership means faster migrations, quicker resolutions to critical issues like lock forj and fewer adhere pings during incidents. Cortex is trusted by leading engineering organizations like Affirm, Trip Advisor, Grammarly, and Sofi. Solve service ownership and unlock your team's full potential with Cortex. Visit cortex.io/pragmatic to learn more. That is cotx.io/pragmatic. Yeah, I we we used to have a joke internally with within I think my team or my broader team back when I was a Uber of it just went around. What's the difference between uh a junior or an entry- level engineer, a senior engineer and a staff engineer? It's like you get a code review, entry- level engineer comes in and says, "Oh, you know, the order is not alphabeticized as per our our our rules like we should change the order of the variables." Senior engineer goes in and and and on the same color line leaves the comment actually we should probably move this out into its own class uh because it should be encapsulated. the staff engineering comes in. Why are we even doing this? There's a service that does exactly this thing. You should be using that one. Yeah. Don't re don't reinvent the wheel. and and and it and you know the thing goes back to that the earlier career you should you should focus on the code for sure but but after a while you need to see the bigger pictures and I often saw some of the most impactful engineers on my team uh in a given let's say quarter if I had to use the business value they did it not by writing code sometimes by deleting code or sometimes pointing out the work we should just avoid by I don't know like merging with something or just saying no pushing back on product etc etc etc yeah no it's skill for sure. Yeah, but you need it's a baseline, right? Like you need to know that, but there's a lot more afterwards. So in in in in Kubernetes, if if if I'm a a ## Using a KEP to propose a new feature—and what’s next [44:37] software or contributor who has an idea, contributing some code, what does that look in terms of release? How will it eventually get uh released? That's one part of my question. And the other other part is is does Kubernetes work in in a similar way to some other open source project where you kind of open up and you depend on what contributions are or do you have a road map saying hey we'd like to encourage people to build this functionality or or solve for this problem. Yeah. Yeah. So, Kubernetes uh like I mentioned earlier has a process called a ke um Kubernetes enhancement proposal. And if there is a feature that you want within Kubernetes um you go open a cap. Uh it's just just an issue in the Kubernetes enhancements proposal or enhancements repo and uh there's there's a template for it. You go open your cap and then a discussion will ensue with whatever SIG owns that. So um if it is something within um I don't know it's a if if it's a storage thing SIG storage will own that cap and they will come weigh in and we'll have a discussion about whether or not this is actually uh useful whether or not this is viable um whether or not we have the resources to like try to get this done right now which is going to depend on other contributors looking at your proposal and going oo yeah I would like to work on that and and when you say discussion is it email Is it async? Is it actually like video call? Uh it so because we're an open source project, we prefer to do everything in public in a place where everybody can see it. So that discussion will happen on the GitHub issue or in Slack uh in a public channel. We very much discourage things from happening behind closed doors unless it's very sensitive like extremely sensitive in a security type of way or uh it could like negatively impact somebody's career. So like code of conduct stuff is always handled behind closed doors. Um but discussions about features that is all very public GitHub issue or the slack channel for for instance sig storage. Um if the uh decision is yeah we would like to do this then work begins. Um your SIG lead will need to opt in. It's just adding a label um sig lead opt in to the cap and that tracks it for inclusion in a release at which point you have a whole bunch of deadlines that you have to hit. Um code freeze is the big one obviously that's is your code in then if it needs documentation or whatever there's documentation freeze there's test freeze these deadlines all exist across the span of 14 weeks and if you hit all of those marks your feature is included in the release uh as an alpha at first um we do allow iterative stages so you don't have to get like the whole thing MVP out the door with your first alpha, you can have an alpha, too. Um, alpha features are off by default, but can be turned on with a feature flag. Uh, once you meet certain requirements to graduate your feature to beta, like you're you're expecting mineral minimal architectural changes to the way it functions and interacts with the rest of Kubernetes, um, we'll graduate you to beta and you are on by default, but can be turned off with a feature flag. Um once it's very stable, very usable and you feel like you're ready to go GA, we gradually general availability, right? Yeah. Yeah. We move it to GA in the next release and it is on by default and cannot be disabled. And then by that time feature flag is removed, right? Yeah. There's no feature flag at that point. And how at any given point in ## Feature flags in Kubernetes [48:20] time roughly how many feature flags does Kubernetes have? Is it going to be in the it's going to be in the hundreds or thousands? Um, at minimum dozens I can go look. Um, let me share my screen and I will show you all of the feature gates we have. Love it. This is what I love about open source. You couldn't do this with commercial closed software. No, not at all. Um, so all of our feature gates are documented. This is usually all the documentation that an alpha requires cuz it's just like turning a thing off or on. Um uh you can you can see here what the default is. So whether it's functional whether it's alpha or beta uh when it was introduced and uh until when um some of these are you know deprecated and will be removed in the future. Some of them uh haven't been graduated since then but there are uh quite a lot. It doesn't look like hundreds but well it it looks several dozens. It it looks wow. And so so those are all features that are that are big enough to eventually some of them will graduate. Uh well hopefully most of them will graduate eventually. Yep. And these are all graduated or deprecated and will be removed at some point. Um yeah. So it's it's quite a lot. Uh we talk about what the feature stages mean here. Like really every every aspect of this project is is so well documented. Even the way the release team runs the way the the project is governed is all documented. We don't um keep anything as like tribal knowledge. Uh we we all we write everything down so that there's no arguing about it. But yeah, there's there's a ton of them. And and also you documenting this much, it just gives me the idea that if anyone wants to get a, you know, a bit of a a cheat of of like how a well-run project this size for forever inspiration. Yeah, you can you can just check it out. You can you can rip rip off the ideas that make sense and also you can adjust it for for your size or or for again you can experiment, right? I I I feel you always want to experiment but getting inspiration of something that works is a good starting point. Yeah, please do. I mean and and just like you know not not all products not all applications need to run on Kubernetes at first that that can very much be overkill um from an infrastructure standpoint not all open source projects need to be run like Kubernetes to begin with like there there are some things that are just like good practice that you should ingrain from the very beginning like if it's a userf facing change it needs documentation but you don't need like a structured release team the way we have it for your fourerson open source project that is releasing like one minor update a year, that's that's unnecessary and you've got like a couple dozen users, but for something the size of Kubernetes or close to the size of Kubernetes, yeah, that's super useful. Uh, rip us off. Absolutely rip us off. That's part of why we document it all. And it's ex Exactly. This this is beautiful like like having this this win and also you don't really need to explain it again and again. It's a point to do people at the docs. Yeah. the the way every role on the release team has to be run is exhaustively documented. So every every single role has a handbook that goes week by week and tells you what to do. Um which is part of why our releases are so smooth. Um if something goes wrong, probably it's documented and we we know how to solve it. Um of course every release we get something new and exciting that goes wrong to document, but uh that's happening less and less over time as we run out of edge cases. So now one one one thing I I'd like to ask you because I I'm really interested ## Why Kat thinks most GenAI tools are scams—and why Kubernetes blocks their use [52:04] on on how I was going. How do you think the Gen AI tools LLMs that more and more engineers are using to you know write code help them write code will impact how Kubernetes is developed? Could it because because it's so extensively documented on one end I'm sure you know like these tools can ingest a lot of things. I I wonder if they could be useful. are are you seeing them being useful in let's say uh using features or for the release team to build your own tools to do some of your work easier? Okay, so my personal opinion I'm going to stress the word personal there is that the overwhelming majority of Genai tools are a scam. Um they they are a scam. They are designed to filch money out of VCs or the public or whoever else. The the people who fund the fund the VCs, the the liquidity partner or LPS. Yeah. The ambassadors. I think I think most of them are a scam. Uh I think a very small minority of them are genuinely useful. Um and the rests are are just it's a nothing burger. Um, as a maintainer within SIG release, uh, we have no use for generative AI or or LLM tools. Most of the issues we are dealing with are people problems. Um, Gen AI and LLMs are not going to resolve that. We're we're people managers. That's what we're doing. Um, so generative AI not useful there. With my SIG docs hat on, um, we actually explicitly have a policy of you are not allowed to use tools like that. Um, we caught somebody submitting a blog that had been written with some some kind of chat GPT tooling and we pulled it and told them not to do that again. Uh, we got a driveby PR from a paid product that claimed to simplify um writing documentation, maintaining documentation and uh handling things like style guide compliance. Kubernetes does have a documentation style guide. Um, we got a drive by PR from them that stipulated, hey, if you accept this, you know, it's we grant you a free license for this tool if as long as we can use your logo. Um, but here's the thing, it didn't work. It was making changes to a bunch of generated files that a human should never touch. And it wasn't actually enforcing style guide compliance. It it made a bunch of mistakes. So even even when the tool sounds useful, um it it is often not like it doesn't actually do what it says it's doing on the tin. And also drive by PR um in exchange for a logo on product. That is that's really poor behavior. If there if you own a product and you're listening to this, do not do that, please. We hate it so much cuz it it creates more work for us cuz we have to have a discussion as leadership about what we're going to do with that. like you caused three meetings. Please don't do that to me. Um, however, where I do ## The use cases Kat would like to have AI tools for [55:04] want to see it is in things like um that are just toil for us. So things like application of of GitHub labels on issues or PRs. Like I would like something to automatically uh if if the wrong language label gets added to a PR, I would like a tool that removes that label and applies the correct linguistic label or the correct area label or it it can't it probably can't make assumptions about priority. But things like handling labeling is like that's that's really annoying work that we have to do on just about every PR. if um a new contributor doesn't understand how our labels work, they they may not apply any of them and somebody like me has to go in and do it. So tooling like that would be useful, but um we haven't we haven't found somebody um offering that to us yet. Yeah. Ju ju just to add to or just to add another perspective on the the the scam part or what I I understand where you're coming from by the way in terms of it is over like I I think it's overhyped in so many ways and I'm a little frustrated that we're we're talking so much about what it could do one day instead of what it is happening. I'm very much focused on how we're using it today. An interesting input I got from John Austerhow the author of Phil philosophy of software design. He's a professor at Stamford and uh he he is actually contributing to the Linux kernel. he's he's doing uh he's implementing a protocol that one of his PhD students came up with and he told me I asked him if he uses uh genai much and and he said he doesn't except he said it was really helpful for him to help start contributing to Linux because he said it's so under document but his crit his criticism was that there's so little documentation that he actually turned to chat GPC to try to make sense of it and it helped him and and his takeaway was he actually said I don't really want to popularize is but but I I wish they would have added some more comments because because then I could have done it without the tools. So I'm actually getting the sense of like Kubernetes is very well documented which will obviously help people who are using these tools to you know d to to to learn. I mean you can probably just go to the source to be fair. You can but like I totally get using something like chatbt uh to explain something complex to you like that that is a good use of an LLM to me. Yeah. Um, it's one of the most produc No, that's that's great. I don't think you should be using it to generate a blog article or generate documentation. You certainly shouldn't be using it to translate your docs. The Kubernetes documentation is localized into like a couple dozen different languages and that's all done by humans, right? Um, so I don't I do not think that you should be using generative AI to do that. But using it to explain a complex topic or like look at some code and give you an idea of where it where it's going, like what it does, that's useful. Like that's that's a great way to use generative AI to me. When do you think Kubernetes is a great choice to to use and when do you think it can actually be an overkill? I I I get this a lot where especially small startups are are thinking should we start with Kubernetes or will it just be such an infrastructure overkill that that we shouldn't use it. So where would you draw that line or how do you see teams making this decision? Um I would ## When to use Kubernetes [58:20] say that uh you should think about how quickly do you think you're going to need to scale rapidly like like really rapidly. Um, if you think that you're going to need to do that pretty abruptly, then I would use Kubernetes earlier. Um, because migrating over to it can be a pain. Um, what you need to really think about is not so much the is Kubernetes itself as an infrastructure tool overkill, it's can you afford the people you need to manage it correctly. uh a mismanaged Kubernetes cluster is a very dangerous thing. Kubernetes is not secure by default. Uh you you do require an expert. Um so my best piece of advice for running Kubernetes early is do not roll your own cluster. Don't do that. Uh do do not do that. use some kind of managed Kubernetes service that handles a lot of the work for you and then you know hire hire an S sur with experience with GKE or AKS or whatever tooling you're using. Uh but just don't don't roll your own cluster. Kubernetes is easier to use than it was before. I don't think uh it's it's as much overkill early as it used to be. Like obvious obviously if you're just running a blog or something putting that on Kubernetes is ridiculous. My personal website does run on Kubernetes, but that's it if it's a bit, right? Like I don't need it to, right? It's it's a bit it's a pet project. Yeah, it's a pet project. It's fun. It's it's easy for me to deploy something to a cluster. I can do that faster than I can do it with like EC2, right? But that's because that's where my experience is. For for most people, that's not true. So, uh run it on some kind of managed service. And if you think that there is a point where you are ever going to need to scale super fast, migrate to Kubernetes before that happens. So that I think the the addition there is is if if you need to scale super fast, you can always choose to just stay on a cloud provider and just pay massive amounts, you will be overcharged for it. So So I guess Kubernetes makes sense when you actually want to control your costs a lot more, right? Yeah. Because because it's scalable. Um it's not like it it scales up and down, right? Like your your costs aren't going to be consistent. Um and that's that can be can make things a little bit difficult to predict, but there are there are companies and tools that help you u manage that aspect of running Kubernetes, but uh it won't be consistently high at least because of that uh up and down scalability. But yeah, just do it do it earlier rather than later uh for me. But if it's just a blog, if it's just a website or whatever, it's just a web application, you probably do not need Kubernetes if it's just a web app. And so for for those who are interested in in getting started with Kubernetes for building their own things to as as a ## Getting started with Kubernetes [01:01:25] user, where would you recommend them to look? And and what about for people who are would be interested in dipping their toes in contributing? What's what's a good place to start for starting to use it? I uh very much recommend our own documentation and GKE's tutorials. GKE's tutorials come with a little sandbox so that you can actually do the thing which is yeah very helpful. That's the way I prefer to learn personally. I need to be told how to do the thing and then actually do it. Um that's not true for everybody but that uh was a really really great resource for me early on. And our own documentation has quick start guides for a variety of ways of getting going with uh with Kubernetes though I have always just used mini cube. Um there are many ways to get a cluster going and it'll walk you through most of them. If you would like to get started contributing to Kubernetes, um you can come talk to me in the Kubernetes Slack. Uh and I will be happy to help you find a place, but I will probably recommend that you start with documentation uh in an area that is of specific interest to you. So if you're particularly interested in storage or networking, I will hand you a storage or networking piece of documentation that I think needs updating. Um, if you like more of a people managey thing, I will be happy to talk to you about the Kubernetes release team. However, again, we get hundreds of applications and there's like 25 spots. So, I would not ride on that exclusively as a way to get involved in the project. But yeah, come talk to me on the Kubernetes Slack workspace uh cat cosgrove and I am more than happy to help you personally. Wonderful. And we've already talked I think about the the b the professional benefits that uh can easily come by being associated with a project like Kubernetes in terms of an active uh contributor or or maintainer. Yeah, it's uh it's definitely a thing. Um release team members put Kubernetes release team uh comm's shadow on their resume. It's on people's LinkedIn. If you look at the number of people on LinkedIn who say they work at Kubernetes, uh myself included. Mine says I work at Kubernetes. Um, you do, right? I I do. Yeah. Yeah, I maintain the thing. So, um, I do work there. But, yeah, like people people put it on their resumes. Um, it it's popular, especially if you're a student about to graduate or you're very early in your career. It can definitely give you a resume boost, but what it also gives you is access to a very, very large network of people with significantly more experience than you who are generally quite friendly. So, it's a really good networking opportunity. We all hang out a lot. We all talk to each other a lot. Um we right before CubeCon, our our big conference that we have twice a year, there is a maintainer summit and we all get together and hang out and talk to each other for a whole day. It's our own little miniature conference. So there there is also the networking opportunity there too. It's not just the the resume fodder. Yeah. And I I like that you say ## How contributing to an open source project is a good way to build your network [01:04:24] networking because I feel it's people are starting to realize engineers are starting to realize with the job market becoming bit bit more frosty that oh having a network could be useful and then there's a question of how do you get get a network and just my two cents is the best network I've ever had is it's not people who are kind of like you I know them with driveby it's people who I actually worked with like did something together and and you know you can either do it at your full-time job as assuming you have one or on a project where you spent extended time and solve the problems which a project like Kubernetes or a different open source project. Yeah, for sure. Like if I've if I've worked with you on the on the Kubernetes project for a little while, I am happy to connect you with anybody else I know. Like if you if you're looking for a job at at Docker or at AWS or Microsoft or whatever, I know executives there by virtue of having done this for so long. And I'm happy to make connections if I've worked with you for a while. I will not just like one PR is not and if if that person has actually just you know put in good solid work. So I think just one more thing if just don't do anything halfass just like yeah don't have you need a whole ass everything. Yeah I I I think the the more the earlier people realize that the the better. So so to to wrap up I've just uh asked some some rapid questions and I'll shoot and then you tell me how like what comes to mind. What is your favorite framework or ## Rapid fire round [01:05:51] programming language? Python, which I shouldn't say because I'm managing the Kubernetes project which is written in Go, but it's Python. That that was really fast. How come? Um, it was not the first language I learned, but it was one of the one of the earlier ones. And I like that even if it's not good at everything, I can make it do just about anything. Um, so that versatility is really valuable to me as a as a prototyping tool. So I will prototype something in Python and if Python is not like the best tool in my toolbox for that particular application, I'll then go actually rebuild it in like Go or whatever is is most um applicable. I'm I'm coming around to Python. I I I never really use Python. I use other languages, C, JavaScript, etc. But you know what's getting me around to to Python? It's now that I'm going to a lot of the getting started guides on, let's say, AI tools, etc. They usually have it in all languages like from from from Python. Usually Python is the first one. And what I've noticed is Python is the the the least verbose. It is and the easiest to read. And I just liked reading it. So now I just set up my my thing. So I'm I'm actually changing my mind on that slowly. I'm I'm starting to see the the poll. Yeah. You can almost just write pseudo code and that's like very close to valid Python. Yeah. Which which rules. So it's awesome. Yeah. You got to do something quick and dirty. drive on. What is a book that you would recommend? Well, um h so I don't read uh workbooks at all. Uh I I actually have a a personal policy of not reading uh books for anything other than fun like for I only read for pleasure and I read quite a lot. Um right now I'm rereading A Fire Upon the Deep by Verer Ving which is very weird, very um very very strange hard science fiction. Um it's one of my favorite books. I would I would absolutely recommend that if you like hard sci-fi. Well, well, Cat, thank you so much for going into the the details of I guess talking through the surprisingly open nature of of Kubernetes and very well doumented nature of it. Thank you. Happy to. I hope you enjoyed this deep dive into how Kubernetes is built, how releases work, and all the organizational work that goes into keeping a project like this running. I found it particularly interesting to learn just how well everything in Kubernetes is documented, including how the project is organized and run. If you ever find yourself inside a large project, feel free to steal ideas on how to organize things from Kubernetes. Thanks a lot for Cat for sharing to all these details. You can connect with her on the links listed in the show notes below. We previously did a deep dive into how Linux was built and into a few other related topics. You can find all of these the pragmatic engineer deep dives in the show notes linked below. If you've enjoyed the podcast, please do subscribe on your favorite podcast platform and on YouTube. This helps more people discover the podcast and a special thanks if you leave a rating. Thanks and see you in the next one.