Show HN: Koreo – A platform engineering toolkit for Kubernetes
koreo.devA large part of our (Real Kinetic's) business is helping organizations implement platform engineering, but we've found the existing tooling to be lacking. For IaC, Terraform state becomes a pain because TF treats infrastructure as "one-shot" commands. The Kubernetes controller model provides a nicer approach to managing infrastructure, but the tooling here is also lacking. For configuration management, Helm just doesn't really scale with complexity, nor does Kustomize. For resource orchestration, Crossplane is a step in the right direction but still has challenges and limitations.
As a result, we ended up building something that's sort of a "meta-controller programming language" on top of Kubernetes called Koreo. It provides a solution for configuration management and resource orchestration in Kubernetes by basically letting you program controllers. We've been using Koreo for a while now to build internal developer platform capabilities for our commercial product and our clients, and we recently open sourced it to share it with the community.
Koreo has some similarities to configuration languages like KCL, Jsonnet, etc. since it is a means of configuration management (e.g. you can define base configurations, apply overlays, point patches, and so forth). Where it really diverges though is Koreo provides a unified approach to config management and resource orchestration. This means you can start to treat Kubernetes resources as "legos" to build pretty sophisticated workflows. For instance, the output of a resource can be used as the input to another resource. This isn't really possible with Helm, even with `lookup` because `lookup` requires the resource to already be in-cluster in order to reference it.
This is why we refer to Koreo as a meta-controller programming language because it effectively lets you program and compose Kubernetes controllers into cohesive platforms—either built-in controllers (think Deployment or StatefulSet), off-the-shelf ones such as AWS ACK or GCP's Config Connector, or custom operators. It lets you build or combine controllers without actually needing to implement an operator. Through this lens, Koreo is really more akin to Crossplane but without some of the limitations such as Providers and cluster-scoped managed resources.
It seems crazy and maybe it is, but I've found working in Koreo to actually be surprisingly fun since it kind of turns Kubernetes primitives into legos you can easily piece together, reuse, and build some pretty cool automated workflows. You can learn more about the motivation and thinking behind it here: https://theyamlengineer.com
I'd love to offer a few words of encouragement for this excellent project. This space only appears crowded to those that have already been here for a while. To everyone else, this might just be the most obvious thing to adopt for Kubernetes and they might not even have heard of Kustomize/Helm. So I would suggest two separate personas to target in the docs and communication: 1) Jaded devs that need to be convinced with detailed proof that Koreo isn't yet-another YAML tool. 2) Everyone else, to whom the pitch is that Koreo is the most obvious choice when the need is A, B, or C.
Looking forward to trying this out soon!
I also think more high level patterns would make Koreo more approachable. Real world problems and how they could be addressed with Koreo.
Thank you! Very helpful feedback.
We're thinking through better examples to try and highlight some of the key differences from other tools without it being too involved.
Really great feedback! Better examples and docs is definitely something high on the list.
I feel like it needs a comparison with https://kro.run/ and crossplane v2 which makes it more generic and less cluster scoped.
I thought Crossplane v2 was only a design proposal at the moment: https://github.com/crossplane/crossplane/pull/6255
But I guess there is an actual preview implementation now? https://docs.crossplane.io/v2.0-preview/
The comparison on Kro would definitely be good to include as there are quite a few similarities. I can write up more on how it compares in a bit.
Koreo and Kro share a lot of similarities in that both allow you to build abstractions that encapsulate a lot of complexity. For instance, in Kro you could implement a ResourceGraphDefinition that builds a "Workload" abstraction that produces a Workload CRD that, say, lets you specify a container image, a database, and a bucket. Then when you create an instance of this CRD, Kro might map this to a Lambda function, RDS instance, and an S3 bucket, perhaps using ACK for example. In Koreo, this would be a Workflow that has various ResourceFunctions which produce the Lambda, RDS, and S3 bucket. Just like with Kro, this would be triggered off of a CRD. In essence, both let a platform team (or whoever) provide high-level APIs that encapsulate resource management.
One difference is just in how the two approach doing this. Koreo takes an approach of providing primitives that can be composed or reused and, importantly, are actually testable (since testing is a first-class thing in Koreo). This lets you more easily validate automations but also makes it easier to provide "building block" like components that can be shared between Workflows.
Another difference is in how Koreo solves configuration management. Rather than relying on string templating or unstructured YAML overlays, Koreo treats configuration as structured data. This allows you to specify and tweak configurations in a predictable and typesafe way by transforming, validating, and composing them programmatically. Koreo is very much modeled after functional programming principles, so we can, for instance, define functions that validate preconditions or apply standard tags to resources in an environment. This model also enables configuration reuse and overrides across teams and environments without introducing tight coupling or duplication. Instead, we can apply configuration "layers" to build up a resource. Kro really focuses more on resource orchestration and leaves the configuration management up to the user.
Also, what's interesting about Koreo is that you can actually build "meta" Workflows. That is, Workflows that produce Workflows, meaning you could implement something like Kro itself in Koreo fairly easily. This is why it's really closer to a programming language and runtime for programming Kubernetes control loops.
Koreo would also allow you to implement a workload spec such as Score rather easily for the same reason. https://score.dev
Yes, those were exactly my thoughts after reading the introduction. It seems the new meta is adding another meta (heh) level on top of Kubernetes, and kro seems to be the most promising at the moment.
Just one more YAML bro I swear trust me bro just one more meta level will solve everything just one more yaml please I promise you it's not bad just write yaml it's simple and clean just some more yaml man just one more yaml file you will certainly not regret it just add one more yaml
(Congratulations on the launch, looks interesting!)
I wrote the core engine behind this and your comment made me laugh pretty hard because I agree. I debated a lot about the best syntax and we experimented with various alternatives approaches. The core engine actually isn't YAML centric at all, in fact it doesn't use YAML—it is just data structures in and out.
We wound up exposing this as YAML purely because there's a lot of tooling out there for dealing with YAML. I am not sure if that's a good reason or not.... Writing the language server was quite nasty because of that choice! Our hope was that the language server and structural type checking (which is very simplistic at the moment) and testing framework can make it feel more like a real language and less YAMLy.
I'm glad it made you laugh, I wrote it in good faith :) Congratulations again on the launch!
A toolkit for a toolkit for a toolkit.
Its toolkits all the way down.
I’ve been really interested in understanding how these things work better, but I’m having a lot of trouble understanding the devex / UX. I _think_ the idea behind Koreo is that any user can upload some resource definition, and koreo essentially shores it up on the backend. If I say I want an s3 bucket, Kori will look at my custom resource, then maybe if I have a label a that says it’s a temporary bucket it adds yams to my resource to add a 30 day lifecycle policy to delete the resource.
But I’m not sure I feel the advantage of this indirection. It feels confusing to be that the applied resource will be different from what’s in VCS, and the code feels super heavy for what you’re getting. I’ve been at like this and cross plane, and can’t quite grok why this is better than doing it in a classical programming language. But I think I’m wrong, can you help me understand?
I maintain the core of Koreo. I'll hit the easy question first, then the more involved questions.
Everything is in-code and designed to have a "proper" SDLC lifecycle—code reviews, approvals, merges. It is designed to be used in gitops workflows.
For the more nuanced questions, here's some background: I like Kustomize and KPT a lot. In _my_ opinion, they should be your starting point. They are clean and easy to reason about tools. They do not work as well when you have more complexity. They're very painful if you've got _dynamic_ values or values you need to inject programmatically (think Helm's values.yaml).
The next important item to note: Koreo's relative value is lower if you're building highly bespoke one-offs or you do not care about having standard resource configurations / application architectures. The value is not zero, but there are lighter solutions and you should consider them instead.
Koreo is meant to model application architectures and resource capabilities. Using your example, you can build a BucketResource. That BucketResource will then ensure that S3 Buckets follow your company standards, including things like automatically handling the IAM setup and permissions for the service that uses the bucket. That lets you define required capabilities: An S3 Bucket is always tagged with the owner service and product domain. In production environments buckets must have lifecycle rules specifying a minimum 30 day retention. In development environments, you lifecycle rules are optional. The developers then only need to specify that their workload uses an S3 Bucket and it will be configured based on your company standards. But, we have designed it so that you can decide how much abstraction is right for your needs—you can directly expose the full, underlying API or you can abstract it more.
Effectively, it gives you an "easy" solution for building a PaaS that implements _your_ standards and opinionation.
Our original versions were directly implemented using go and Python. The issue is that iterating on the application models was much, much slower. This approach allows us to rapidly implement new capabilities and features, and even expose unique or experimental architectures to only certain application domains.
Definitely excited to explore how it simplifies complex workflows—turning K8s components into “legos” is a neat concept
I feel I've built something similar using fluxcd + cuelang. FluxCD allows having order through depends on and how you organize your Kustomization resource. I still believe the FluxCD project needs a UI that matches what you can get from the cli. CLI has so many features that you might or might not get fully from the available UI's.
Agreed. I'm hoping the old Weaveworks GUI makes a comeback in the near future under the new organisation, as it looked to do a better job than anything else I've seen so far.
why k8s ecosystem is so filled with yaml on-top-of-yaml, configuration on-top-of-configuration? where does this end? can we have less yaml, less configuration, not more?
The underlying engine is actually completely decoupled from yaml as it's really just structured data. We chose to stick with yaml for the "interface" since that is ultimately what it's managing—Kubernetes manifests. Personally, I tend to prefer keeping the configuration true to the underlying thing being configured rather than abstracting it away with DSLs, but I realize it's not for everyone.
I think since the complexity is high and you cannot run away from that other than making a very small box that not a lot of use cases fit into. The main use case of k8s is handling more complex use cases so you'd remove a large portion of your audience by constraining the config.
Agreed. YAML was a great upgrade to traditional ini files, but its usage beyond any non-static config use case was a mistake. Not just k8s configs, but also ansible (Chef's ruby was so much better, chef just lost because it required an agent), and GitHub actions.
I think I can see some of where this could be utilized, but I think I'm still missing a step and I'm hopeful someone can fill me in.
There's a comparison against Argo Workflows, but with the description here and in other comments, Koreo seems to be aiming more for what I would use Argo CD for - managing the entire state of the cluster, the controllers, configuration, etc. Because of it tying into repos, you can then define the entire state of your cluster in code, and Argo CD has tools for doing some of the interpolation of variables into your YAML.
The project looks cool, and I don't think that the world suffers from having multiple ways of doing something, I just want to understand it better.
My understanding of Argo CD is limited so forgive me but I believe Koreo is a bit more similar to Argo Workflows. Where the scripts and logic is baked into the Argo Workflow image, in Koreo it leverages other K8s controllers to do the heavy lifting of interacting with the APIs. If I understand Argo CD correctly, it is an implementation of Argo Workflows. In this way, you are able to build your own CI/CD workflow via Koreo if you so wanted.
Since I'm Ops type so I took a look. Here are my thoughts in random order.
Templating systems are always frustrating, and I couldn't find CLI to spit out exactly what I was going to get. Kustomize ability to build exactly what cluster is going to consume is one of those features you miss when you don't have it.
Tying it to Kubernetes is both good and bad. Alot of companies use Kubernetes so for those companies, this is great. Downside is I think many companies are not ready to deal with complexity of Kubernetes so system that could put them outside of it might be great. That's just taste I guess.
This is a crowded field so good luck I guess.
Finally, the problem here everyone is trying to solve is skill gap and that's hard to fix with technology. Most devs are bad at Ops and that's where friction comes. It's like watching Product Owner develop their feature using LowCode or AI. It works until it doesn't and when it doesn't, here we go. I also realize few companies want us around since they see as pure money sink.
Most of my frustration around building platforms is lack of communication. Most of it due to developers not understanding or just not thinking about it (See skill issue above) so Ops is forced to put in something ugly to get them into Prod at 11 hour so we don't get tossed under the bus.
Awesome feedback, thank you!
The CLI is presently being reworked to expose that capability. Right now, you can use FunctionTest in order to validate that you get your expected outputs from an input. This works well because you can test many scenarios, including error conditions. The CLI does not currently emit the materialized manifests purely because our initial use cases needed to map values from reconciled manifests into other manifests. It is completely viable to emit the materialized manifests though.
Honestly, our ambition is to develop a tool that makes the operations and platform engineering people's lives better. Our team is comprised of software engineers who've worked in the operations and platform engineering space for a long time. It is a crowded space for sure, but I am optimistic, probably foolishly so, that we can develop something that is more pleasant for the platform folks than many of the other options.
We'd welcome any other feedback or thoughts on how to accomplish that.
> The CLI does not currently emit the materialized manifests purely because our initial use cases needed to map values from reconciled manifests into other manifests. It is completely viable to emit the materialized manifests though.
Only in cases where the values are statically known, however. If you have resources that depend on the output of another resource, then we can't know that at "template time" as you pointed out.
We use Python to generate these configs at my work. Ends up working out pretty well. I previously worked on the biggest deployment of gcl (the inspiration for KCL, Jsonnet) at Google and it was a giant nightmare and the cause of many outages.
Agreed. I used json-e back in the day, and have been using pulumi more recently, though my pulumi configs are simpler and I'm not sure how well it scales to really complex workloads/job graphs.
https://igor.moomers.org/posts/building-etl-kubernetes
> Koreo is the engine that powers Konfigurate, a batteries-included developer platform for startups and scaleups.
Yaml is no go for me.
Gradually typed languages, with support of unkown values, like Nickel can be good.
But if Kore starts where Kustomize ends (selectors like, inheritance like, binding like things) and covers 100% of Kustomize. But builds on top (i see CEL usage). I can be intresting to use this instead of Kustomize.
If we're going to ditch Kustomize, let's ditch it for something that isn't YAML, because Kustomize has the virtue of being established.
twitter thread,
> However, the two tools are not mutually exclusive. In some cases it can make sense to use Kustomize in combination with Koreo!
Would be awesome Koreo make Kustomize no need. I feel insane getting into project where Helm, Kustomize and Koreo used at same time(they always do this).
So if Koreo and Helm only, would be nice gradual step. One bad yaml with other bad yaml.
Honestly, Koreo can cover most of the use cases of the other tools just fine. We leverage Koreo to manage deployments of the Koreo controllers along with their service accounts and RBAC—it works quite well. We also use it with Helm in order to install, for example, ACK controllers which have existing Helm Charts.
Our design is very, very deeply inspired by Kustomize. If Kustomize or kpt are sufficient, I would strongly encourage using one of them. We developed Koreo because we have cases where we need to sequence resource creation (for instance to map values from one resource to another) and wanted to programmatically set values.
Just get cdktf to manage the YAML, you may laugh but it's not so horrible as it sounds.
I feel like I should just be able to generate k8s configs from Typescript with Typeconf if I add all the types there, is anyone interested in this?
Yes. CDK has made me a fan of Typescript.
Timoni ticked so many boxes for me. Hoe does this compare?
I am not honestly familiar with Timoni, but glancing through the docs it looks like it heavily centered around Kubernetes deployments? Our initial use cases were centered around IaC for non-k8s workloads (think systems such as ECS or Cloud Run).
In terms of the YAML aspect, internally the core system does not use YAML. We opted to stick with YAML as the interface for the time being purely because it is familiar and there's loads of tooling.
We've been working to improve the structural and the type checking further and I'm hopeful to have a substantial improvement to that released "soon." It is queued up behind some other improvements within the language server we need to make first though.
It's YAML.
Here's the tension I find with projects like these: App developer knowledge seems to end with a helm chart. Anything more complex than that, they won't be able to deliver themselves. For platform/k8s admins, these tools are more cumbersome than just writing a dedicated operator in go.
What advantages does this offer over rolling my own CRD and operator? Assume it takes me 4 hours to write an operator end to end.
You should write your operator. You are correct: it is quite straightforward and you'll have solid understanding of what is happening and how it works.
That is how this project started. We wrote custom operators for the various platform components; that worked outstandingly well. Shockingly well to be honest.
This system evolved from that approach because we needed a way to rapidly customize our application models. If you don't need to support varied use cases _and_ have standardization writing your own is the way to go.
Where Koreo excels is when your developer knowledge ends with a Dockerfile and your platform team is able to abstract service deployment away from them. Many organizations use a "god chart" to do such a thing but those will become cumbersome over time. Koreo's value prop also is greatly enhanced when paired with ACK or Config Connector as you have an organization specific DSL that will integrate your container runtime with your cloud resources. E.g. mount your rds address on your pod in one workflow.
Why half of Kubernetes and AWS look like technogies to support... themselves instead of shipping a new product such as LXC/LXD?
Most of the current technologies can be virtualized, and with LVM snapshots are a breeze, even extending media it's perfectly done. There's no need to use half-backed namespaces when kernel-level deduplition for memory pages exist when you run similar parallel VM's.
I find virtualisation far easier than containers. Not as fast to deploy, sure; but far more manageable for rollbacks.