This looks really cool!

I wrote a few applications using both the official Kubernetes client and the asyncio variant from Tomasz. Both have the same problems to me (that you exposed on the comparison page):

The documentation is really terrible (it's even non-existent for kubernetes-asyncio). For the official client, it's hidden deep inside the repository, across hundreds of files (592 at the moment); half the link cross doc don't work, there are rendering issues, etc. It's really a gigantic mess.

The API is extremely verbose, directly the underlying Kubernetes API exposed to Python. It gets the job done and it's better than shell script, but the verbosity + the total lack of a decent documentation (cf. above) makes it hard to use and see what you can do with it. Most of the time, you have to fire the REPL and try a bunch of calls until it makes sense. I like that most of the responses are proper objects, but they display as dict in the REPL and of course, the responses format are not documented in the doc :)

Last but not least: they both lack support for fancy authentication mechanism. We go to Kubernetes through a fancy proxy that require specific TLS settings + an external exec plugin to get the credentials. The official Kubernetes client should have the right support (for what we need at least) in the next major version but the asyncio variant not yet (I made a PR to fix this, but this requires a new version of aiohttp which is not released yet...) Both clients are very similar for the autogenerated code, but also subtly different for all the things around so you can't really expect 100% that what works on one will work on the other.

I'm not sure how kr8s works with this fancy auth system, but at least for the first 2 points, it seems like a huge win, I'll give it a try!

Awesome thanks! Kr8s supports the exec auth and TLS options. If you run into anything that’s missing though please raise an issue and we will get it added.

In the past I used PyKube (https://pykube.readthedocs.io/en/latest/readme.html), which did the job. One issue that pykube has is that there is no autocomplete on API object fields. Does kr8s wire this up? Seems you could use a TypedDict on newer versions of Python.

Back in the old days you wouldn't get autocomplete on k8s YAML files either, but now the schema is well-known and so your IDE will complete all the fields (even CRDs). These days you get this with Pulumi, if I'm not mistaken.

I’m not exactly sure what you mean but we can definitely improve auto completion if that’s valuable. Maybe you could try it out and give us some feedback in an issue?

Another way of putting it would be - are your object manifests typed according to the OpenAPI schema?

If we have:

    pod = Pod({
        "apiVersion": "v1",
        "kind": "Pod",
        "metadata": {
            "name": "my-pod",
        },
        "spec": {
            "containers": [{"name": "pause", "image": "gcr.io/google_containers/pause",}]
        },
    })

Is there a mypy type for the dict to tell you that only specific keys are valid, and `apiVersion` etc. must be present? And will I get a squiggly in my IDE if I type `apiversion`?

Not currently. You might want to check out Lightkube for this. You can use Lightkube models with kr8s too.

Requesting objects without a specific version/apigroup is pretty dangerous, you're effectively opting out of K8s' primary backwards compatibility measure.

Looks great, but I'm bashing my head trying to figure out what the r means in kr8s?

Nothing, they want it pronounced like "kraits". A krait is a type of sea snake, and Python is snake-themed.

To put one more option out there, we use Hikaru (https://pypi.org/project/hikaru/) in Robusta.dev (https://github.com/robusta-dev/robusta) and have been pretty happy with it. Example code below:

  with Pod().read(name='thename', namespace='the-namespace') as p:
      p.labels['new-label'] = 'value'

Does this read or also write?

I have literally just spent the last few weeks writing pydantic models for Kubernetes for an internal testing tool. I’ll see how this compares, it looks outstanding.

Thanks!

This is awesome. Jacob, your work on Dask Kubernetes has been immensely helpful to me. The world of distributed Python grows stronger!

Thanks!

The comparison doesn’t include the (perhaps confusingly named) openshift library. All the ansible kubernetes modules rely heavily on it because its support for dynamic client (where you just want to apply a manifest and don’t know in advance that it’s a Deployment and a Service and a ConfigMap) is first rate.

Wow I’ve not heard of that one! Do you have a link?

Haha, I meant to add one to my comment but struggled to do so on mobile

https://github.com/openshift/openshift-restclient-python

I used this in my KiwiPyCon conference talk yesterday! I used it with Kopf to write a basic little kubernetes operator in Python.

Wish I had seen this thread before my talk, but the YouTube video will be online soon and I can link to it when they are uploaded.

This looks great, will be giving it a go today to replace some increasingly complex bash scripts!

Why not just use the official python kubernetes client? https://github.com/kubernetes-client/python

I built kr8s because the official autogenerated library is extremely verbose, low level and hard to use.

Kr8s is modelled after kubectl instead of the REST API to try and be more accessible to developers.

These low-level constructs in the client libraries exist so that people can build controllers and operators on top of it, no? There's a lot that goes into making of implementing a Kubernetes client such as keeping up with semantic changes (dry run, server-side apply, tracing, rate limiting, patch support etc). Does your library also cover these?

Anything you can do with the official library you can do with kr8s. Some things will require writing low level HTTP requests if we haven’t implemented them yet, but it’s not so different to the official low level auto generated API.

The goal here is to make easy things easy, while ensuring anything is possible.

The official python client does not have feature parity with kubectl; I had to reimplement parts of kubectl in my own scripts, like the 'apply' and 'exec' subcommands and less-flaky CRD deployments.

Kr8s already has things like port forwarding and I’m working on apply right now. Exec is next on my list

here, take my code: https://github.com/dharmab/homelab-k3s/blob/2f46393aeeb73673...

Heh thanks. We are trying to avoid binary dependencies like kubectl because you can’t install the with pip.

that exec function is pure python, and the order of operations in that deploy function is important (initial deployment of resource out of order will likely either fail outright or cause certain Pods to fail to become read)

seconding the answer given by OP...I've also found the raw clients very low level and hard to use. I appreciate the higher level abstraction here, emphasis on "batteries included" and usability mirroring kubectl

Awesome! Let me know how you get on and don't hesitate to open issues if you run into problems or have thoughts about the API design.

Without opening a philosophical discussion on Python package management what were the pragmatic reasons from switching from Poetry to Hatch?

I'm about to run my first bit of kr8s code!

Poetry is a locking package manager. That’s extremely useful for applications where you don’t want dependencies to drift. However for a library you want pinning a to be looser so the benefits of poetry aren’t as strong. Given that hatch is the official method from the PyPA it felt like a better choice.

I stopped at the 3rd example. Kr8s and lightkube are the only solutions I'd dare to use.

Lightkube is also awesome. The focus of kr8s is on simplicity, similarity to kubectl and having batteries-included. Lightkube focuses on type safety and API correctness.

They also work together and you can cast Lightkube objects to kr8s and back again.

This is delightful. I'll be taking it for a spin today.

Thanks! Don’t hesitate to open issues if you find bugs or have feedback :)

Awesome library, but I can't view the documentation at work since the doc site uses http. Please consider using https.

The docs does use https. Let me know where you found an http link so I can fix it.

https://docs.kr8s.org/en/latest/

Interesting, I'm seeing an invalid certificate warning when I visit the site, it might be on my end. Awesome project nonetheless!

does this provide sane default behavior for refreshing expired auth tokens and retrying random client timeouts? the "official" SDK has driven me absolutely crazy trying to wrap all the different pieces with retries so that basic long-running operations won't randomly raise spurious exceptions...

It will retry expired auth but backing off the timeout is a good idea!

How do you connect remotely? The examples don't easily show this.

The same way you do with kubectl, you configure your cluster in your kube config file. We can definitely add more examples though!

The Use case I have, is to use this within Jupyter notebooks for Devops / deployment journaling. Having something natively python that maps to actual commands is a huge bonus here.

There is sufficient contextual padding that I feel this is not a problem.

I am unlikely to be talking about Rust crates anywhere near the same conversation as kr8s code.

Yup, naming things is hard!

How do you pronounce it?

Krewbernetes?

Krates?

Nice :)

The cycle of tooling

Step 1: Look I've made this new great tool that uses a declarative language, you simply declare what you want your environment to look like and boom it will build it out. No more having to figure out a complicated logic flows that can fail or having code that dies on weird error cases, and you don't have to be a developer to use it.

Step 2: The tool isn't powerful or flexible enough and the declarative code is getting to cumbersome so we've introduced some basic templating and logic flow to this tool to make it more flexible and reduce bloat.

Step 3: The control flow isn't powerful enough to really meet our needs so we have moved it entirely to a full programming language.

Step 4: Go back to step 1.

It's often better to provide both - especially for something like k8s. Declarative languages and programming languages serve entirely different use cases.

They no more preclude each other than garbage collected and non garbage collected programming languages like C and python do.

To be fair... k8s offers both:

Write declarative code with yaml and kubectl. This solves most use cases and interactions.

Write operators in a variety of languages (usually golang) if you need direct control of the underlying control loop, advanced resources, and other use cases to extend or customize how your cluster works. This is far more flexible and powerful than most people understand and of itself has layers ranging from moderate to very complex.

Furthermore there is the api aggregation layer you can extend if you need to customize the core api of your cluster.

Steps 1,2,3 create separate solutions that people can use as they see fit. You make it seem like there is no place for anything but a declarative programming language in this area.