Johannes Schnatterer's Avatar

Johannes Schnatterer

@schnatterer.info.bsky.social

Software engineer, author, speaker. Field CTO of Cloudogu. My particular interests are #k8s, #GitOps, #PlatformEngineering, #o11y, #IaC, #DevOps technical leadership and of course #FLOSS/ #FOSS/ #OSS. I like owning my data and devices.

19 Followers  |  14 Following  |  30 Posts  |  Joined: 22.11.2024  |  2.3585

Latest posts by schnatterer.info on Bluesky

XKCD 2347 comes to mind, but I don't feel like laughing.

13.08.2025 20:15 โ€” ๐Ÿ‘ 0    ๐Ÿ” 0    ๐Ÿ’ฌ 0    ๐Ÿ“Œ 0
Preview
Health of External Secrets project ยท Issue #5084 ยท external-secrets/external-secrets Update 2: OMG thank you all for signing up. We weren't expecting such a positive response from the community <3 Update We've decided to stop releases until more long-term maintainers join our team....

An important building block of many cloud-native architectures just had to stop maintenance:
#ExternalSecretsOperator #ESO.

github.com/external-sec...

The unfortunate fate of so many open source projects: We all use them, but we don't support them. Maintainers burn out.

13.08.2025 20:15 โ€” ๐Ÿ‘ 0    ๐Ÿ” 0    ๐Ÿ’ฌ 1    ๐Ÿ“Œ 0
Preview
Cloudogu Platform More flexibility for your software tools.

Facing challenges with air-gapped envs or multi-tenant architectures?
I'd love to hear from you! Schedule here:
๐Ÿ—“๏ธ platform.cloudogu.com/person/johan...

15.05.2025 09:48 โ€” ๐Ÿ‘ 1    ๐Ÿ” 0    ๐Ÿ’ฌ 0    ๐Ÿ“Œ 0

Our goal is to provide a simple UX to platform admins:
Add one file to git, a new tenant gets rolled out.
Change one version number in git, all tenants get upgraded.

#GitOps at it's best!

3/x

15.05.2025 09:48 โ€” ๐Ÿ‘ 1    ๐Ÿ” 0    ๐Ÿ’ฌ 1    ๐Ÿ“Œ 0

The next big thing we are working on is the option to role out dedicated instances per tenant, that are managed centrally.

Kind of like an IDP as a Service (Is #IDPaaS a thing? ๐Ÿ˜…)

2/x

15.05.2025 09:48 โ€” ๐Ÿ‘ 0    ๐Ÿ” 0    ๐Ÿ’ฌ 1    ๐Ÿ“Œ 0
Diagram showing all parts of GOP
K8s-Cluster containing Vault, External Secrets Operator, Grafana, Prometheus, Ingress-nginx, Argo CD.
Next to it, Cloudogu Ecosystem containing SCM-Manager and Jenkins.

Air-Gapped presenting the challenges:
Argo CD: No fetching helm charts from the internet
K8s Cluster: No fetching images from the internet
Jenkins: No fetching plugins, images, libs, packages, build packs, maven wrapper, etc.
from the internet

Diagram showing all parts of GOP K8s-Cluster containing Vault, External Secrets Operator, Grafana, Prometheus, Ingress-nginx, Argo CD. Next to it, Cloudogu Ecosystem containing SCM-Manager and Jenkins. Air-Gapped presenting the challenges: Argo CD: No fetching helm charts from the internet K8s Cluster: No fetching images from the internet Jenkins: No fetching plugins, images, libs, packages, build packs, maven wrapper, etc. from the internet

Diagram depicting Multi-tenant Architecture. 
One Management Cluster/NS and two Tenant Cluster/NS.
Inside of each are Management/Tenant Git Repos and Cluster Resources: Argo CD, Vault, Grafana, External Secrets Operator.
Inside the Management Cluster/NS is also GOP.
Inste the Tenanat Cluster/NS are also Tenant Applications written in Typescritp, PHP, Python, NodeJS, C#, GO or Java.

Arrows show a sequence:
1. Platform Admin Creates, Updates or Deletes Tenant in Management Git Repos
2. Management Argo CD Reads Management Git Repos
3. Argo CD Runs GOP
4. GOP inits Management and Tenant Repos
5. Management Argo Deploys Cluster Resources
6. Tenant Argo CDs read Tenant Git Repos
7. Tenant Argo CDs deploy Tenant Applications

Diagram depicting Multi-tenant Architecture. One Management Cluster/NS and two Tenant Cluster/NS. Inside of each are Management/Tenant Git Repos and Cluster Resources: Argo CD, Vault, Grafana, External Secrets Operator. Inside the Management Cluster/NS is also GOP. Inste the Tenanat Cluster/NS are also Tenant Applications written in Typescritp, PHP, Python, NodeJS, C#, GO or Java. Arrows show a sequence: 1. Platform Admin Creates, Updates or Deletes Tenant in Management Git Repos 2. Management Argo CD Reads Management Git Repos 3. Argo CD Runs GOP 4. GOP inits Management and Tenant Repos 5. Management Argo Deploys Cluster Resources 6. Tenant Argo CDs read Tenant Git Repos 7. Tenant Argo CDs deploy Tenant Applications

#GitOpsPlayground (GOP) version 0.11.0 finally facilitates running in air-gapped environments:

It can provide standardized #IDPโ€‹s,
even when are they are #airgapped,
even when they run on #OpenShift.
๐Ÿฅณ

github.com/cloudogu/git...

1/x

15.05.2025 09:48 โ€” ๐Ÿ‘ 2    ๐Ÿ” 0    ๐Ÿ’ฌ 1    ๐Ÿ“Œ 0
Preview
GitHub - argoproj-labs/gitops-promoter: A GitOps Environment Promotion Tool A GitOps Environment Promotion Tool. Contribute to argoproj-labs/gitops-promoter development by creating an account on GitHub.

The central elements are the PromotionStrategy and CommitStatus CRDs.

This has the potential to replace promotion CI pipelines ๐Ÿš€

github.com/argoproj-lab...

04.04.2025 08:15 โ€” ๐Ÿ‘ 0    ๐Ÿ” 0    ๐Ÿ’ฌ 0    ๐Ÿ“Œ 0
No More Pipelines: Reconciling Environment Promotion Via Commit Statuses W... M. Crenshaw & Z. Aller
YouTube video by CNCF [Cloud Native Computing Foundation] No More Pipelines: Reconciling Environment Promotion Via Commit Statuses W... M. Crenshaw & Z. Aller

ArgoCon: Upgrade on the future experience of UI promotion

@crenshaw-dev.bsky.social and Zach Aller give a sneak peek on how automatic promotion between envs could look like in the future with Argo Project CD.

www.youtube.com/watch?v=Usi3...

04.04.2025 08:15 โ€” ๐Ÿ‘ 0    ๐Ÿ” 0    ๐Ÿ’ฌ 1    ๐Ÿ“Œ 0
Preview
GitHub - argoproj-labs/gitops-promoter: A GitOps Environment Promotion Tool A GitOps Environment Promotion Tool. Contribute to argoproj-labs/gitops-promoter development by creating an account on GitHub.

The central elements are the PromotionStrategy and CommitStatus CRDs.

This has the potential to replace promotion CI pipelines ๐Ÿš€

github.com/argoproj-lab...

04.04.2025 08:13 โ€” ๐Ÿ‘ 0    ๐Ÿ” 0    ๐Ÿ’ฌ 0    ๐Ÿ“Œ 0

Here is an example
github.com/prometheus-c...

24.03.2025 14:37 โ€” ๐Ÿ‘ 0    ๐Ÿ” 0    ๐Ÿ’ฌ 0    ๐Ÿ“Œ 0

While contributing to the kube-prometheus-stack chart last year, #TDD helped me to grasp input and output, especially when dealing with several values.

I would start using helm-unittest for every new helm chart project.

Does anyone have different experiences? Are there any contending frameworks?

24.03.2025 14:37 โ€” ๐Ÿ‘ 0    ๐Ÿ” 0    ๐Ÿ’ฌ 1    ๐Ÿ“Œ 0
Terminal emulator showing this command:
helm unittest . -f  unittests/prometheus/scrape_config_selector_test.yaml 

And its output

 PASS  test scrapeConfigSelector        unittests/prometheus/scrape_config_selector_test.yaml

Charts:      1 passed, 1 total
Test Suites: 1 passed, 1 total
Tests:       4 passed, 4 total
Snapshot:    0 passed, 0 total
Time:        152.612251ms

Terminal emulator showing this command: helm unittest . -f unittests/prometheus/scrape_config_selector_test.yaml And its output PASS test scrapeConfigSelector unittests/prometheus/scrape_config_selector_test.yaml Charts: 1 passed, 1 total Test Suites: 1 passed, 1 total Tests: 4 passed, 4 total Snapshot: 0 passed, 0 total Time: 152.612251ms

YAML definition of a Helm unit test.
Test file can be found at this URL
https://github.com/prometheus-community/helm-charts/blob/1251b0/charts/kube-prometheus-stack/unittests/prometheus/scrape_config_selector_test.yaml

YAML definition of a Helm unit test. Test file can be found at this URL https://github.com/prometheus-community/helm-charts/blob/1251b0/charts/kube-prometheus-stack/unittests/prometheus/scrape_config_selector_test.yaml

Who writes #unittests for #helm #charts? ๐Ÿ™‹

Same as for code, they speed up development and prevent recursions.

The helm-unittest plugin makes them easy to use.
github.com/helm-unittes...

I found helm-unittest easy to read, write and execute. See for yourself ๐Ÿ‘‡๏ธ

24.03.2025 14:37 โ€” ๐Ÿ‘ 0    ๐Ÿ” 0    ๐Ÿ’ฌ 1    ๐Ÿ“Œ 0

For our next release after 2025030800, we've added support for the Android 15 QPR2 Terminal for running other operating systems using hardware virtualization. It's currently only a terminal but Android is adding support for graphics and GPU acceleration for a future release.

09.03.2025 14:27 โ€” ๐Ÿ‘ 67    ๐Ÿ” 7    ๐Ÿ’ฌ 1    ๐Ÿ“Œ 2

Du hast spannende Neuigkeiten oder mรถchtest Deine Erfahrungen teilen โ€“ rund um #PlatformEngineering #DeveloperExperience & Co?
Dann freuen wir uns auf Deine Vorschlรคge beim #CallforProposals fรผr die #CLC_Conf im November!

@ixmagazin.bsky.social @dpunkt.bsky.social

28.02.2025 13:52 โ€” ๐Ÿ‘ 2    ๐Ÿ” 1    ๐Ÿ’ฌ 0    ๐Ÿ“Œ 0
Preview
Helm Chart should support setting annotations-risk-level in configmap ยท Issue #12618 ยท kubernetes/ingress-nginx With controller 1.12 / chart 4.12, annotations seem to be validated differently, breaking our setup, because we use nginx.ingress.kubernetes.io/configuration-snippet annotations, which are rated cr...


See here for details:

github.com/kubernetes/i...

13.01.2025 17:03 โ€” ๐Ÿ‘ 0    ๐Ÿ” 0    ๐Ÿ’ฌ 0    ๐Ÿ“Œ 0
An example for blocking access to a path via ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-metrics-blocker
  namespace: my-ns
  labels:
  annotations:
    acme.cert-manager.io/http01-ingress-class: nginx
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
    # Allow access from within the cluster only
    nginx.ingress.kubernetes.io/whitelist-source-range: 10.0.0.0/8
spec:
  tls:
    - hosts:
        - "example.com"
      secretName: example-certificate
  rules:
    - host: "example.com"
      http:
        paths:
          - path: /metrics
            pathType: Prefix
            backend:
              service:
                name: my-svc
                port:
                  number: 8080

An example for blocking access to a path via ingress apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: example-metrics-blocker namespace: my-ns labels: annotations: acme.cert-manager.io/http01-ingress-class: nginx kubernetes.io/ingress.class: nginx kubernetes.io/tls-acme: "true" # Allow access from within the cluster only nginx.ingress.kubernetes.io/whitelist-source-range: 10.0.0.0/8 spec: tls: - hosts: - "example.com" secretName: example-certificate rules: - host: "example.com" http: paths: - path: /metrics pathType: Prefix backend: service: name: my-svc port: number: 8080

A more secure alternative would be to get rid of the annotation.

The best I could come up with is an additional ingress that only allows access to the path from within the cluster.

๐—ช๐—ต๐—ผ ๐—ฐ๐—ฎ๐—ป ๐—ฝ๐—ฟ๐—ผ๐˜ƒ๐—ถ๐—ฑ๐—ฒ ๐—ฎ๐—ป ๐—ฒ๐—ฎ๐˜€๐—ถ๐—ฒ๐—ฟ ๐˜€๐—ผ๐—น๐˜‚๐˜๐—ถ๐—ผ๐—ป?

13.01.2025 17:03 โ€” ๐Ÿ‘ 0    ๐Ÿ” 0    ๐Ÿ’ฌ 1    ๐Ÿ“Œ 0


An insecure workaround is to allow these annotations via values.yaml:

```
allowSnippetAnnotations: true
config:
annotations-risk-level: Critical
```

13.01.2025 17:03 โ€” ๐Ÿ‘ 0    ๐Ÿ” 0    ๐Ÿ’ฌ 1    ๐Ÿ“Œ 0

Heads up #ingressNginx users!

Controller version 1.12 / chart 4.12, contains breaking changes.

Even though a minor release, one change blocks risky annotations like the 'snippet' annotations.

These are often used to block access to specific URLs, like /metrics.

13.01.2025 17:03 โ€” ๐Ÿ‘ 0    ๐Ÿ” 0    ๐Ÿ’ฌ 1    ๐Ÿ“Œ 0
Preview
GitHub ยท Build and ship software on a single, collaborative platform Join the world's most widely adopted, AI-powered developer platform where millions of developers, businesses, and the largest open source community build software that advances humanity.

UPDATE

No, the charts were deleted accidentally and are now back on.
github.com/artifacthub/...

13.01.2025 12:25 โ€” ๐Ÿ‘ 0    ๐Ÿ” 0    ๐Ÿ’ฌ 0    ๐Ÿ“Œ 0
Preview
Artifact Hub Find, install and publish Cloud Native packages

Nothing to be found in the announcements
github.com/bitnami/char...
blog.bitnami.com/2024/11/bitn...

ArtifactHub Page of Bitnami organization:
artifacthub.io/packages/sea...

ArtifactHub Page of Helm Chart bitnami/mongodb: artifacthub.io/packages/hel...

13.01.2025 09:53 โ€” ๐Ÿ‘ 0    ๐Ÿ” 0    ๐Ÿ’ฌ 1    ๐Ÿ“Œ 0
ArtifactHub Page of Helm Chart bitnami/mongodb showing: "Sorry, the package you requested was not found."

ArtifactHub Page of Helm Chart bitnami/mongodb showing: "Sorry, the package you requested was not found."

ArtifactHub Page of Bitnami organization, showing only one reamining helm chart.

ArtifactHub Page of Bitnami organization, showing only one reamining helm chart.

Wait, did #Bitnami remove their charts from #ArtifactHub? ๐Ÿ˜ฑ

I understand they launched premium, introducing rate limit and DockerHub and stopped supporting non-LTS versions in their free tier.

But leaving ArtifactHub? Why?

Can someone share insights here?

13.01.2025 09:53 โ€” ๐Ÿ‘ 0    ๐Ÿ” 0    ๐Ÿ’ฌ 1    ๐Ÿ“Œ 0

๐Ÿ’ก Sidenote: I have to add on top of the rate limits, Bitnami also stops maintaining LTS branches, so if you're using tools like Postgresql MongoDB you'll only get support for the latest major version

13.01.2025 08:14 โ€” ๐Ÿ‘ 0    ๐Ÿ” 0    ๐Ÿ’ฌ 0    ๐Ÿ“Œ 0
Preview
2025: Increasing GitOps productivity Entdecken Sie das deutsche GitOps Buch. Lernen Sie Best Practices fรผr Continuous Deployment, Kubernetes und sichere GitOps Workflows kennen.

I'm curious about what the rather quick stint through the Through of Dillusion ahead will cause.

Here are my predictions for 2025:
gitops-book.dev/blog/2025-01...

What are your predictions or expectations for 2025 in terms of GitOps?

Did I miss any progress on promotion UX in the flux ecosystem?

09.01.2025 14:32 โ€” ๐Ÿ‘ 0    ๐Ÿ” 0    ๐Ÿ’ฌ 0    ๐Ÿ“Œ 0
Gartner Hype Cycle Diagram for Emerging Technologies 2024, showing, among other things, GitOps almost in the center, almost at the top of the Peak of inflated expectations, color-coded in white, which means Plateau will be reached in <2 years.

Gartner Hype Cycle Diagram for Emerging Technologies 2024, showing, among other things, GitOps almost in the center, almost at the top of the Peak of inflated expectations, color-coded in white, which means Plateau will be reached in <2 years.

Gartner Hype Cycle Diagram for Emerging Technologies 2023, showing, among other things, GitOps in the lower left corner, in the innovation trigger section, color-coded in light blue, which means Plateau will be reached in 2-5 years.

Gartner Hype Cycle Diagram for Emerging Technologies 2023, showing, among other things, GitOps in the lower left corner, in the innovation trigger section, color-coded in light blue, which means Plateau will be reached in 2-5 years.

Gartner Hype Cycle Diagram for Software Engineering 2022, showing, among other things, GitOps in the lower left corner, in the innovation trigger section, color coded in light blue, which means Plateau will be reached in 2-5 years.

Gartner Hype Cycle Diagram for Software Engineering 2022, showing, among other things, GitOps in the lower left corner, in the innovation trigger section, color coded in light blue, which means Plateau will be reached in 2-5 years.

Gartner Hype Cycle Diagram for Agile and DevOps 2021, showing, among other things, GitOps in the lower left corner, in the innovation trigger section, color coded in light blue, which means Plateau will be reached in 2-5 years.

Gartner Hype Cycle Diagram for Agile and DevOps 2021, showing, among other things, GitOps in the lower left corner, in the innovation trigger section, color coded in light blue, which means Plateau will be reached in 2-5 years.

In 2024 #GitOps reached the peak of inflated expectations.

It could be reaching the plateau of productivity in less than two years, so Gartner predicts.

That's some major progress after being stuck in the lower left corner for as far as I can remember. Which is 2021 ๐Ÿ˜…

09.01.2025 14:32 โ€” ๐Ÿ‘ 0    ๐Ÿ” 0    ๐Ÿ’ฌ 1    ๐Ÿ“Œ 0

Authenticate with DockerHub

And of course, sign up for bitnami premium

08.01.2025 10:24 โ€” ๐Ÿ‘ 0    ๐Ÿ” 0    ๐Ÿ’ฌ 1    ๐Ÿ“Œ 0
Preview
Artifact Registry documentation ย |ย  Google Cloud A universal package manager for all your build artifacts and dependencies. Fast, scalable, reliable and secure.

Use a pull through mirror for images, like mirror.gcr.io, which is automatically in place in GKE, for example but can be configured to be used locally. See cloud.google.com/artifact-reg...

Use official Charts and Images if possible.
For example, there is the registry.k8s.io/kubectl since 1.28.0

08.01.2025 10:24 โ€” ๐Ÿ‘ 0    ๐Ÿ” 0    ๐Ÿ’ฌ 1    ๐Ÿ“Œ 0

This impacts their Docker images and might impact their Helm Charts as well.

What can we do about it?

Pull charts from this "registry" (index.yaml at GitHub) httpsโ€‹://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami

08.01.2025 10:24 โ€” ๐Ÿ‘ 0    ๐Ÿ” 0    ๐Ÿ’ฌ 1    ๐Ÿ“Œ 0
bitnami by VMware Tanzu logo

bitnami by VMware Tanzu logo

#Bitnami rate limit for DockerHub became effective on January 6. What to do about it?

On rather short notice, bitnami introduced rate limits on their DockerHub Account, "due to business and partner requirements".

github.com/bitnami/cont...

08.01.2025 10:24 โ€” ๐Ÿ‘ 0    ๐Ÿ” 0    ๐Ÿ’ฌ 1    ๐Ÿ“Œ 0
GitOps Safety: Rendering Accurate ArgoCD Diffs Directly on... Dag Bjerre Andersen & Regina Voloshin
YouTube video by CNCF [Cloud Native Computing Foundation] GitOps Safety: Rendering Accurate ArgoCD Diffs Directly on... Dag Bjerre Andersen & Regina Voloshin

argocd-diff-preview generates diffs using ephemeral #k8s clusters that can then be easily commented on PRs and basically integrates with all CI and SCM systems.
๐Ÿ“ฝ๏ธ www.youtube.com/watch?v=3aeP...

4/4

22.11.2024 16:46 โ€” ๐Ÿ‘ 0    ๐Ÿ” 0    ๐Ÿ’ฌ 0    ๐Ÿ“Œ 0
Space Age GitOps: Lifting off with Argo Promotions (Live Demo!) - Michael Crenshaw & Zach Aller
YouTube video by CNCF [Cloud Native Computing Foundation] Space Age GitOps: Lifting off with Argo Promotions (Live Demo!) - Michael Crenshaw & Zach Aller

gitops-promoter, still in development in argo labs, together with a new hydrating phase inside Argo CD, might bring merging and Pull Request (for now with GItHub only) integration into Argo CD.
๐Ÿ“ฝ๏ธ www.youtube.com/watch?v=2JmL...

3/

22.11.2024 16:46 โ€” ๐Ÿ‘ 0    ๐Ÿ” 0    ๐Ÿ’ฌ 1    ๐Ÿ“Œ 0

@schnatterer.info is following 14 prominent accounts