Private Helm Chart Repository with ChartMuseum (Day 28)

I've been accumulating custom Helm charts, and figured I could use something with functionality to docker registries, a place that gives me:

• Some kind of versioning
• Central place to distribute charts
• Avoid some manual inconsistencies while updating charts

The Solution: ChartMuseum

I decided to set up a self-hosted Helm chart repository using:

• Minio as a S3-compatible storage alternative
• ChartMuseum as the Helm repository server
• Traefik (as always) for service routing

ChartMuseum artifact hub Chart (Initial Attempt)

The initial plan was simple, use the available ChartMuseum chart with Minio as the backend and override the following values:

env:
  open:
    STORAGE: amazon
    STORAGE_AMAZON_BUCKET: helm-charts
    STORAGE_AMAZON_PREFIX: ""
    STORAGE_AMAZON_REGION: eu-west-1
    STORAGE_AMAZON_ENDPOINT: https://minio-s3.home.mrdvince.me 
    AWS_SDK_LOAD_CONFIG: "1"
    DISABLE_API: false
  # Trying to use existing secrets
  existingSecret: minio-secret
  existingSecretMappings:
    AWS_ACCESS_KEY_ID: minio_access_key
    AWS_SECRET_ACCESS_KEY: minio_secret_key
extraArgs:
  - --storage-amazon-force-path-style=true

But this immediately ran into issues:

1. The chart's existingSecretMappings mechanism was finicky
2. The secret injection wasn't working as expected
3. Logs showed credential errors despite having the correct secret:

ERROR: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors

Building a Custom Chart

After hours of troubleshooting, I decided to build a simpler, custom deployment. I ran helm create chartmuseum which starts you off with a template chart.

Here's the basic structure:

├── Chart.yaml
├── rendered-app.yaml
├── templates
│   ├── NOTES.txt
│   ├── _helpers.tpl
│   ├── deployment.yaml
│   ├── hpa.yaml
│   ├── ingress.yaml
│   └── service.yaml
└── values.yaml

The Configuration

After several iterations, here's a rendered deployment snippet of the configuration I landed on:

# Source: chartmuseum/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: chartmuseum
  labels:
    helm.sh/chart: chartmuseum-0.1.0
    app.kubernetes.io/name: chartmuseum
    app.kubernetes.io/instance: chartmuseum
    app.kubernetes.io/version: "v0.16.2"
    app.kubernetes.io/managed-by: Helm
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: chartmuseum
      app.kubernetes.io/instance: chartmuseum
  template:
    metadata:
      labels:
        helm.sh/chart: chartmuseum-0.1.0
        app.kubernetes.io/name: chartmuseum
        app.kubernetes.io/instance: chartmuseum
        app.kubernetes.io/version: "v0.16.2"
        app.kubernetes.io/managed-by: Helm
    spec:
      containers:
        - name: chartmuseum
          args:
            - --port=8080
            - --storage-amazon-endpoint=https://minio-s3.home.mrdvince.me
            - --storage-amazon-force-path-style=false
            - --disable-api=false
            - --debug
          image: "ghcr.io/helm/chartmuseum:v0.16.2"
          imagePullPolicy: IfNotPresent
          env:
            - name: STORAGE
              value: amazon
            - name: STORAGE_AMAZON_BUCKET
              value: helm-charts
            - name: STORAGE_AMAZON_PREFIX
              value: ""
            - name: STORAGE_AMAZON_REGION
              value: eu-west-1
          envFrom:
            - secretRef:
                name: minio-secret
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP

But when attempting to connect to Minio, I hit more errors:

DEBUG Fetching chart list from storage
ERROR RequestError: send request failed
caused by: Get "https://helm-charts.minio-s3.home.mrdvince.me/?prefix=": 
dial tcp: lookup helm-charts.minio-s3.home.mrdvince.me on 10.96.0.10:53: no such host

And well, this was where I left things at the end of the Day - with a DNS resolution issue to solve the next day.