100daysofhomelab

Chartmuseum Repository continued ... (Day 29)

Vince

• 2 min read
Chartmuseum Repository continued ... (Day 29)
Photo by Markos Mant / Unsplash

Continuing from the previous ChartMuseum setup entry, today was about solving the DNS resolution and network connectivity issues (some of which had to do with my firewall blocking traffic across certain VLANs).

The DNS Challenge

Yesterday ended with this error:

DEBUG Fetching chart list from storage
ERROR RequestError: send request failed
caused by: Get "https://helm-charts.minio-s3.home.mrdvince.me/?prefix=": 
dial tcp: lookup helm-charts.minio-s3.home.mrdvince.me on 10.96.0.10:53: no such host

The error revealed two important things:

  1. ChartMuseum was trying to access helm-charts.minio-s3.home.mrdvince.me (bucket name + endpoint)
  2. Kubernetes CoreDNS (10.96.0.10) couldn't resolve this hostname (expected and makes sense)

S3/minio has two URL addressing styles:

  1. Path-style: https://endpoint/bucket-name/
  2. Virtual-hosted style: https://bucket-name.endpoint/

I had DNS set up for the virtual-hosted style (as shown by the dig output):

$ dig helm-charts.minio-s3.home.mrdvince.me
...
;; ANSWER SECTION:
helm-charts.minio-s3.home.mrdvince.me. 10 IN A 192.168.50.10

But Kubernetes pods couldn't access my home DNS server so the solution:

  1. Configure CoreDNS to forward queries for my domain to my home DNS:
e.g
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        # existing config...
    }
    home.mrdvince.me:53 {
        forward . 192.168.50.120
    }
  1. Fallback to Direct IP

I went with option 2 after trying the CoreDNS approach and realizing it broke some syncs on argo based on how my DNS rewrites are set up on Adguard (something to fix another day)

However, this did bring up a TLS verification error:

caused by: Get "https://192.168.50.190/helm-charts?prefix=": tls: failed to verify certificate: 
x509: cannot validate certificate for 192.168.50.190 because it doesn't contain any IP SANs

Looking at the comments, and issues on Chart Museum's repo I found setting AWS_INSECURE_SKIP_VERIFY to true allows skipping self-signed certificate verifications.

Final config values:

- args:
    - --port=8080
    - --storage-amazon-endpoint=https://192.168.50.190
    - --storage-amazon-force-path-style=true
    - --disable-api=false
    - --debug
- env:
    - name: STORAGE
      value: "amazon"
    - name: STORAGE_AMAZON_BUCKET
      value: "helm-charts"
    - name: STORAGE_AMAZON_PREFIX
      value: ""
    - name: STORAGE_AMAZON_REGION
      value: "eu-west-1"
    - name: AWS_INSECURE_SKIP_VERIFY
      value: "true"
- envFrom:
    - secretRef:
        name: minio-chartmuseum-secret

Using the Helm Repository

Now with everything working, I can manage my Helm charts using:

# Install the push plugin
helm plugin install https://github.com/chartmuseum/helm-push

# Add the private repository
helm repo add local-charts https://chartmuseum.atlas.home.mrdvince.me

# Package and push a chart
helm package ./chartmuseum
helm cm-push ./chartmuseum-0.1.0.tgz local-charts

# Or push directly from directory
helm cm-push ./chartmuseum local-charts

# Update and search
helm repo update
helm search repo local-charts

Next Steps

  • Set up automated chart builds with CI/CD
  • Potentially try to implement chart testing before publishing