I have a small k3s cluster that i use for testing and learning kubernetes. I have an hasicorp vault issue that I cannot solve on my own.
My current helm deployment of hashicorp vault has TLS and UI enabled. I also defined an ingress in the value-override.yml When connecting to the DNS name that points to my k3s cluster, I receive a "bad gateway" error and in the logs from the vault pod, the following error:
2025-01-29T12:16:18.833Z [INFO] http: TLS handshake error from 10.42.0.52:46042: remote error: tls: bad certificate
2025-01-29T12:16:19.659Z [INFO] http: TLS handshake error from 10.42.0.52:58742: remote error: tls: bad certificate
2025-01-29T12:16:19.952Z [INFO] http: TLS handshake error from 10.42.0.52:58752: remote error: tls: bad certificate
2025-01-29T12:16:20.157Z [INFO] http: TLS handshake error from 10.42.0.52:58766: remote error: tls: bad certificate
So, I can connect to the vault UI via regular HTTPS without port specification, but It goes wrong somewhere between the ingress and service/pod. I would like to avoid using LB for this because of the mandatory port usage after the node IP address. When defining a LB and I list the ports for it, it works.
My helm override yml for vault values looks like this:
global:
namespace: vault-dev
enabled: true
tlsDisable: false
server:
ingress:
enabled: true
ingressClassName: "traefik" # Use Traefik as the ingress controller
annotations:
traefik.ingress.kubernetes.io/router.entrypoints: "websecure"
traefik.ingress.kubernetes.io/router.tls: "true"
hosts:
- host: "vault.home.arpa" # Replace with your domain
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: vault # Placeholder for the Vault service name
port:
number: 8200
tls:
- secretName: ingress-secret # Ensure this secret exists with your SSL certs
hosts:
- "vault.home.arpa"
extraEnvironmentVars:
VAULT_CACERT: /vault/userconfig/vault-server-tls/vault.ca
VAULT_SKIP_VERIFY: "true"
volumes:
- name: userconfig-vault-server-tls
secret:
defaultMode: 420
secretName: vault-server-tls # Matches the ${SECRET_NAME} from above
volumeMounts:
- mountPath: /vault/userconfig/vault-server-tls
name: userconfig-vault-server-tls
readOnly: true
standalone:
enabled: true
config: |
ui = true
listener "tcp" {
tls_disable = 0
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
tls_key_file = "/vault/userconfig/vault-server-tls/vault.key"
tls_client_ca_file = "/vault/userconfig/vault-server-tls/vault.ca"
}
storage "file" {
path = "/vault/data"
}
ui:
enabled: true
# serviceType: LoadBalancer
The TLS certificates were created as described here as an example for creating a self signed CA for the k3s cluster and for everything inside the vault. When I exec into the vault, I can see the certificates.
My guess is that the ingress is not servicing the correct vault crt/key as it should, but i'm out of idea's and knowledge on where to troubleshoot.
The running config of the ingress service:
$ kubectl describe ingress vault
Name: vault
Labels: app.kubernetes.io/instance=vault
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=vault
helm.sh/chart=vault-0.29.1
Namespace: vault-dev
Address: 192.168.178.211,192.168.178.212,192.168.178.213
Ingress Class: traefik
Default backend: <default>
TLS:
ingress-secret terminates vault.home.arpa
Rules:
Host Path Backends
---- ---- --------
vault.home.arpa
/ vault:8200 (10.42.3.73:8200)
Annotations: meta.helm.sh/release-name: vault
meta.helm.sh/release-namespace: vault-dev
traefik.ingress.kubernetes.io/router.entrypoints: websecure
traefik.ingress.kubernetes.io/router.tls: true
Events: <none>
The ingress-secret for TLS/KEY
$ kubectl describe secrets ingress-secret
Name: ingress-secret
Namespace: vault-dev
Labels: io.portainer.kubernetes.configuration.owner=administrator
io.portainer.kubernetes.configuration.owner.id=1
Annotations: <none>
Type: kubernetes.io/tls
Data
====
tls.key: 1708 bytes
tls.crt: 1211 bytes