¶ Description:
This exercise is to test using self-signed certificates with cert-manager and how Aerospike interacts with rotated certificates. We will setup 2 separate issuers; one to handle the CA side, and the other dedicated for client certificates. The client certs will be signed by our CA cert, and configured to be rotated every 15 minutes. The CA cert is configured to have a duration of 1 year and will be rotated 2/3 of the duration by default (8 months in this example).
¶ Create the cluster in GAIA:
¶ Deploy the AKO via OLM:
If you’re using the latest gaia-eks.sh console script then the deploy-ako-olm script should automatically be in the docker image.
If it is NOT then you can manually clone from here: https://github.com/colton-aerospike/deploy-olm-ako/tree/master
./setup_olm.sh -ncolton -v1.29¶ Install cert-manager repo and CRDs:
If helm is not already installed in the environment you can find it here: https://helm.sh/docs/intro/install/
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --set installCRDs=true¶ Setup Issuers and Certificates
issuer_for_ca.yaml
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: custom-ca-issuer
namespace: aerospike
spec:
selfSigned: {}cert_for_ca.yaml
# cert_for_ca.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: aerospike-ca
namespace: aerospike
spec:
secretName: aerospike-ca-cert
duration: 8760h # 5 years
issuerRef:
kind: Issuer
name: custom-ca-issuer
commonName: "Aerospike CA"
isCA: trueissuer_for_clients.yaml
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: aerospike-client-issuer
namespace: aerospike
spec:
ca:
secretName: aerospike-ca-certcert_for_clients.yaml
# cert_for_clients.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: aerospike-client-tls
namespace: aerospike
spec:
secretName: aerospike-client-tls
duration: 1h # Correct the comment if duration is intended to be different
renewBefore: 45m
issuerRef:
kind: Issuer
name: aerospike-client-issuer
dnsNames:
- aerocluster.aerospike.svc.cluster.local
- aerocluster
kubectl apply -f issuer_for_ca.yaml
kubectl apply -f cert_for_ca.yaml
kubectl apply -f issuer_for_clients.yaml
kubectl apply -f cert_for_clients.yaml¶ Deploy the Aerospike Cluster:
The AerospikeCluster Yaml:
apiVersion: asdb.aerospike.com/v1
kind: AerospikeCluster
metadata:
name: aerocluster
namespace: aerospike
spec:
size: 3
image: aerospike/aerospike-server-enterprise:7.0.0.6
operatorClientCert:
secretCertSource:
caCertsSource:
secretName: aerospike-ca-cert
secretName: aerospike-client-tls
secretNamespace: aerospike
clientCertFilename: tls.crt
clientKeyFilename: tls.key
storage:
cleanupThreads: 10
filesystemVolumePolicy:
initMethod: deleteFiles
cascadeDelete: true
blockVolumePolicy:
cascadeDelete: true
initMethod: dd
volumes:
- name: workdir
aerospike:
path: /opt/aerospike
source:
persistentVolume:
storageClass: gp2
volumeMode: Filesystem
size: 10Gi
- name: aerospike-config-secret
source:
secret:
secretName: aerospike-secret
aerospike:
path: /etc/aerospike/secret
- name: tls
source:
secret:
secretName: aerospike-client-tls
aerospike:
path: /etc/aerospike/tls
- name: tls-ca
source:
secret:
secretName: aerospike-ca-cert
aerospike:
path: /etc/aerospike/tls/ca
podSpec:
aerospikeInitContainer:
resources:
limits:
cpu: "16"
memory: 32Gi
requests:
cpu: "1"
memory: 1Gi
metadata:
annotations:
example: annotation
multiPodPerHost: true
sidecars:
- name: exporter
image: aerospike/aerospike-prometheus-exporter:latest
ports:
- containerPort: 9145
name: exporter
env:
- name: "AS_AUTH_USER"
value: "admin"
- name: "AS_AUTH_PASSWORD"
value: "admin123"
- name: "AS_AUTH_MODE"
value: "internal"
rackConfig:
namespaces:
- test
racks:
- id: 1
- id: 2
- id: 3
aerospikeAccessControl:
users:
- name: admin
secretName: auth-secret
roles:
- sys-admin
- user-admin
- read-write
- sindex-admin
aerospikeConfig:
service:
feature-key-file: /etc/aerospike/secret/features.conf
security: {}
network:
service:
port: 3000
tls-port: 4333
tls-name: aerocluster
fabric:
port: 3001
tls-port: 3011
tls-name: aerocluster
heartbeat:
port: 3002
tls-port: 3012
tls-name: aerocluster
tls:
- name: aerocluster
cert-file: /etc/aerospike/tls/tls.crt
key-file: /etc/aerospike/tls/tls.key
ca-file: /etc/aerospike/tls/ca/ca.crt
namespaces:
- name: test
replication-factor: 2
nsup-period: 120
strong-consistency: true
storage-engine:
type: device
files:
- /opt/aerospike/test1.data
filesize: 1073741824kubectl apply -f aerospike_strong_consistency_ebs_withTLS.yaml¶ Connect to the cluster:
The below will create a temporary pod which mounts the same secret to access the TLS files in order to connect to the cluster.
Full example output:
kubectl run -it --rm --restart=Never aerospike-tools -n aerospike \
--image=aerospike/aerospike-tools:latest \
--overrides='
{
"apiVersion": "v1",
"spec": {
"containers": [
{
"name": "aerospike-tools",
"image": "aerospike/aerospike-tools:latest",
"tty": true,
"stdin": true,
"args": ["asadm", "-h", "aerocluster.aerospike.svc.cluster.local:4333", "-U", "admin", "-P", "admin123", "--tls-enable", "--tls-name", "aerocluster", "--tls-cafile", "/etc/aerospike/ssl/ca.crt", "--tls-certfile", "/etc/aerospike/ssl/tls.crt", "--tls-keyfile", "/etc/aerospike/ssl/tls.key"],
"volumeMounts": [
{
"mountPath": "/etc/aerospike/ssl",
"name": "tls-vol"
}
]
}
],
"volumes": [
{
"name": "tls-vol",
"secret": {
"secretName": "aerospike-client-tls"
}
}
]
}
}'
kubectl run -it --rm --restart=Never aerospike-tools -n aerospike \
--image=aerospike/aerospike-tools:latest \
--overrides='
{
"apiVersion": "v1",
"spec": {
"containers": [
{
"name": "aerospike-tools",
"image": "aerospike/aerospike-tools:latest",
"tty": true,
"stdin": true,
"args": ["asadm", "-h", "aerocluster.aerospike.svc.cluster.local:4333", "-U", "admin", "-P", "admin123", "--tls-enable", "--tls-name", "aerocluster", "--tls-cafile", "/etc/aerospike/ssl/ca.crt", "--tls-certfile", "/etc/aerospike/ssl/tls.crt", "--tls-keyfile", "/etc/aerospike/ssl/tls.key"],
"volumeMounts": [
{
"mountPath": "/etc/aerospike/ssl",
"name": "tls-vol"
}
]
}
],
"volumes": [
{
"name": "tls-vol",
"secret": {
"secretName": "aerospike-client-tls"
}
}
]
}
}'¶ Validate Certificates:
The below commands will extract the dedicate self-signed CA cert and client certificate into local files. We can then run various openssl commands to validate the authenticity of the certs and ensure they’re signed correctly by what we expect.
❯ kubectl get secret aerospike-ca-cert -n aerospike -o jsonpath="{.data['tls\.crt']}" | base64 --decode > ca.crt
❯ kubectl get secret aerospike-client-tls -n aerospike -o jsonpath="{.data['tls\.crt']}" | base64 --decode > client.crt
❯ openssl verify -CAfile ca.crt client.crt
client.crt: OK
❯ openssl x509 -in ca.crt -noout -subject
subject=CN = Aerospike CA
❯ openssl x509 -in client.crt -noout -text | grep -A1 'Authority Key Identifier'
X509v3 Authority Key Identifier:
04:02:91:6B:25:6C:AE:B4:60:45:43:D2:A9:EA:C6:12:B2:96:C2:3E
❯ openssl x509 -in ca.crt -noout -text | grep -A1 'Subject Key Identifier'
X509v3 Subject Key Identifier:
04:02:91:6B:25:6C:AE:B4:60:45:43:D2:A9:EA:C6:12:B2:96:C2:3E¶ Monitor cert-manager pods for rotations:
kubectl logs -n cert-manager $(kubectl get pods -n cert-manager -l app=cert-manager -o name) -fExpected logs during successful rotation:
I0415 14:05:48.001765 1 trigger_controller.go:215] "Certificate must be re-issued" logger="cert-manager.certificates-trigger" key="aerospike/aerospike-client-tls" reason="Renewing" message="Renewing certificate as renewal was scheduled at 2024-04-15 14:05:48 +0000 UTC"
I0415 14:05:48.001824 1 conditions.go:203] Setting lastTransitionTime for Certificate "aerospike-client-tls" condition "Issuing" to 2024-04-15 14:05:48.0018166 +0000 UTC m=+257190.171006678
I0415 14:05:48.047786 1 controller.go:162] "re-queuing item due to optimistic locking on resource" logger="cert-manager.certificates-key-manager" key="aerospike/aerospike-client-tls" error="Operation cannot be fulfilled on certificates.cert-manager.io \"aerospike-client-tls\": the object has been modified; please apply your changes to the latest version and try again"
I0415 14:05:48.069973 1 conditions.go:263] Setting lastTransitionTime for CertificateRequest "aerospike-client-tls-4" condition "Approved" to 2024-04-15 14:05:48.06991436 +0000 UTC m=+257190.239104438
I0415 14:05:48.094461 1 conditions.go:263] Setting lastTransitionTime for CertificateRequest "aerospike-client-tls-4" condition "Ready" to 2024-04-15 14:05:48.094447164 +0000 UTC m=+257190.263637232
I0415 14:05:48.137206 1 controller.go:162] "re-queuing item due to optimistic locking on resource" logger="cert-manager.certificates-readiness" key="aerospike/aerospike-client-tls" error="Operation cannot be fulfilled on certificates.cert-manager.io \"aerospike-client-tls\": the object has been modified; please apply your changes to the latest version and try again"
I0415 14:05:48.153795 1 controller.go:162] "re-queuing item due to optimistic locking on resource" logger="cert-manager.certificates-key-manager" key="aerospike/aerospike-client-tls" error="Operation cannot be fulfilled on certificates.cert-manager.io \"aerospike-client-tls\": the object has been modified; please apply your changes to the latest version and try again"¶ View Details of Certificate kinds:
We can describe both certificates in one command. If we review the below we can see a few things:
- The CA cert is set to have a duration of 1 year and will renew on 2024-12-14T21:13:43Z
- The client cert has a duration of 1 hour and will be renewed 45 minutes beforehand. This translates to being rotated every 15 minutes.
- The client cert has been rotated 4 times since its been configured initially
❯ kubectl get cert -naerospike
NAME READY SECRET AGE
aerospike-ca True aerospike-ca-cert 53m
aerospike-client-tls True aerospike-client-tls 45m❯ kubectl describe cert -naerospike
Name: aerospike-ca
Namespace: aerospike
Labels: <none>
Annotations: <none>
API Version: cert-manager.io/v1
Kind: Certificate
Metadata:
Creation Timestamp: 2024-04-15T13:13:42Z
Generation: 1
Resource Version: 1035802
UID: 9215c181-8ab9-47a3-a276-41b54e1f2eb1
Spec:
Common Name: Aerospike CA
Duration: 8760h
Is CA: true
Issuer Ref:
Kind: Issuer
Name: custom-ca-issuer
Secret Name: aerospike-ca-cert
Status:
Conditions:
Last Transition Time: 2024-04-15T13:13:43Z
Message: Certificate is up to date and has not expired
Observed Generation: 1
Reason: Ready
Status: True
Type: Ready
Not After: 2025-04-15T13:13:43Z
Not Before: 2024-04-15T13:13:43Z
Renewal Time: 2024-12-14T21:13:43Z
Revision: 1
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Issuing 52m cert-manager-certificates-trigger Issuing certificate as Secret does not exist
Normal Generated 52m cert-manager-certificates-key-manager Stored new private key in temporary Secret resource "aerospike-ca-d7kxh"
Normal Requested 52m cert-manager-certificates-request-manager Created new CertificateRequest resource "aerospike-ca-1"
Normal Issuing 52m cert-manager-certificates-issuing The certificate has been successfully issued
Name: aerospike-client-tls
Namespace: aerospike
Labels: <none>
Annotations: <none>
API Version: cert-manager.io/v1
Kind: Certificate
Metadata:
Creation Timestamp: 2024-04-15T13:20:47Z
Generation: 1
Resource Version: 1048885
UID: 89b0931d-50e4-4aed-8ee1-b4cca539678c
Spec:
Dns Names:
aerocluster.aerospike.svc.cluster.local
aerocluster
Duration: 1h
Issuer Ref:
Kind: Issuer
Name: aerospike-client-issuer
Renew Before: 45m
Secret Name: aerospike-client-tls
Status:
Conditions:
Last Transition Time: 2024-04-15T13:20:48Z
Message: Certificate is up to date and has not expired
Observed Generation: 1
Reason: Ready
Status: True
Type: Ready
Not After: 2024-04-15T15:05:48Z
Not Before: 2024-04-15T14:05:48Z
Renewal Time: 2024-04-15T14:20:48Z
Revision: 4
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Issuing 45m cert-manager-certificates-trigger Issuing certificate as Secret does not exist
Normal Generated 45m cert-manager-certificates-key-manager Stored new private key in temporary Secret resource "aerospike-client-tls-p7ssx"
Normal Requested 45m cert-manager-certificates-request-manager Created new CertificateRequest resource "aerospike-client-tls-1"
Normal Issuing 30m cert-manager-certificates-trigger Renewing certificate as renewal was scheduled at 2024-04-15 13:35:48 +0000 UTC
Normal Requested 30m cert-manager-certificates-request-manager Created new CertificateRequest resource "aerospike-client-tls-2"
Normal Issuing 15m cert-manager-certificates-trigger Renewing certificate as renewal was scheduled at 2024-04-15 13:50:48 +0000 UTC
Normal Requested 15m cert-manager-certificates-request-manager Created new CertificateRequest resource "aerospike-client-tls-3"
Normal Issuing 48s (x4 over 45m) cert-manager-certificates-issuing The certificate has been successfully issued
Normal Reused 48s (x3 over 30m) cert-manager-certificates-key-manager Reusing private key stored in existing Secret resource "aerospike-client-tls"
Normal Issuing 48s cert-manager-certificates-trigger Renewing certificate as renewal was scheduled at 2024-04-15 14:05:48 +0000 UTC
Normal Requested 48s cert-manager-certificates-request-manager Created new CertificateRequest resource "aerospike-client-tls-4"