feat: use the staging lets encrypt certificate by default to prevent being rate limited

Signed-off-by: Devin Buhl <devin@buhl.casa>

README.md

@@ -382,7 +382,7 @@ task cluster:resources
 
 ☢️ If you run into problems, you can run `task ansible:nuke` to destroy the k3s cluster and start over.
 
-🧠 Now it's time to pause and go get some coffee ☕ because next is describing how DNS is handled.
+🧠 Now it's time to pause and go get some coffee ☕ because next is describing additional things like how DNS is handled.
 
 ## 📣 Post installation
 
@@ -390,9 +390,18 @@ task cluster:resources
 
 [direnv](https://direnv.net/) will make it so anytime you `cd` to your repo's directory it export the required environment variables (e.g. `KUBECONFIG`). To set this up make sure you [hook it into your shell](https://direnv.net/docs/hook.html) and after that is done, run `direnv allow` while in your repos directory.
 
+### 📜 Certificates
+
+By default this template will deploy a wildcard certificate with the Let's Encrypt staging servers. This is to prevent you from getting rate-limited on configuration that might not be valid on bootstrap using the production server. Once you have confirmed the certificate is created and valid, make sure to switch to the Let's Encrypt production servers as outlined in the certificate manifest.
+
+- To view the certificate request run `kubectl -n networking get certificaterequests`
+- To verify the certificate is created run `kubectl -n networking get certificates`
+
+📍 Do not enable the production certificate until you are sure you will keep the cluster up for more than a few hours.
+
 ### 🌐 DNS
 
-📍 The `external-dns` application created in the `networking` namespace will handle creating public DNS records. By default, `echo-server` and the `flux-webhook` are the only public sub-domains exposed. In order to make additional applications public you must set an ingress annotation (`external-dns.alpha.kubernetes.io/target`) like done in the `HelmRelease` for `echo-server`.
+The `external-dns` application created in the `networking` namespace will handle creating public DNS records. By default, `echo-server` and the `flux-webhook` are the only public sub-domains exposed. In order to make additional applications public you must set an ingress annotation (`external-dns.alpha.kubernetes.io/target`) like done in the `HelmRelease` for `echo-server`.
 
 For split DNS to work it is required to have `${SECRET_DOMAIN}` point to the `${METALLB_K8S_GATEWAY_ADDR}` load balancer IP address on your home DNS server. This will ensure DNS requests for `${SECRET_DOMAIN}` will only get routed to your `k8s_gateway` service thus providing **internal** DNS resolution to your cluster applications/ingresses from any device that uses your home DNS server.
 

kubernetes/apps/kube-system/local-path-provisioner/app/helmrelease.yaml

@@ -33,7 +33,7 @@ spec:
     nodePathMap:
       - node: DEFAULT_PATH_FOR_NON_LISTED_NODES
         paths: ["/var/lib/rancher/k3s/storage"]
-    # Note: Do not enable Flux variable substitution on this HelmRelease
+    # NOTE: Do not enable Flux variable substitution on this HelmRelease
     configmap:
       setup: |-
         #!/bin/sh

kubernetes/apps/networking/ingress-nginx/app/helmrelease.yaml

@@ -57,7 +57,7 @@ spec:
           namespaceSelector:
             any: true
       extraArgs:
-        default-ssl-certificate: "networking/${SECRET_DOMAIN/./-}-production-tls"
+        default-ssl-certificate: "networking/${SECRET_DOMAIN/./-}-staging-tls"
       resources:
         requests:
           cpu: 10m

kubernetes/apps/networking/ingress-nginx/certificates/kustomization.yaml

@@ -2,4 +2,9 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
-  - ./certificates.yaml
+  - ./staging.yaml
+  # NOTE: Once you are able to verify the the staging certificate is valid uncomment out the
+  # certificate resource below and update the ingress-nginx helm chart with new production
+  # certificate name. This is to prevent you from getting rate-limited on configuration that might
+  # not be valid on bootstrap. No need to comment out the staging certificate they can coexist.
+  # - ./production.yaml

kubernetes/apps/networking/ingress-nginx/certificates/production.yaml

@@ -13,18 +13,3 @@ spec:
   dnsNames:
     - "${SECRET_DOMAIN}"
     - "*.${SECRET_DOMAIN}"
----
-apiVersion: cert-manager.io/v1
-kind: Certificate
-metadata:
-  name: "${SECRET_DOMAIN/./-}-staging"
-  namespace: networking
-spec:
-  secretName: "${SECRET_DOMAIN/./-}-staging-tls"
-  issuerRef:
-    name: letsencrypt-staging
-    kind: ClusterIssuer
-  commonName: "${SECRET_DOMAIN}"
-  dnsNames:
-    - "${SECRET_DOMAIN}"
-    - "*.${SECRET_DOMAIN}"

kubernetes/apps/networking/ingress-nginx/certificates/staging.yaml

@@ -0,0 +1,15 @@
+---
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: "${SECRET_DOMAIN/./-}-staging"
+  namespace: networking
+spec:
+  secretName: "${SECRET_DOMAIN/./-}-staging-tls"
+  issuerRef:
+    name: letsencrypt-staging
+    kind: ClusterIssuer
+  commonName: "${SECRET_DOMAIN}"
+  dnsNames:
+    - "${SECRET_DOMAIN}"
+    - "*.${SECRET_DOMAIN}"

kubernetes/apps/system-upgrade/kustomization.yaml

@@ -3,4 +3,7 @@ apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
   - ./namespace.yaml
+  # NOTE: Only enable system-upgrade-controller if you also update the version of k3s in the following file
+  # otherwise if you run ansible against an already provisioned cluster you might downgrade it
+  # https://github.com/onedr0p/flux-cluster-template/blob/main/ansible/inventory/group_vars/kubernetes/k3s.yml
   # - ./system-upgrade-controller/ks.yaml