14. April 2020

Kubernetes Single Node Cluster aufsetzen

Ziel des Artikels ist es, die notwendigen Schritte zum Aufsetzen eines Kubernetes Systems mit folgenden Anforderungen zu dokumentieren:

• Lauffähig auf einem einzelnen Netcup vServer (keine High-Availability)
• Integration in Gitlab AutoDevops möglich
• IPv4/IPv6 Dual-Stack fähig
• Mit Kubernetes Network Policies abgesichert
• Nutzung der lokalen Platte als Persistent Storage
• Mögliche Erweiterbarkeit auf mehrere Nodes

Basisimage

• Ubuntu 18.04 LTS Docker Image von Netcup, kleine Partition oder
• Ubuntu 20.04 LTS Minimal Image von Netcup, kleine Partition

Absicherungen basierend auf

• https://help.ubuntu.com/lts/serverguide/security.html
• https://help.replicated.com/community/t/managing-firewalls-with-ufw-on-kubernetes/230

System absichern

Mit dem vorkonfigurierten Root-User per SSH:

adduser jo
adduser jo sudo
# nur bei 20.04
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"
apt install docker-ce docker-ce-cli containerd.io
#
adduser docker

Mit dem neuen Benutzer jo per SSH:

ssh-keygen
echo "ssh-rsa AAAAB...3w== ssh-jowi-privat-aes" >> ~/.ssh/authorized_keys

sudo passwd -l root
sudo ufw allow OpenSSH
sudo ufw allow out http
sudo ufw allow out https
# allow dns queries
sudo ufw allow out 53/udp
# allow ntp systemd time syncing
sudo ufw allow out 123
sudo ufw default deny outgoing
sudo ufw default deny incoming
sudo ufw enable

SSH verlegen um Platz für Gitlab Port zu schaffen:

sudo ufw allow 2222
sudo sed -i -e 's/#Port 22/Port 2222/' /etc/ssh/sshd_config
sudo systemctl restart ssh

Paritionierung

sudo fdisk /dev/sda
# n enter enter enter w
sudo mke2fs /dev/sda4
echo "/dev/sda4 /srv ext4 defaults 0 2"  | sudo tee -a /etc/fstab
sudo mount /srv

Automatische Updates aktivieren

sudo sed -i 's/\/\/Unattended-Upgrade::Automatic-Reboot/Unattended-Upgrade::Automatic-Reboot/g' /etc/apt/apt.conf.d/50unattended-upgrades
cat <<EOF | sudo tee -a /etc/apt/apt.conf.d/20auto-upgrades
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "7";
APT::Periodic::Unattended-Upgrade "1";
EOF

Aufsetzen des Clusters

Basierend auf der offiziellen Dokumentation. Zusätzlich IPv6 Forwarding aktivieren (Zeile 4).

Vorbereitungen

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv6.conf.all.forwarding = 1
EOF
sudo sysctl --system

cat <<EOF | sudo tee /etc/default/kubelet
KUBELET_EXTRA_ARGS=--feature-gates="IPv6DualStack=true"
EOF

cat <<EOF | sudo tee /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

sudo mv /var/lib/docker /srv/docker
sudo ln -s /srv/docker /var/lib/docker
sudo systemctl enable docker
sudo systemctl restart docker
sudo ufw allow 6443
sudo ufw allow out to 172.18.0.0/24
sudo ufw allow out to 172.18.1.0/24
sudo ufw allow out to fc00::/64
sudo ufw allow out to fc01::/110
sudo ufw allow in from 172.18.0.0/24
sudo ufw allow in from 172.18.1.0/24
sudo ufw allow in from fc00::/64
sudo ufw allow in from fc01::/110

Paketinstallation

Zum Zeitpunkt des Artikels wird Version 1.18.2 installiert.

sudo apt-get update && sudo apt-get upgrade
sudo apt-get install -y apt-transport-https curl mc ipvsadm
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl

sudo snap install helm --classic
helm repo add stable https://kubernetes-charts.storage.googleapis.com
helm repo update

Bash Tab-Vervollständigung analog zu https://kubernetes.io/de/docs/tasks/tools/install-kubectl/

echo 'source <(kubectl completion bash)' >>~/.bashrc

Kubeadm Setup

cat <<EOF > kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
featureGates:
  IPv6DualStack: true
kind: ClusterConfiguration
kubernetesVersion: 1.18.1
networking:
  serviceSubnet: "172.18.1.0/24,fc01::/110"
  podSubnet: "172.18.0.0/24,fc00::/64"
  dnsDomain: "cluster.local"
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
EOF
sudo kubeadm init --config kubeadm-config.yaml

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) \$HOME/.kube/config

kubectl taint nodes --all node-role.kubernetes.io/master-

# edit /etc/kubernetes/manifests/kube-apiserver.yaml
# add ServerSideApply=false to --feature-gate parameter

Calico Networking Setup

Siehe auch https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3705 zur Erhöhung der MTU auf 1500, da es sonst zu Konnektivitätsproblemen kommen kann.

curl https://docs.projectcalico.org/manifests/calico.yaml -o calico.yaml
cat <<"EOF" | patch
--- calico.yaml.sav     2020-05-08 10:47:50.400000000 +0200
+++ calico.yaml 2020-05-08 10:49:34.700000000 +0200
@@ -14,7 +14,7 @@ data:
   # Configure the MTU to use for workload interfaces and the
   # tunnels.  For IPIP, set to your network MTU - 20; for VXLAN
   # set to your network MTU - 50.
-  veth_mtu: "1440"
+  veth_mtu: "1500"

   # The CNI network configuration to install on each node.  The special
   # values in this config will be automatically populated.
@@ -30,8 +30,13 @@ data:
           "nodename": "__KUBERNETES_NODE_NAME__",
           "mtu": __CNI_MTU__,
           "ipam": {
+              "assign_ipv4": "true",
+              "assign_ipv6": "true",
               "type": "calico-ipam"
           },
+          "container_settings": {
+            "allow_ip_forwarding": true
+          },
           "policy": {
               "type": "k8s"
           },
@@ -671,6 +676,8 @@ spec:
             # no effect. This should fall within `--cluster-cidr`.
             # - name: CALICO_IPV4POOL_CIDR
             #   value: "192.168.0.0/16"
+            - name: CALICO_IPV4POOL_CIDR
+              value: "172.18.0.0/24"
             # Disable file logging so `kubectl logs` works.
             - name: CALICO_DISABLE_FILE_LOGGING
               value: "true"
@@ -685,6 +692,14 @@ spec:
               value: "info"
             - name: FELIX_HEALTHENABLED
               value: "true"
+            - name: IP6
+              value: "autodetect"
+            - name: CALICO_IPV6POOL_CIDR
+              value: "fc00::/64"
+            - name: FELIX_IPV6SUPPORT
+              value: "true"
+            - name: CALICO_IPV6POOL_NAT_OUTGOING
+              value: "true"
           securityContext:
             privileged: true
           resources:
EOF
kubectl apply -f calico.yaml

Loadbalancer Setup

Auf Basis von https://metallb.universe.tf/installation/

kubectl create namespace kube-lb
cat <<EOF > metallb-values.yaml
configInline:
  peers:
  address-pools:
  - name: default4
    protocol: layer2
    addresses:
    - 152.89.xxx.xxx/32
    - 2a03:4000:39:xxxx:xxxx:xxxx:xxxx:xxxx/128
controller:
  image:
    tag: v0.9.3
speaker:
  image:
    tag: v0.9.3
EOF
helm upgrade -i metallb -n kube-lb stable/metallb -f metallb-values.yaml

Hostpath Provider

mkdir -p /srv/k8s-storage
cat <<EOF > hostpath-provisioner.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hostpath-provisioner
  labels:
    k8s-app: hostpath-provisioner
  namespace: kube-system
spec:
  replicas: 1
  revisionHistoryLimit: 0
  selector:
    matchLabels:
      k8s-app: hostpath-provisioner
  template:
    metadata:
      labels:
        k8s-app: hostpath-provisioner
    spec:
      serviceAccountName: k8s-hostpath
      containers:
        - name: hostpath-provisioner
          image: cdkbot/hostpath-provisioner-amd64:1.0.0
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: PV_DIR
              value: /srv/k8s-storage
            #- name: PV_RECLAIM_POLICY
            #  value: Retain
          volumeMounts:
            - name: pv-volume
              mountPath: /srv/k8s-storage
      volumes:
        - name: pv-volume
          hostPath:
            path: /srv/k8s-storage
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: k8s-hostpath
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: microk8s.io/hostpath
#reclaimPolicy: Retain //is ignored (see above)
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: k8s-hostpath
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: k8s-hostpath
rules:
- apiGroups: [""]
  resources:
  - persistentvolumeclaims
  verbs:
  - list
  - get
  - watch
  - update
- apiGroups: [""]
  resources:
  - persistentvolumes
  verbs:
  - list
  - get
  - update
  - watch
  - create
  - delete
- apiGroups: [""]
  resources:
    - events
  verbs:
    - create
    - list
    - patch
- apiGroups: ["storage.k8s.io"]
  resources:
    - storageclasses
  verbs:
    - list
    - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: k8s-hostpath
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: k8s-hostpath
subjects:
  - kind: ServiceAccount
    name: k8s-hostpath
    namespace: kube-system
EOF
kubectl apply -f hostpath-provisioner.yaml

NGINX Ingress

Auf Basis von https://github.com/helm/charts/tree/master/stable/nginx-ingress

sudo ufw allow in 22
sudo ufw allow in http
sudo ufw allow in https

curl https://raw.githubusercontent.com/helm/charts/master/stable/nginx-ingress/values.yaml -o nginx-ingress-values.yaml
patch <<EOF
--- nginx-ingress-values.yaml.sav       2020-04-20 09:08:30.152000000 +0200
+++ nginx-ingress-values.yaml   2020-04-20 09:11:40.840000000 +0200
@@ -264,7 +264,7 @@ controller:
     ## Set external traffic policy to: "Local" to preserve source IP on
     ## providers supporting it
     ## Ref: https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-typeloadbalancer
-    externalTrafficPolicy: ""
+    externalTrafficPolicy: "Local"

     # Must be either "None" or "ClientIP" if set. Kubernetes will default to "None".
     # Ref: https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies
@@ -567,6 +567,7 @@ imagePullSecrets: []
 ##
 tcp: {}
 #  8080: "default/example-tcp-svc:9000"
+22: "gitlab/gitlab-gitlab-shell:22"

 # UDP service key:value pairs
EOF
kubectl create namespace "kube-ingress"
helm upgrade -i -n kube-ingress nginx-ingress stable/nginx-ingress -f nginx-ingress-values.yaml

cat <<EOF > nginx-ingress-service-v6.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx-ingress-controller-v6
  namespace: kube-ingress
spec:
  externalTrafficPolicy: Local
  ipFamily: IPv6
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  - name: https
    port: 443
    protocol: TCP
    targetPort: https
  - name: git
    port: 22
    protocol: TCP
    targetPort: 22
  selector:
    app: nginx-ingress
    component: controller
    release: nginx-ingress
  sessionAffinity: None
  type: LoadBalancer
EOF
kubectl apply -f nginx-ingress-service-v6.yaml

Certmanager

Automatisches Ausstellen von Zertifikaten mittels LetsEncrypt

Auf Basis von https://cert-manager.io/docs/tutorials/acme/ingress/

Achtung: ein abweichender Namespace (z.B. kube-cert-manager) führt leider zu Problemen, hier müssten die CRDs modifiziert werden.

curl https://raw.githubusercontent.com/jetstack/cert-manager/master/deploy/charts/cert-manager/values.yaml -o cert-manager-values.yaml
patch <<EOF
--- cert-manager-values.yaml.sav        2020-04-21 15:20:14.080000000 +0200
+++ cert-manager-values.yaml    2020-04-21 15:22:26.104000000 +0200
@@ -115,10 +115,10 @@ podLabels: {}

 nodeSelector: {}

-ingressShim: {}
-  # defaultIssuerName: ""
-  # defaultIssuerKind: ""
-  # defaultIssuerGroup: ""
+ingressShim:
+  defaultIssuerName: "letsencrypt-prod"
+  defaultIssuerKind: "ClusterIssuer"
+  defaultIssuerGroup: "cert-manager.io"

 prometheus:
   enabled: true
EOF

helm repo add jetstack https://charts.jetstack.io
# add crds
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v0.14.2/cert-manager.crds.yaml
# install chart
kubectl create namespace cert-manager
helm upgrade -i -n cert-manager cert-manager jetstack/cert-manager -f cert-manager-values.yaml

# add issuer
cat <<EOF > cert-manager-issuer.yaml
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: xxx@xxx
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-prod
    # Enable the HTTP-01 challenge provider
    solvers:
    - http01:
        ingress:
          class: nginx
EOF

Verwendung: Ein Ingress muss folgende Annotation aufweisen um vom Reverse-Proxy geroutet zu werden und ein TLS-Zertifikat mit dem defaultIssuer zu erhalten:

metadata:
  name: kuard
  annotations:
    kubernetes.io/ingress.class: "nginx"
    kubernetes.io/tls-acme: "true"

Kubernetes Dashboard

Auf Basis von:

• https://kubernetes.io/docs/tasks/debug-application-cluster/resource-usage-monitoring/#resource-metrics-pipeline
• https://github.com/kubernetes-sigs/metrics-server

curl https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-rc7/aio/deploy/recommended.yaml -o kube-dashboard.yaml
sed -i -e 's/namespace: kubernetes-dashboard/namespace: kube-dashboard/g' kube-dashboard.yaml
cat <<EOF | patch
--- kube-dashboard.yaml.sav     2020-04-17 20:06:59.396000000 +0200
+++ kube-dashboard.yaml 2020-04-23 07:44:28.536000000 +0200
@@ -15,7 +15,7 @@
 apiVersion: v1
 kind: Namespace
 metadata:
-  name: kubernetes-dashboard
+  name: kube-dashboard

 ---

@@ -194,7 +194,8 @@ spec:
               protocol: TCP
           args:
             - --auto-generate-certificates
-            - --namespace=kubernetes-dashboard
+            - --namespace=kube-dashboard
+            - --token-ttl=43200
             # Uncomment the following line to manually specify Kubernetes API server Host
             # If not specified, Dashboard will attempt to auto discover the API server and connect
             # to it. Uncomment only if the default does not work.
EOF

cat <<EOF >> kube-dashboard.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kube-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kube-dashboard
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
    ingress.kubernetes.io/ssl-redirect: "true"
    ingress.kubernetes.io/use-port-in-redirects: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
  name: dashboard
  namespace: kube-dashboard
spec:
  tls:
  - hosts:
    - dashboard.mydomain.com
    secretName: dashboard.mydomain.com-tls
  rules:
  - host: dashboard.mydomain.com
    http:
      paths:
      - backend:
          serviceName: kubernetes-dashboard
          servicePort: 8443
        path: /
EOF
kubectl apply -f kube-dashboard.yaml

# Get token for login
kubectl -n kube-dashboard describe secret \
  $(kubectl -n kube-dashboard get secret | grep admin-user | awk '{print $1}')

• Zugriff ohne Ingress
  • Run kubectl proxy
  • via http://localhost:8001/api/v1/namespaces/kube-dashboard/services/https:kubernetes-dashboard:/proxy/#/login
• Zugriff über Ingress
  • via https://dashboard.mydomain.com

Gitlab

Installation

Verwendendung des Standard-Helm-Charts unter Berücksichtigung folgender Modifikationen https://docs.gitlab.com/charts/advanced/external-nginx/index.html and https://docs.gitlab.com/charts/installation/tls.html

helm repo add gitlab https://charts.gitlab.io/
kubectl create namespace gitlab
kubectl create secret generic gitlab-smtp-creds --from-literal=password='xxx' -n gitlab

Gitlab Integration

Den Cluster in Gitlab hinzufügen (nicht durch Gitlab managen lassen):

cat <<EOF > gitlab-admin-service-account.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: gitlab-admin
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: gitlab-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: gitlab-admin
  namespace: kube-system
EOF
kubectl apply -f gitlab-admin-service-account.yaml

Folgende Informationen in der Einrichtung angeben:

# API URL
kubectl cluster-info | grep 'Kubernetes master' | awk '/http/ {print $NF}'
# CA Certificate
kubectl get secret $(kubectl get secrets | grep token | cut -d " " -f 1) \
  -o jsonpath="{['data']['ca\.crt']}" | base64 --decode
# Token
kubectl -n kube-system describe secret $(kubectl -n kube-system get secret |\
  grep gitlab-admin | awk '{print $1}')

Vorhandene Docker Projekte für Kubernetes anpassen

• Workaround für neue Kubernetes Versionen Kubernetes 1.16+

cd repo
cat <<EOF > .gitlab/auto-deploy-values.yaml
deploymentApiVersion: apps/v1
EOF

• Auto-Devops konfigurieren mit eingenem Namespace und Domain

# derived from https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml

include:
  - template: Auto-DevOps.gitlab-ci.yml

variables:
  CODE_QUALITY_DISABLED: "1"
  POSTGRES_ENABLED: "false"
  TEST_DISABLED: "1"
  PERFORMANCE_DISABLED: "1"
  ADDITIONAL_HOSTS: "app2.joachim-wilke.de"

production:
  environment:
    name: production
    url: https://app.joachim-wilke.de
    kubernetes:
      namespace: jowi-app

• Bei Bedarf eigenes Chart auf Basis des offiziellen Charts ableiten (Nutzung von Git Subtree)

git remote add -f autodeploy https://gitlab.com/gitlab-org/charts/auto-deploy-app.git
git subtree add --prefix chart autodeploy master --squash

# On each update:
git fetch autodeploy master
git subtree pull --prefix chart autodeploy master --squash

• Network Policies mit Default Chart aktivieren:
  https://docs.gitlab.com/ee/topics/autodevops/stages.html#network-policy

# allow no egress traffic, ingress only from ingress proxy
networkPolicy:
  enabled: true
  spec:
    podSelector: {}
    policyTypes:
      - Ingress
      - Egress
    ingress:
      - from:
          - namespaceSelector: {}
          - podSelector:
              matchLabels:
                app: "nginx-ingress"

Cheat-Sheet

Firewall

• Get Status: sudo ufw status verbose
• Applikationsprofile aufrufen: sudo ufw app list

Kubernetes

• Reset Kubeadm: kubeadm reset && iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X && ipvsadm -C
• Get Node Status: kubectl get nodes -o wide
• Get Status: watch kubectl get all -A

Offene Punkte

• Helm Chart auch für das Dashboard verwenden
• Bringt der iptables proxy mode, im Vergleich zu ipvs Vorteile? Aktuell wird iptables von calico i.V.m. DualStack noch nicht unterstützt.
• Anzahl Replicas des Coredns Pods auf 1 beschränken (per default sind es 2)
• Ist es sinnvoll (und bei Netcup technisch machbar), global routbare IPv6 Adressen in den Pods zu verwenden? Siehe auch https://forum.netcup.de/administration-eines-server-vserver/vserver-server-kvm-server/9577-ipv6-addressvergabe-an-docker-container/
• Was hat es mit den Meldungen im Syslog auf sich: systemd[1551]: Failed to canonicalize path permission denied message?
• Nginx Ingress Default Backend anpassen (eigene Fehlerseite)
• Certmanager im Namespace kube-cert-manager deployen
• Alle Sicherheitschecks von https://en.internet.nl/ bestehen
• Alle Checks des Kube-Bench bestehen kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/master/job-master.yaml

Letzte Änderung: 10. Juli 2020

Webserver

TLS, SSL, HTTPS, ACME, letsencrypt, Kubernetes, Ubuntu, Kubeadm, IPv6, Dualstack, GitLab