official cheat sheet https:\/\/kubernetes.io\/docs\/reference\/kubectl\/cheatsheet\/<\/a>, here some favourites of mine, not only CKS specific.<\/p>\n Worth to read https:\/\/itnext.io\/cks-cka-ckad-changed-terminal-to-remote-desktop-157a26c1d5e<\/a> about Copy Past etc.<\/p>\n https:\/\/trainingportal.linuxfoundation.org\/learn\/dashboard\/<\/a><\/p>\n \u2192 https:\/\/trainingportal.linuxfoundation.org\/learn\/course\/certified-kubernetes-security-specialist-cks\/<\/a><\/p>\n https:\/\/kubernetes.io\/docs\/reference\/kubectl\/cheatsheet\/#bash<\/a><\/p>\n Regarding acloud.guru CKS course https:\/\/learn.acloud.guru\/course\/certified-kubernetes-security-specialist\/dashboard<\/a><\/p>\n Some facts:<\/p>\n https:\/\/kubernetes.io\/docs\/setup\/production-environment\/tools\/kubeadm\/install-kubeadm\/<\/a><\/p>\n https:\/\/kubernetes.io\/docs\/setup\/production-environment\/tools\/kubeadm\/create-cluster-kubeadm\/<\/a><\/p>\n On my test cluster, the worker node cannot connect to the master, test was to disable ipv6, according to https:\/\/github.com\/kubernetes-sigs\/kubespray\/issues\/6014<\/a>, but not working yet<\/p>\n Source: acloud.guru course <\/p>\n Additional dash here would mean a separate item in the array. This ingress would be allowed for pods in namespace "test" OR apps with label "client". If you want to have it combined, you need to write it without dash so it only allows if both conditions are met<\/p>\n kube-bench <\/p>\n Secret and Ingress needed, important bookmark: https:\/\/kubernetes.io\/docs\/concepts\/services-networking\/ingress\/#tls<\/a><\/p>\n self signed cert:<\/p>\n base64 could bi imported in the yaml in vim with:<\/p>\n Multi line YAML string:<\/p>\n source https:\/\/kubernetes.io\/docs\/tasks\/tools\/install-kubectl-linux\/#install-kubectl-binary-with-curl-on-linux<\/a>, some basteling to have it one-line done by me<\/p>\n Source: acloud.guru course <\/p>\n https:\/\/kubernetes.io\/docs\/tasks\/configure-pod-container\/configure-service-account\/<\/a><\/p>\n https:\/\/kubernetes.io\/docs\/reference\/access-authn-authz\/rbac\/<\/a><\/p>\n Source: acloud.guru course <\/p>\n It is possible to run the container in the Host Namespace<\/p>\n To be enabled with hostIPC, hostNetwork, hostPID<\/p>\n Priviledged Mode: containers[].securityContext.priviledged \u2192 true. With that Pod has access to Host Ressources.<\/p>\n Only relevant for Amazon AWS<\/p>\n two modes<\/p>\n -C complain mode<\/p>\n https:\/\/kubernetes.io\/docs\/tutorials\/security\/apparmor\/#example<\/a><\/p>\n example<\/p>\n Source: acloud.guru course <\/p>\n Can be set in Pod spec and Container spec. Options differ for securityContext if you run it in Pod or Container context.<\/p>\n Will be deprecated, documentation for v1.25.x not available any more, go to https:\/\/v1-24.docs.kubernetes.io\/<\/a>, see https:\/\/v1-24.docs.kubernetes.io\/docs\/concepts\/security\/pod-security-policy\/<\/a><\/p>\n for old documentation<\/p>\n not turned on by default, admission controller must be enabled with <\/p>\n Once turned on, you have to activate at least one policy, otherwise all pods are rejected.<\/p>\n RBAC needed to authorize via ServiceAccount (preferred over User)<\/p>\n rules to create an object.<\/p>\n Rego language used for actual rule programming in yaml.<\/p>\n A Constraint Template needed and we can create one or more constraints out of it. A constraint is an implementation of a constraint template.<\/p>\n The Constraint Template implements a new kind which we use to create new Constraint objects<\/p>\n Could be interesting for multi tenancy, but is costly regarding performance.<\/p>\n gVisor\/runsc: Kernel inside Kernel, something between Container and VM.<\/p>\n Kata Containers: lightweight VMs<\/p>\n https:\/\/kubernetes.io\/docs\/concepts\/containers\/runtime-class\/<\/a> \u2192 https:\/\/kubernetes.io\/docs\/concepts\/containers\/runtime-class\/#usage<\/a><\/p>\n usage <\/p>\n Mutual Transport Layer Security; both communication parties fully authenticate each other.<\/p>\n Certificate Signing<\/p>\n sidenode interesting tool cfssl to create certificates: https:\/\/github.com\/cloudflare\/cfssl<\/a><\/p>\n Create a CertificateSigningRequest object https:\/\/kubernetes.io\/docs\/tasks\/tls\/managing-tls-in-a-cluster\/#create-a-certificatesigningrequest-object-to-send-to-the-kubernetes-api<\/a><\/p>\n Approve the CertificateSigningRequest<\/p>\n Source: acloud.guru course <\/p>\n see exam tips<\/p>\n https:\/\/kubernetes.io\/docs\/concepts\/security\/overview\/<\/a><\/p>\n OPA Gatekeeper. ConstraintTemplate with rules and defining new Kind, and K8sAllowedRepos with actual whitelisting<\/p>\n https:\/\/docs.docker.com\/engine\/reference\/builder\/<\/a><\/p>\n To look for in Dockerfiles:<\/p>\n Don't use:<\/p>\n Use Trivy, documentation at https:\/\/github.com\/aquasecurity\/trivy\/blob\/main\/docs\/index.md<\/a><\/p>\n Install via apt according to: https:\/\/github.com\/aquasecurity\/trivy\/blob\/main\/docs\/getting-started\/installation.md#debianubuntu<\/a><\/p>\n for Linux Mint you have to use the UBUNTU_CODENAME so we can use this command to add the deb: <\/p>\n# `tail -f` pod\nkubectl logs my-pod --tail=10 -f\n\n# create yaml from image\nkubectl -n mynamespace run mynginx --dry-run=client --image=nginx:1.14 -o yaml > mynginx.yml\n\n# permanently save the namespace for all subsequent kubectl commands in that context.\nkubectl config set-context --current --namespace=ggckad-s2\n\n# get all objects in a namespace\nk get all -n $namespace\nkubectl get all -A # in all namespaces\nkubectl get sa # ServiceAccounts not included\n\n# show RBAC\nkubectl describe role $ROLE\n\n# use --show-labels to help with network plolicies\nkubectl -n mordor get pods mtdoom --show-labels\nkubectl get namespaces --show-label<\/code><\/pre>\n0. Course Meta Information<\/h1>\n
0.1. Exam Info<\/h2>\n
0.1.1. Exam Remote Desktop<\/h3>\n
0.1.2. Allowed Documentation for the Exam<\/h3>\n
\n
0.1.3. Portal<\/h3>\n
0.2. Prepare Environment in Course Labs and Exam<\/h2>\n
0.2.1. Kube env<\/h3>\n
source <(kubectl completion bash) # setup autocomplete in bash into the current shell, bash-completion package should be installed first.\necho "source <(kubectl completion bash)" >> ~\/.bashrc # add autocomplete permanently to your bash shell.\n\nalias k=kubectl\ncomplete -o default -F __start_kubectl k\n\ncat >> ~\/.bash_aliases <<-EOF\nalias k=kubectl\ncomplete -o default -F __start_kubectl k\nEOF\n\n# some tweak for `less`\nexport LESS="IFRX"\necho 'export LESS="IFRX"' >> ~\/.bashrc<\/code><\/pre>\n0.2.2. Vimrc<\/h3>\n
cat >> ~\/.vimrc <<-EOF\nset laststatus=2\nset hlsearch\nset number\nset tabstop=2 shiftwidth=2 expandtab\nset autoindent\nEOF<\/code><\/pre>\n0.3. Acloud.Guru Course Tips and Tricks<\/h2>\n
\n
\n
\n1. 01 - Install Kubeadm<\/h1>\n<\/li>\n<\/ul>\n
1.1. IPv6<\/h2>\n
# if IPV6 tagged to your network interface ,do this commands to disable.\n\nifconfig -a | grep inet6\n\n# add lines in \/etc\/sysctl.d\/99-kubernetes-cri.conf\nnet.ipv6.conf.all.disable_ipv6 = 1\nnet.ipv6.conf.default.disable_ipv6 = 1\n\nsudo sysctl --system\n<\/code><\/pre>\n2. 02 - Cluster Setup Review<\/h1>\n
2.1. Exam Tips Section 2<\/h2>\n
\n
\n
\n
\n
\n
\n2.2. Network policies<\/h2>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
2.2.1.1.1. deny all network policy<\/h3>\n
apiVersion: networking.k8s.io\/v1\nkind: NetworkPolicy\nmetadata:\n name: test-network-policy\n namespace: default\nspec:\n podSelector: {}\n policyTypes:\n - Ingress\n - Egres<\/code><\/pre>\n2.3. CSI Benchmark<\/h2>\n
$ wget -O kube-bench-control-plane.yaml https:\/\/raw.githubusercontent.com\/aquasecurity\/kube-bench\/main\/job-master.yaml\n$ wget -O kube-bench-node.yaml https:\/\/raw.githubusercontent.com\/aquasecurity\/kube-bench\/main\/job-node.yaml\n\n$ kubectl create -f kube-bench-control-plane.yaml\njob.batch\/kube-bench-master created\n$ kubectl create -f kube-bench-node.yaml\njob.batch\/kube-bench-node created\n$ kubectl get jobs.batch\nNAME COMPLETIONS DURATION AGE\nkube-bench-master 1\/1 12s 15s\nkube-bench-node 0\/1 7s 7s\n\n$ kubectl logs kube-bench-master-r8qkx > kube-bench-results-control-plane.log\n$ kubectl logs kube-bench-node-mvkb8 > kube-bench-results-node.log<\/code><\/pre>\n2.4. Implementing TLS with Ingress<\/h2>\n
openssl req -nodes -new -x509 -keyout accounts.key -out accounts.crt -subj "\/CN=accounts.svc"<\/code><\/pre>\n:read !base64 accounts.key<\/code><\/pre>\napiVersion: v1\nkind: Secret\nmetadata:\n name: accounts-secret\n namespace: accounts\ndata:\n tls.crt: |\n LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUREekNDQWZlZ0F3SUJBZ0lVUFN4Yk9qdGFF\n M3BpQnUveW5XQVNpSG9rcTBvd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0Z6RVZNQk1HQTFVRUF3d01Z\n V05qYjNWdWRITXVjM1pqTUI0WERUSXlNVEV5TVRFM01qRXhORm9YRFRJeQpNVEl5TVRFM01qRXhO\n ...\n VFBuUlV6SUpZUlE9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==\n tls.key: |\n LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2d0lCQURBTkJna3Foa2lHOXcwQkFRRUZB\n ...\ntype: kubernetes.io\/tls<\/code><\/pre>\n2.5. Verify your kubectl version sha256 checksum<\/h2>\n
# get version of kubectl\nkubectl version --short --client\n\n# get checksum of current version\ncurl -LO "https:\/\/dl.k8s.io\/`kubectl version --short --client | cut -f3 -d' '`\/bin\/linux\/amd64\/kubectl.sha256"\n\n# check the checksum of your kubecetl binary\necho "$(<kubectl.sha256) `whereis kubectl | cut -f2 -d' '`" | sha256sum --check\n\n# direct check for kubectl\necho $(curl -Ls "https:\/\/dl.k8s.io\/`kubectl version --short --client | cut -f3 -d' '`\/bin\/linux\/amd64\/kubectl.sha256") $(whereis kubectl | cut -f2 -d' ') | sha256sum --check\necho $(curl -sL "https:\/\/dl.k8s.io\/`kubectl version --short --client 2>\/dev\/null | grep "Client Version" | cut -f2 -d: | tr -d "[:blank:]"`\/bin\/linux\/amd64\/kubectl.sha256" | tee) \/usr\/bin\/kubectl | sha256sum --check\n\n# you can do the same for kubelet\necho $(curl -Ls "https:\/\/dl.k8s.io\/`kubelet --version | cut -f2 -d' '`\/bin\/linux\/amd64\/kubelet.sha256") $(whereis kubelet | cut -f2 -d' ') | sha256sum --check<\/code><\/pre>\n3. Cluster Hardening<\/h1>\n
3.1. Exam Tips Section 3<\/h2>\n
Service Accounts\n\n Tip 1: If a container is compromised, an attacker could use the ServiceAccount to access the Kubernetes API\n\n Tip 2: Use Kubernetes RBAC to control what ServiceAccounts can do and limit permissions\n\nService Account Permissions \/ Role Based Access Control (RBAC)\n\n Tip 1: Examine existing RoleBindings and ClusterRoleBindings to determine what permissions a service account has\n\n Tip 2: Design your RBAC setup in such a way that service accounts don't have unnecessary permissions\n\n Tip 3: You can bind multiple roles to an account. Use this to keep Roles separate rather that overloading them with a lot of permissions\n\n Tip 4: You can bind a ClusterRole with a RoleBinding to to proviede the necessary permissions only within the RoleBindings's namespace\n\nRestricting Access to the Kubernetes API\n\n Tip 1: Use RBAC to control user permissions within the API. Make sure accounts do not have permissions they do not need\n\n Tip 2: Limit network access to the API to prevent attackers from being able to communicate\n\nKeep k8s up to date\n\n Tip 1: Keep k8s up to date to take advantage of the latest security patches\n\n Tip 2: Since version 1.19 major releases receive approximately one year of patch support.<\/code><\/pre>\n3.2. Service Accounts<\/h2>\n
---\n\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n name: build-robot\n namespace: default\n\n---\napiVersion: rbac.authorization.k8s.io\/v1\nkind: Role\nmetadata:\n namespace: default\n name: pod-reader\nrules:\n- apiGroups: [""] # "" indicates the core API group\n resources: ["pods"]\n verbs: ["get", "watch", "list"]\n\n---\n\napiVersion: rbac.authorization.k8s.io\/v1\n# This role binding allows "jane" to read pods in the "default" namespace.\n# You need to already have a Role named "pod-reader" in that namespace.\nkind: RoleBinding\nmetadata:\n name: read-pods\n namespace: default\nsubjects:\n# You can specify more than one "subject"\n- kind: User\n name: jane # "name" is case sensitive\n apiGroup: rbac.authorization.k8s.io\nroleRef:\n # "roleRef" specifies the binding to a Role \/ ClusterRole\n kind: Role #this must be Role or ClusterRole\n name: pod-reader # this must match the name of the Role or ClusterRole you wish to bind to\n apiGroup: rbac.authorization.k8s.io<\/code><\/pre>\nkubectl describe roles\nName: auth-sa-role\nLabels: <none>\nAnnotations: <none>\nPolicyRule:\n Resources Non-Resource URLs Resource Names Verbs\n --------- ----------------- -------------- -----\n endpoints [] [] [get list]\n pods\/log [] [] [get list]\n pods [] [] [get list]\n services [] [] [get list]<\/code><\/pre>\n4. System Hardening<\/h1>\n
4.1. Exam Tips Section 4<\/h2>\n
\n
\n
\n
\n
\n
\n
container.apparmor.security.beta.kkubernetes.io\/nginx: localhost\/k8s-deny-write<\/code><\/pre>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n4.2. Host OS Security<\/h2>\n
4.3. IAM Roles<\/h2>\n
4.4. AppArmor<\/h2>\n
4.4.1.1.1. to check if AppArmor is enabled on your nodes<\/h3>\n
kubectl get nodes -o=jsonpath='{range .items[*]}{@.metadata.name}: {.status.conditions[?(@.reason=="KubeletReady")].message}{"\\n"}{end}'<\/code><\/pre>\n\n
sudo apparmor_parser \/path\/to\/file <\/code><\/pre>\napiVersion: v1\nkind: Pod\n metadata:\n name: password-db\n namespace: auth\n annotations:\n # Tell Kubernetes to apply the AppArmor profile.\n container.apparmor.security.beta.kubernetes.io\/password-db: localhost\/k8s-deny-write\nspec:\n containers:\n - name: password-db\n image: radial\/busyboxplus:curl\n command: ['sh', '-c', 'while true; do if echo "The password is hunter2" > password.txt; then echo "Password hunter2 logged."; else echo "Password log attempt blocked."; fi; sleep 5; done']<\/code><\/pre>\n5. Minimizing Microservice Vulnerabilities<\/h1>\n
5.1. Exam Tips Section 5<\/h2>\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
5.2. securityContexts<\/h2>\n
5.3. PodSecurityPolicies<\/h2>\n
5.4. Using Pod Security Policies<\/h2>\n
5.4.1. Enable the Use of PodSecurityPolicies in the Cluster<\/h3>\n
\n
sudo vi \/etc\/kubernetes\/manifests\/kube-apiserver.yaml<\/code><\/pre>\n<\/li>\n- --enable-admission-plugins=NodeRestriction,PodSecurityPolicy<\/code><\/pre>\n<\/li>\n:wq<\/code><\/pre>\n<\/li>\nkubectl get nodes <\/code><\/pre>\n<\/li>\n<\/ol>\n5.5. OPA Gatekeeper<\/h2>\n
5.6. Secrets<\/h2>\n
cloud_user@k8s-control:~$ kubectl get secrets -n users\nNAME TYPE DATA AGE\ndb-pass Opaque 1 94m\ncloud_user@k8s-control:~$ kubectl get secrets -n users db-pass -o yaml\napiVersion: v1\ndata:\n password: aHVudGVyMgo=\nkind: Secret\nmetadata:\n\n...\n\ncloud_user@k8s-control:~$ echo aHVudGVyMgo= | base64 --decode\nhunter2<\/code><\/pre>\n5.7. Runtime Sandboxes<\/h2>\n
5.7.1. implementations<\/h2>\n
5.7.1.1.1. Install gVisor<\/h3>\n
curl -fsSL https:\/\/gvisor.dev\/archive.key | sudo apt-key add -\nsudo add-apt-repository "deb [arch=amd64,arm64] https:\/\/storage.googleapis.com\/gvisor\/releases release main"\nsudo apt-get update && sudo apt-get install -y runsc<\/code><\/pre>\n# RuntimeClass is defined in the node.k8s.io API group\napiVersion: node.k8s.io\/v1\nkind: RuntimeClass\nmetadata:\n # The name the RuntimeClass will be referenced by.\n # RuntimeClass is a non-namespaced resource.\n name: myruntime\n# The name of the corresponding CRI configuration\n# this is the name we configured in \/etc\/containerd\/config.toml\nhandler: runsc<\/code><\/pre>\napiVersion: v1\nkind: Pod\nmetadata:\n name: mypod\nspec:\n runtimeClassName: runsc-sandbox\n\nkubectl exec non-sandbox-pod -- dmesg\n\n# first line shows kernel of current os, because it is running un-sandboxed.\n\n kubectl exec sandbox-pod -- dmesg<\/code><\/pre>\n5.8. mTLS and Certificates<\/h2>\n
5.8.1. Create a CertificateSigningRequest<\/h3>\n
sudo apt-get install -y golang-cfssl<\/code><\/pre>\n5.8.1.1.1. Generate a CSR file.<\/h3>\n
# according to https:\/\/kubernetes.io\/docs\/tasks\/tls\/managing-tls-in-a-cluster\/#create-a-certificate-signing-request\ncat <<EOF | cfssl genkey - | cfssljson -bare k8cert.chloesoe.ch\n{\n "hosts": [\n "saanen.chloesoe.ch",\n "lauenen.chloesoe.ch",\n "213.167.224.157",\n "185.142.215.9"\n ],\n "CN": "system:node:tls-pod.tls-test.pod.cluster.local",\n "key": {\n "algo": "ecdsa",\n "size": 256\n },\n "names": [\n {\n "O": "system:nodes"\n }\n ]\n}\nEOF\n\n# copy the base64 of the csr for later usag\ncat k8cert.chloesoe.ch.csr | base64<\/code><\/pre>\n# create CSR yaml, with base64 from CSR\ncat >> tls-svc-csr.yml <<EOF\napiVersion: certificates.k8s.io\/v1\nkind: CertificateSigningRequest\nmetadata:\n name: tls-svc-csr\nspec:\n request: $(cat k8cert.chloesoe.ch.csr | base64 | tr -d '\\n')\n signerName: example.com\/serving\n usages:\n - digital signature\n - key encipherment\n - server auth\nEOF\n\n# create object\nkubectl create -f tls-svc-csr.yml\n\n# check status\nkubectl get csr<\/code><\/pre>\nsudo falco -r nginx-rules.yml -M 45kubectl get csr -o wide\nNAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION\ntls-svc-csr 5s example.com\/serving kubernetes-admin <none> Pending\n\n# approve\nkubectl certificate approve tls-svc-csr\n\n# now approved, should alse be Issued in order to have a certificate\nkubectl get csr\nNAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION\ntls-svc-csr 51s example.com\/serving kubernetes-admin <none> Approved,Issued\n\n# Retrieve the signed certificate data.\nkubectl get csr tls-svc-csr -o yaml\nkubectl get csr tls-svc-csr -o jsonpath='{.status.certificate}' | base64 --decode<\/code><\/pre>\n6. Supply Chain Security Intro<\/h1>\n
6.1. Exam Tips Section 6<\/h2>\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n-Scanning<\/p>\n\n
\n
\n
6.2. Images<\/h2>\n
6.3. Whitelisting Registries<\/h2>\n
6.3.1.1.1. k8sallowedrepos.yml<\/h3>\n
apiVersion: templates.gatekeeper.sh\/v1beta1\nkind: ConstraintTemplate\nmetadata:\n name: k8sallowedrepossudo falco -r nginx-rules.yml -M 45 spec:\n crd:\n spec:\n names:\n kind: K8sAllowedRepos\n validation:\n # Schema for the `parameters` field\n openAPIV3Schema:\n properties:\n repos:\n type: array\n items:\n type: string\n targets:\n - target: admission.k8s.gatekeeper.sh\n rego: |\n package k8sallowedrepos\n violation[{"msg": msg}] {\n container := input.review.object.spec.containers[_]\n satisfied := [good | repo = input.parameters.repos[_] ; good = startswith(container.image,repo)]\n not any(satisfied)\n msg := sprintf("container <%v> has an invalid image repo <%v>, allowed repos are %v",[container.name, container.image, input.parameters.repos])\n }\n violation[{"msg": msg}] {\n container := input.review.object.spec.initContainers[_]\n satisfied := [good | repo = input.parameters.repos[_] ; good = startswith(container.image,repo)]\n not any(satisfied)\n msg := sprintf("container <%v> has an invalid image repo <%v>, allowed repos are %v",[container.name, container.image, input.parameters.repos])\n}<\/code><\/pre>\n6.3.1.1.2. whitelist-dockerhub.yml<\/h3>\n
apiVersion: constraints.gatekeeper.sh\/v1beta1\nkind: K8sAllowedRepos\nmetadata:\n name: whitelist-dockerhub\nspec:\n match:\n kinds:\n - apiGroups: [""]\n kinds: ["Pod"]\n parameters:\n repos:\n - "docker.io"<\/code><\/pre>\n6.4. Image Validation<\/h2>\n
6.5. Static Analysis<\/h2>\n
6.5.1. Dockerfiles<\/h3>\n
\n
6.5.2. YAML files<\/h3>\n
\n
6.6. Vulnerability Scanning<\/h2>\n
echo "deb [signed-by=\/usr\/share\/keyrings\/trivy.gpg] https:\/\/aquasecurity.github.io\/trivy-repo\/deb $(cat \/etc\/os-release | grep UBUNTU_CODENAME | awk -F'[\/=]' '{ print $2}') main" | sudo tee -a \/etc\/apt\/sources.list.d\/trivy.list<\/code><\/pre>\n6.6.1. get trivy running<\/h3>\n
\n
\n
sudo apt install golang-docker-credential-helpers<\/code><\/pre>\n<\/li>\n