General Cheatsheet

official cheat sheet https://kubernetes.io/docs/reference/kubectl/cheatsheet/, here some favourites of mine, not only CKS specific.

# `tail -f` pod
kubectl logs my-pod --tail=10 -f

# create yaml from image
kubectl -n mynamespace run mynginx --dry-run=client --image=nginx:1.14 -o yaml  > mynginx.yml

# permanently save the namespace for all subsequent kubectl commands in that context.
kubectl config set-context --current --namespace=ggckad-s2

# get all objects in a namespace
k get all -n $namespace
kubectl get all -A    # in all namespaces
kubectl get sa   # ServiceAccounts not included

# show RBAC
kubectl describe role $ROLE

# use --show-labels to help with network plolicies
kubectl -n mordor get pods mtdoom --show-labels
kubectl get namespaces --show-label

0. Course Meta Information

0.1. Exam Info

0.1.1. Exam Remote Desktop

Worth to read https://itnext.io/cks-cka-ckad-changed-terminal-to-remote-desktop-157a26c1d5e about Copy Past etc.

0.1.2. Allowed Documentation for the Exam

• https://kubernetes.io/docs/
• https://github.com/kubernetes
• https://kubernetes.io/blog
• https://github.com/aquasecurity/trivy
• https://docs.sysdig.com
• https://falco.org/docs
• https://gitlab.com/apparmor/apparmor/-/wikis/Documentation

0.1.3. Portal

https://trainingportal.linuxfoundation.org/learn/dashboard/

→ https://trainingportal.linuxfoundation.org/learn/course/certified-kubernetes-security-specialist-cks/

0.2. Prepare Environment in Course Labs and Exam

0.2.1. Kube env

https://kubernetes.io/docs/reference/kubectl/cheatsheet/#bash

source <(kubectl completion bash) # setup autocomplete in bash into the current shell, bash-completion package should be installed first.
echo "source <(kubectl completion bash)" >> ~/.bashrc # add autocomplete permanently to your bash shell.

alias k=kubectl
complete -o default -F __start_kubectl k

cat >> ~/.bash_aliases <<-EOF
alias k=kubectl
complete -o default -F __start_kubectl k
EOF

# some tweak for `less`
export LESS="IFRX"
echo 'export LESS="IFRX"' >> ~/.bashrc

0.2.2. Vimrc

cat >> ~/.vimrc <<-EOF
set laststatus=2
set hlsearch
set number
set tabstop=2 shiftwidth=2 expandtab
set autoindent
EOF

0.3. Acloud.Guru Course Tips and Tricks

Regarding acloud.guru CKS course https://learn.acloud.guru/course/certified-kubernetes-security-specialist/dashboard

Some facts:

• it could be worth, to have an own cluster running to directly test interesting stuff there.
• To listen to all chapters and to the chapter labs, I invested 26h. This is without the 12 thirty minute labs at the end.
  • So you could take 3 education leave days to go throu the whole course
  • Sometimes you can spend a bit more time, if you're interested in a topic
• For the labs and the killer.sh preparation you can take the other 2d of education leave
  

  1. 01 - Install Kubeadm

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/

1.1. IPv6

On my test cluster, the worker node cannot connect to the master, test was to disable ipv6, according to https://github.com/kubernetes-sigs/kubespray/issues/6014, but not working yet

# if IPV6 tagged to your network interface ,do this commands to disable.

ifconfig -a | grep inet6

# add lines in /etc/sysctl.d/99-kubernetes-cri.conf
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

sudo sysctl --system

2. 02 - Cluster Setup Review

2.1. Exam Tips Section 2

Source: acloud.guru course

• NetworkPolicy
  • Tip 1: Default deny + Targeted policies to block all but necessary traffic in a namespace
  • Tip 2: Use an empty podSelector: {} to apply the to all Pods in the namespace
  • Tip 3: Pay attention to the difference between one rule with multiple selectors and multiple rules (e.g. a dash "-" can make a difference)
• CIS Benchmark
  • Tip 1: CIS Kubernetes Benchmark - A set of standards and best practices for a secure Kubernetes cluster
  • Tip 2: kube-bench - Runs automated tests to checvk how well your cluster conforms to the CIS Benchmark
  • Tip 3: kubeadm clusters use a kubelet config file located at /var/lib/kubelet/config.yaml on each node
  • Tip 4: In a kubeadm cluster, manifest files for control plane components (API server, etcd, etc.) can be found in /etc/kubernetes/manifests on the control plane server
• Ingress:
  • Tip 1: You can implement TLS termination using an Ingress
  • Tip 2: Store TLS certificates using a Secret. Pass the Secret to the Ingress using spec.tls[].secretName
  • Tip 3: Bookmark the Ingres -TLS documentation (https://kubernetes.io/docs/concepts/services-networking/ingress/#tls)
• More Tips:
  • Tip 1: Kubernetes servers use certain ports to provide Kubernetes functionality. So be aware that you may need to protect those ports in the real world.
  • Tip 2: if you use any graphical interfaces for Kubernetes, be aware that they also present a target for attack, and make sure to keep them secure.
  • Tip 3: When installing Kubernetes binaries manually, you can use those checksums to validate your Kubernetes binaries to make sure that they haven't been altered.
    

    2.2. Network policies

Additional dash here would mean a separate item in the array. This ingress would be allowed for pods in namespace "test" OR apps with label "client". If you want to have it combined, you need to write it without dash so it only allows if both conditions are met

2.2.1.1.1. deny all network policy

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: test-network-policy
  namespace: default
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egres

2.3. CSI Benchmark

kube-bench

$ wget -O kube-bench-control-plane.yaml https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job-master.yaml
$ wget -O kube-bench-node.yaml https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job-node.yaml

$ kubectl create -f kube-bench-control-plane.yaml
job.batch/kube-bench-master created
$ kubectl create -f kube-bench-node.yaml
job.batch/kube-bench-node created
$ kubectl get jobs.batch
NAME                COMPLETIONS   DURATION   AGE
kube-bench-master   1/1           12s        15s
kube-bench-node     0/1           7s         7s

$ kubectl logs kube-bench-master-r8qkx > kube-bench-results-control-plane.log
$ kubectl logs kube-bench-node-mvkb8 > kube-bench-results-node.log

2.4. Implementing TLS with Ingress

Secret and Ingress needed, important bookmark: https://kubernetes.io/docs/concepts/services-networking/ingress/#tls

self signed cert:

openssl req -nodes -new -x509 -keyout  accounts.key -out accounts.crt -subj "/CN=accounts.svc"

base64 could bi imported in the yaml in vim with:

:read !base64 accounts.key

Multi line YAML string:

apiVersion: v1
kind: Secret
metadata:
  name: accounts-secret
  namespace: accounts
data:
  tls.crt: |
    LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUREekNDQWZlZ0F3SUJBZ0lVUFN4Yk9qdGFF
    M3BpQnUveW5XQVNpSG9rcTBvd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0Z6RVZNQk1HQTFVRUF3d01Z
    V05qYjNWdWRITXVjM1pqTUI0WERUSXlNVEV5TVRFM01qRXhORm9YRFRJeQpNVEl5TVRFM01qRXhO
    ...
    VFBuUlV6SUpZUlE9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
  tls.key: |
    LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2d0lCQURBTkJna3Foa2lHOXcwQkFRRUZB
    ...
type: kubernetes.io/tls

2.5. Verify your kubectl version sha256 checksum

source https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/#install-kubectl-binary-with-curl-on-linux, some basteling to have it one-line done by me

# get version of kubectl
kubectl version --short --client

# get checksum of current version
curl -LO "https://dl.k8s.io/`kubectl version --short --client | cut -f3 -d' '`/bin/linux/amd64/kubectl.sha256"

# check the checksum of your kubecetl binary
echo "$(<kubectl.sha256)  `whereis kubectl | cut -f2 -d' '`" | sha256sum --check

# direct check for kubectl
echo $(curl -Ls "https://dl.k8s.io/`kubectl version --short --client | cut -f3 -d' '`/bin/linux/amd64/kubectl.sha256") $(whereis kubectl | cut -f2 -d' ') | sha256sum --check
echo $(curl -sL "https://dl.k8s.io/`kubectl version --short --client 2>/dev/null | grep "Client Version" | cut -f2 -d: | tr -d "[:blank:]"`/bin/linux/amd64/kubectl.sha256" | tee) /usr/bin/kubectl  | sha256sum --check

# you can do the same for kubelet
echo $(curl -Ls "https://dl.k8s.io/`kubelet --version | cut -f2 -d' '`/bin/linux/amd64/kubelet.sha256") $(whereis kubelet | cut -f2 -d' ') | sha256sum --check

3. Cluster Hardening

3.1. Exam Tips Section 3

Source: acloud.guru course

Service Accounts

    Tip 1: If a container is compromised, an attacker could use the ServiceAccount to access the Kubernetes API

    Tip 2: Use Kubernetes RBAC to control what ServiceAccounts can do and limit permissions

Service Account Permissions / Role Based Access Control (RBAC)

    Tip 1: Examine existing RoleBindings and ClusterRoleBindings to determine what permissions a service account has

    Tip 2: Design your RBAC setup in such a way that service accounts don't have unnecessary permissions

    Tip 3: You can bind multiple roles to an account. Use this to keep Roles separate rather that overloading them with a lot of permissions

    Tip 4: You can bind a ClusterRole with a RoleBinding to to proviede the necessary permissions only within the RoleBindings's namespace

Restricting Access to the Kubernetes API

    Tip 1: Use RBAC to control user permissions within the API. Make sure accounts do not have permissions they do not need

    Tip 2: Limit network access to the API to prevent attackers from being able to communicate

Keep k8s up to date

    Tip 1: Keep k8s up to date to take advantage of the latest security patches

    Tip 2: Since version 1.19 major releases receive approximately one year of patch support.

3.2. Service Accounts

https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/

https://kubernetes.io/docs/reference/access-authn-authz/rbac/

---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: build-robot
  namespace: default

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: pod-reader
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["pods"]
  verbs: ["get", "watch", "list"]

---

apiVersion: rbac.authorization.k8s.io/v1
# This role binding allows "jane" to read pods in the "default" namespace.
# You need to already have a Role named "pod-reader" in that namespace.
kind: RoleBinding
metadata:
  name: read-pods
  namespace: default
subjects:
# You can specify more than one "subject"
- kind: User
  name: jane # "name" is case sensitive
  apiGroup: rbac.authorization.k8s.io
roleRef:
  # "roleRef" specifies the binding to a Role / ClusterRole
  kind: Role #this must be Role or ClusterRole
  name: pod-reader # this must match the name of the Role or ClusterRole you wish to bind to
  apiGroup: rbac.authorization.k8s.io

kubectl describe roles
Name:         auth-sa-role
Labels:       <none>
Annotations:  <none>
PolicyRule:
  Resources  Non-Resource URLs  Resource Names  Verbs
  ---------  -----------------  --------------  -----
  endpoints  []                 []              [get list]
  pods/log   []                 []              [get list]
  pods       []                 []              [get list]
  services   []                 []              [get list]

4. System Hardening

4.1. Exam Tips Section 4

Source: acloud.guru course

• Host OS Secuirty
  • Tip 1: Protect your hosts from attacks that might come from within containers.
  • Tip 2: Beware of Pod settingse like hostIPC, hostNetwork, hostPID. Use them only when absolutely necessary.
• IAM Roles
  • Tip 1: If you are running K8s in AWS, your container applications may be able to access IAM credentials
  • Tip 2: Avoid providing unnecessary permissions to IAM roles (principle of least privilege)
  • Tip 3: If your app does not need IAM access, consider blockeing access to IAM credentials via firewall, NetworkPolicy, etc.
• Network Level Secuirty
  • Tip 1: By default, anyone who can access the cluster network can communicate with all Pods and Services in the cluster
  • aka, log-in via SSH you have full access
  • Tip 2: When possible, limit access to the cluster network from outside
• AppArmor
  • Tip 1: AppArmor is a Linux kerne security module that allows granular control over what individual programs can and cannot do
  • Tip 2: Load a profile in enforce mode (aka "enforcing the profile") to actively prevent programs from doing anything the profile does not allow.
  • Tip 3: Load a profile in complain mode to simly report on what the program is doing
• AppArmor in Containers
  • Tip 1: Use apparmor_parser command to load an AppArmonr profile from a file. It will load the profile in enforcing mode by default.
  • Tip 2: Use Pod annotations to apply an AppArmor profile to a container: For example:

    container.apparmor.security.beta.kkubernetes.io/nginx: localhost/k8s-deny-write

4.2. Host OS Security

It is possible to run the container in the Host Namespace

To be enabled with hostIPC, hostNetwork, hostPID

Priviledged Mode: containers[].securityContext.priviledged → true. With that Pod has access to Host Ressources.

4.3. IAM Roles

Only relevant for Amazon AWS

4.4. AppArmor

4.4.1.1.1. to check if AppArmor is enabled on your nodes

kubectl get nodes -o=jsonpath='{range .items[*]}{@.metadata.name}: {.status.conditions[?(@.reason=="KubeletReady")].message}{"\n"}{end}'

two modes

• enforce mode
• complain mode

sudo apparmor_parser /path/to/file 

-C complain mode

https://kubernetes.io/docs/tutorials/security/apparmor/#example

example

apiVersion: v1
kind: Pod
 metadata:
  name: password-db
  namespace: auth
  annotations:
    # Tell Kubernetes to apply the AppArmor profile.
    container.apparmor.security.beta.kubernetes.io/password-db: localhost/k8s-deny-write
spec:
  containers:
  - name: password-db
    image: radial/busyboxplus:curl
    command: ['sh', '-c', 'while true; do if echo "The password is hunter2" > password.txt; then echo "Password hunter2 logged."; else echo "Password log attempt blocked."; fi; sleep 5; done']

5. Minimizing Microservice Vulnerabilities

5.1. Exam Tips Section 5

Source: acloud.guru course

• Managing Container Access with Security Contexts
  • Tip 1: securityContext offers a variety of security and access control-related settings
  • Tip 2: spec.securityContext sets securityContext settings at the Pod level. These settings apply to all containers in the Pod
  • Tip 3: spec.containers[].securityContext sets securityContext settings at the container level. These settings apply to individual containers within the pod
• PodSecurityPolicies
  • Tip 1: Use Pod security policies to enforce desired security configurations for new Pods
  • Tip 2: Pod security policies can reject Pods that don't meet the desired standard, or modify Pods by applying default settings.
• Using Pod Security Policies
  • Tip 1: To use them you must first enable PodSecurityPolicy admission controller. Use the --enable-admission-plugins flag on the kube-apiserver to do this
  • Use documentation for v1.24.x, does not exist on v1.25.x documentation https://v1-24.docs.kubernetes.io/docs/concepts/security/pod-security-policy/
  • Where to enable isn't good documented, you have to add PodSecurityPolicy to --enable-admission-plugins in /etc/kubernetes/manifests/kube-apiserver.yaml
  • Tip 2: In order to create a Pod, a user (or the Pod's ServiceAccount) must be authorized to use a PodSecurityPolicy via the use verb in RBAC
  • Tip 3: To apply a PodSecurityPolicy within the context of a specific namepace, authorize a ServiceAccount in that namespace to use the policy.
• OPA Gatekeeper
  • Tip 1: Open Policy Agent (OPA) Gatekeeper allows you to enforce custom policies on any k8s object at creation time
  • Tip 2: Constraint Templates define reusable constraint logic and any parameters that can be passed in
  • Tip 3: Constraint objects apply a Constraint Template to a specific group of potential incoming objects, alongside specific parameters.
• Secrets
  • Tip 1: Secrets store sensitive data, and can pass it to containers
  • Tip 2: You can pass secret data to a container using either environment variables or mounted volumes
  • Tip 3: To retrieve secret data from the command line, you can use kubectl get -o yaml to get the base64-encoded data, then decode it with bas64 --decode
• Understanding Container Runtime Sandboxes
  • Tip 1: Container Runtime Sandboxes provide a specialized runtime with additional layers of isolation, allowing you to run untrusted workloads more securely
  • Tip 2: gVisor creates a runtime sandbox by running a Linux application kernel within the host OS. runssc is the OCI-compliant container runtime that allows Kubernetes to interface with gVisor
  • Tip 3. Kata Containers create a sandbox by transparently running containers inside of lightweight VMs.
• Creating a Container Runtime Sandbox
  • Tip 1: Use a RuntimeClass to define a specialized container runtime configuration, such as one that will use gVisor/runsc
  • Tip 2: Set the runtimeClassName property in a Pod specification to make the Pod use the container runtime sandbox
• Understanding Pod-to-Pod mTLS
  • Tip 1: mTLS means clients and servers mutually authenticate wit heach other and encrypt their communications
  • Tip 2: You can obtain certificates using the K8s API
• Signing Certificates
  • Tip 1: Create a CertificateSigningRequest object to request a new ceritificate
  • Tip 2: Manage, approve, or deny requests via the command line with kubectl certificate
  • Tip 3: Once approved, the signed certificate can be retrieved from the status.certificate field of the CertificateSigningRequest

5.2. securityContexts

Can be set in Pod spec and Container spec. Options differ for securityContext if you run it in Pod or Container context.

5.3. PodSecurityPolicies

Will be deprecated, documentation for v1.25.x not available any more, go to https://v1-24.docs.kubernetes.io/, see https://v1-24.docs.kubernetes.io/docs/concepts/security/pod-security-policy/

for old documentation

not turned on by default, admission controller must be enabled with

5.4. Using Pod Security Policies

Once turned on, you have to activate at least one policy, otherwise all pods are rejected.

RBAC needed to authorize via ServiceAccount (preferred over User)

5.4.1. Enable the Use of PodSecurityPolicies in the Cluster

1. Edit the manifest file for the Kube API server:

   sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml

2. Under the command section in the YAML file, modify the --enable-admission-plugins flag to read:

   - --enable-admission-plugins=NodeRestriction,PodSecurityPolicy

3. Save and exit the file by pressing the Escape key and entering:

   :wq

4. Ensure the files are working correctly. This may take a few tries before results appear:

   kubectl get nodes 

5.5. OPA Gatekeeper

rules to create an object.

Rego language used for actual rule programming in yaml.

A Constraint Template needed and we can create one or more constraints out of it. A constraint is an implementation of a constraint template.

The Constraint Template implements a new kind which we use to create new Constraint objects

5.6. Secrets

cloud_user@k8s-control:~$ kubectl get secrets -n users
NAME      TYPE     DATA   AGE
db-pass   Opaque   1      94m
cloud_user@k8s-control:~$ kubectl get secrets -n users db-pass -o yaml
apiVersion: v1
data:
  password: aHVudGVyMgo=
kind: Secret
metadata:

...

cloud_user@k8s-control:~$ echo aHVudGVyMgo= | base64 --decode
hunter2

5.7. Runtime Sandboxes

Could be interesting for multi tenancy, but is costly regarding performance.

5.7.1. implementations

gVisor/runsc: Kernel inside Kernel, something between Container and VM.

Kata Containers: lightweight VMs

https://kubernetes.io/docs/concepts/containers/runtime-class/ → https://kubernetes.io/docs/concepts/containers/runtime-class/#usage

5.7.1.1.1. Install gVisor

curl -fsSL https://gvisor.dev/archive.key | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64,arm64] https://storage.googleapis.com/gvisor/releases release main"
sudo apt-get update && sudo apt-get install -y runsc

# RuntimeClass is defined in the node.k8s.io API group
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  # The name the RuntimeClass will be referenced by.
  # RuntimeClass is a non-namespaced resource.
  name: myruntime
# The name of the corresponding CRI configuration
# this is the name we configured in /etc/containerd/config.toml
handler: runsc

usage

apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  runtimeClassName: runsc-sandbox

kubectl exec non-sandbox-pod -- dmesg

# first line shows kernel of current os, because it is running un-sandboxed.

 kubectl exec sandbox-pod -- dmesg

5.8. mTLS and Certificates

Mutual Transport Layer Security; both communication parties fully authenticate each other.

Certificate Signing

sidenode interesting tool cfssl to create certificates: https://github.com/cloudflare/cfssl

5.8.1. Create a CertificateSigningRequest

sudo apt-get install -y golang-cfssl

5.8.1.1.1. Generate a CSR file.

# according to https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/#create-a-certificate-signing-request
cat <<EOF | cfssl genkey - | cfssljson -bare k8cert.chloesoe.ch
{
    "hosts": [
        "saanen.chloesoe.ch",
        "lauenen.chloesoe.ch",
        "213.167.224.157",
        "185.142.215.9"
    ],
    "CN": "system:node:tls-pod.tls-test.pod.cluster.local",
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
    {
        "O": "system:nodes"
    }
    ]
}
EOF

# copy the base64 of the csr for later usag
cat k8cert.chloesoe.ch.csr | base64

Create a CertificateSigningRequest object https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/#create-a-certificatesigningrequest-object-to-send-to-the-kubernetes-api

# create CSR yaml, with base64 from CSR
cat >> tls-svc-csr.yml <<EOF
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: tls-svc-csr
spec:
  request: $(cat k8cert.chloesoe.ch.csr | base64 | tr -d '\n')
  signerName: example.com/serving
  usages:
  - digital signature
  - key encipherment
  - server auth
EOF

# create object
kubectl create -f tls-svc-csr.yml

# check status
kubectl get csr

Approve the CertificateSigningRequest

sudo falco -r nginx-rules.yml -M 45kubectl get csr -o wide
NAME          AGE   SIGNERNAME                    REQUESTOR          REQUESTEDDURATION   CONDITION
tls-svc-csr   5s    example.com/serving           kubernetes-admin   <none>              Pending

# approve
kubectl certificate approve tls-svc-csr

# now approved, should alse be Issued in order to have a certificate
kubectl get csr
NAME          AGE   SIGNERNAME                    REQUESTOR          REQUESTEDDURATION   CONDITION
tls-svc-csr   51s   example.com/serving           kubernetes-admin   <none>              Approved,Issued

# Retrieve the signed certificate data.
kubectl get csr tls-svc-csr -o yaml
kubectl get csr tls-svc-csr -o jsonpath='{.status.certificate}' | base64 --decode

6. Supply Chain Security Intro

6.1. Exam Tips Section 6

Source: acloud.guru course

• Images
  • Minimizing Base Image Attack Surface
    • Tip 1: Try to use images that run up-to-date software to minimize software vulnerabilities
    • Tip 2: Minimize the presence of unnecessary software in images that could increase security risks
    • Tip 3: Beware of the possiblity of images that have been compromised by an attacker
• Whitelisting Registries
  • Tip 1: Limit users to only trusted image registries to prevent them from running images from untrusted sources in the cluster
  • Tip 2: You can limit registries using OPA Gatekeeper
• Image Validation
  • Tip 1: Container images can be signed with a hash generated from the image contents
  • Tip 2: To validate the image, you can append the hash to the image reference in your container spec with image: imageName:tag@sha256:hash
• Static Analyzing
  • Dockerfiles
    • Tip 1: Avoid user root → Final USER directive must not be root or 0
    • Tip 2: Avoid :latest tag
    • Tip 3: Avoid unnecessary Software
    • Tip 4: Avoid not storing sensitive Data
  • YAML Files
    • Tip 1: When possible, avoid Host Namespaces in your Pod configuration (i.e. hostIPC, hostNetwork, hostPID)
    • Tip 1: When possible, avoid Privileged mode with privileged: true
    • Tip 3: Avoid running user as root or 0 in securityContext.runAsUser
    • Tip 3: Don't use the :latest tag, but instead use a specific fixed tag to avoid downloading a new and potentially unvetted image
• Vulnerability Scanning
  • Tip 1: Vulnerability scanning allows you to scan images to detect security vulnerabilities that have already been discovered and documented by security researchers
  • Tip 2: Trivy is a command-line tool that allows you to scan images by name and tag
  • Tip 3: Scan an image name with Trivy with: trivy image quay.io/keycloak/keycloak:20.0.0
  • Tip 4: In older versions it could be only trivy $IMAGENAME without "image" keyword
• Vulnerabiltiy Scanning with an Admission Controller
  -Scanning
  • Tip 1: Admission controllers intercept requests to the Kubernetes API before objects are created. They can allow objects to be created, they can prevent the creation of those objects, or make changes to objects before creating them.
  • Tip 2: The ImagePolicyWebhook admission controller allows you to use customizable logic to approve or deny the creation of workloads based upon the container images being used.
  • Tip 3: You can use the ImagePolicyWebhook admission controller to have an external application scan images for vulnerabilities automatically as workloads are created.
    • Set up and Image Server
  • Tip 1: First, the ImagePolicyWebhook admission controller sends a JSON request to an external service to determine if images are allowed.
  • Tip 2: The external service provides a JSON response indicating whether the images are allowed or disallowed.
    • Configuring the ImagePolicyWebhook Admission Controller
  • Tip 1: Use the --enable-admission-plugins flag in the kube-apiserver manifest to enable the ImagePolicyWebhook admission controller.
  • Tip 2: Use the --admission-control-config-file flag, also in that same manifest, to specify the location of the admission control configuration file.
  • Tip 3: If the config files are on the host file system, you may need to mount them to the kube-apiserver container.
  • Tip 4: In the admission control config file, the kubeConfigFile option specifies the location of a kubeconfig. This kubeconfig file is what tells ImagePolicyWebhook how to reach the webhook backend.

6.2. Images

see exam tips

https://kubernetes.io/docs/concepts/security/overview/

6.3. Whitelisting Registries

OPA Gatekeeper. ConstraintTemplate with rules and defining new Kind, and K8sAllowedRepos with actual whitelisting

6.3.1.1.1. k8sallowedrepos.yml

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
 name: k8sallowedrepossudo falco -r nginx-rules.yml -M 45 spec:
  crd:
    spec:
      names:
        kind: K8sAllowedRepos
      validation:
        # Schema for the `parameters` field
        openAPIV3Schema:
          properties:
            repos:
              type: array
              items:
                type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
    rego: |
      package k8sallowedrepos
      violation[{"msg": msg}] {
      container := input.review.object.spec.containers[_]
        satisfied := [good | repo = input.parameters.repos[_] ; good = startswith(container.image,repo)]
        not any(satisfied)
        msg := sprintf("container <%v> has an invalid image repo <%v>, allowed repos are %v",[container.name, container.image, input.parameters.repos])
      }
      violation[{"msg": msg}] {
      container := input.review.object.spec.initContainers[_]
        satisfied := [good | repo = input.parameters.repos[_] ; good = startswith(container.image,repo)]
        not any(satisfied)
        msg := sprintf("container <%v> has an invalid image repo <%v>, allowed repos are %v",[container.name, container.image, input.parameters.repos])
}

6.3.1.1.2. whitelist-dockerhub.yml

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sAllowedRepos
metadata:
  name: whitelist-dockerhub
spec:
  match:
    kinds:
    - apiGroups: [""]
      kinds: ["Pod"]
  parameters:
    repos:
    - "docker.io"

6.4. Image Validation

6.5. Static Analysis

6.5.1. Dockerfiles

https://docs.docker.com/engine/reference/builder/

To look for in Dockerfiles:

1. Avoid user root
2. Avoid latest tag
3. unnecessary Software
4. Verify no sensitive Data

6.5.2. YAML files

Don't use:

1. Host Namespaces (aka hostIPC, hostNetwork, hostPID)
2. Privileged mode
3. :latest tag
4. Run as root

6.6. Vulnerability Scanning

Use Trivy, documentation at https://github.com/aquasecurity/trivy/blob/main/docs/index.md

Install via apt according to: https://github.com/aquasecurity/trivy/blob/main/docs/getting-started/installation.md#debianubuntu

for Linux Mint you have to use the UBUNTU_CODENAME so we can use this command to add the deb:

echo "deb [signed-by=/usr/share/keyrings/trivy.gpg] https://aquasecurity.github.io/trivy-repo/deb  $(cat /etc/os-release  | grep UBUNTU_CODENAME | awk -F'[/=]' '{ print $2}') main" | sudo tee -a /etc/apt/sources.list.d/trivy.list

6.6.1. get trivy running

1. create token at github, (see this issue for the why)
   1. https://github.com/settings/tokens
   2. Copy the token
2. Probably something more to install, because docker login does not work https://stackoverflow.com/a/74006782/7311363

   sudo apt install golang-docker-credential-helpers

3. according to https://github.com/aquasecurity/trivy/issues/2373#issuecomment-1163355311

   export CR_PAT=ghp_NEW_TOKEN
   echo $CR_PAT | docker login ghcr.io -u $USERNAME --password-stdin
   Login Succeeded
   
   trivy image alpine:latest
   
   # you can now check on other registries
   trivy image quay.io/vshn/keycloak-theme:v1.4.0

6.6.2. some commands

find image from pods:

# only for one specific pod
kubectl -n questionablesoft get pod admin-web-host -o jsonpath='{..image}{"\n}'

# for all pods in a namespace
kubectl -n questionablesoft get pods  -o jsonpath='{range .items[*]}{.containers}{"\n"}{end}'

# get all images from a all namespace and log vulnerabilitiy to log file (and replace / in log file name
kubectl get pod -A -o jsonpath='{range .items[*]}{..image}{"\n"}{end}' | awk '{ print $1 }' | xargs -I{} sh -c 'trivy image {} > $(echo {} | sed "s|\/|-|g").log'
# check which logs have critical vulnerabilites
grep -lre "CRITICAL: [1-9]"

6.7. Vulnerabiltiy Scanning with an Admission Controller

how to turn on https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/, keyword is ImagePolicyWebhook

some changes in /etc/kubernetes/manifest/kube-apiserver.yaml

in the AdmissionConfiguration template we have a kubeConfigFile. In this file we need to configure the admission server for the checks. It must be with https

6.8. Copy from Lab

6.8.1. Configure the Admission Controller

1. Edit the admission-control.conf file:
   sudo vi /etc/kubernetes/admission-control/admission-control.conf
2. Paste in the ImagePolicyWebhook:

apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: ImagePolicyWebhook
  configuration:
    imagePolicy:
      kubeConfigFile: /etc/kubernetes/admission-control/imagepolicy_backend.kubeconfig
      allowTTL: 50
      denyTTL: 50
      retryBackoff: 500
      defaultAllow: false

6.8.2. Backend Webhook

1. Edit the kubeconfig file:
   sudo vi /etc/kubernetes/admission-control/imagepolicy_backend.kubeconfig
2. Set the location of the backend image scanning service:
   server: https://acg.trivy.k8s.webhook:8090/scan
3. To save and exit the file, press Escape, type :wq, and hit Enter.

6.8.3. Enable Any Necessary Admission Control Plugins

1. Edit the kube-apiserver manifest:
   sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml
2. In the command container, scroll down to --enable-admission-plugins and add ImagePolicyWebhook:
   --enable-admission-plugins=NodeRestriction,ImagePolicyWebhook
3. To save and exit the file, press Escape, type :wq, and hit Enter.
4. Once the kube-apiserver has been re-created, create the good-pod Pod:
   kubectl create -f good-pod.yml
   The Pod should be successfully created.
5. Attempt to create the bad-pod Pod:
   kubectl create -f bad-pod.yml
   Pod creation should fail and return an error due to image vulnerabilities.

7. Monitoring, Logging, and Runtime Security

7.1. Exam Tips Section 7

Source: acloud.guru course

• Bahvioral Analytics
  • Understanding Bahvioral Analytics
    • Tip 1: Behavioral analytics is the process of monitoring what's happening within a system to detect malicious activity.
    • Tip 2: Secondly, one way to perform behavioral analytics in Kubernetes is to use tools like Falco.
  • Analyzing Container Behavior with Falco
    • Tip 1: You can run Falco from the command line with the falco command, and you can view Falco's options with falco --help.
    • Tip 2: Use the -r flag to pass in a Falco rules file
    • Tip 3: Use falco --list to see all available fields for conditions and outputs.
    • Tip 4: Use the -M flag to set the number of seconds that Falco should collect data for. (e.g. -M 45 to run for 45 seconds)
• Immutable Containers
  • Tip 1: Immutability means that containers do not change at runtime by downloading and running new code or changing their code in any other way.
  • Tip 2: Containers that use privileged mode such as securityContext.privileged may also be considered mutable.
  • Tip 3: Set a readOnlyRootFilesystem using container.securityContext.readOnlyRootFilesysstem to prevent a container from writing to its file system and therefore, help make that container immutable.
  • Tip 4: If an application does need to write to files such as for caching or logging, you can use an emptyDir volume alongside readOnlyRootFilesystem for those locations where the application does actually need to write data.
• Audit Logging
  • Understanding Audit Logging
    • Tip 1: Kubernetes auditing allows you to capture logs of all changes made through the Kubernetes API
    • Tip 2: Your audit log policy rules, the level, identifies how detailed the log data should be for the rule.
    • None - which logs nothing,$
    • RequestResponse - which logs everything, including the request and response body.
    • Request - logs only the request body, but not the response body.
    • Metadata - logs only the basic high-level metadata about the request.
    • Tip 3: In the audit log policy rules, that resources field identifies which Kubernetes resource types the rule applies to
    • Tip 4: The namespaces field, which is optional, limits the rule to only specific namespaces.
  • Setting Up Audit Logging
    • Tip 1: Define audit rules in the audit policy configuration file
    • Tip 2: kube-apiserver flags for audit logging in /etc/kubernetes/manifests/kube-apiserver.yaml:
    • --audit-policy-file - Points to the audit policy config file
    • --audit-log-path - Location of output files
    • --audit-log-maxage - The number of days to keep old log files
    • --audit-log-maxbackup - The number old log files to keep

7.2. Bahvioral Analytics

Official docs: https://falco.org/docs/

Falco a tool for such analysis

Falco can be run in cli with:

# -r <file> to supply a custom rules file
# -M <seconds> to run Falco for X seconds
# falco should run on the worker where the container is runnig
falco -r rules.yml -M 45

7.2.1. falco rules

example from Lab, see https://falco.org/docs/rules/supported-fields/ for available fields

- rule: spawned_process_in_nginx_container
  desc: A process was spawned in the Nginx container.
  condition: container.name = "nginx" and evt.type = execve  and evt.dir=<
  output: "Process Spawned, %evt.time.iso8601, %evt.datetime, %proc.name, %user.uid, %container.id, %container.name, %container.image"
  priority: WARNING

7.3. Immutable Containers

Immutability:

Use Volume Mounts if you want to write

7.3.1. how to check for immutability

1. use kubectl get pod $POD -o yaml and check the spec section for Priviledge modes or R/W filesystems
   1. see securityContext:
2. something like that:

   spec:
     containers:
     - image: nginx:1.19.1
       imagePullPolicy: IfNotPresent
       name: nginx
       resources: {}
       securityContext:
         allowPrivilegeEscalation: true
         runAsUser: 0

7.4. Audit Logging

https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/

7.4.1. Lab notes

my basteling for /etc/kubernetes/audit-policy.yaml, based on https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/#audit-policy

apiVersion: audit.k8s.io/v1
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "RequestReceived"
rules:
# Log request and response bodies for all changes to namespaces.
  - level: RequestResponse
    resources:
    - group: ""
      resources: ["namespaces"]
# Log request bodies (but not response bodies) for changes to Pods and Services in the web namespace.
  - level: Request
    resources:
    - group: ""
      resources: ["pods", "services"]
    namespaces: ["web"]
# Log metadata for all changes to Secrets.
  - level: Metadata
    resources:
    - group: "" # core API group
      resources: ["secrets"]

# A catch-all rule to log all other requests at the Metadata level.
  - level: Metadata

Edit kube-apiserver flags for audit logging in /etc/kubernetes/manifests/kube-apiserver.yaml:

- command:
  - kube-apiserver
  - ...
  - --audit-policy-file=/etc/kubernetes/audit-policy.yaml
  - --audit-log-path=/var/log/kubernetes/k8s-audit.log
  - --audit-log-maxage=60
  - --audit-log-maxbackup=1

Acloud.Guru Lab01 - Service Account

Fix Pod's Service Account

my basteling:

cloud_user@k8s-cli:~/lab01$ cat buffy-role-sa.yml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: buffy-role
  namespace: sunnydale
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["list"]

cloud_user@k8s-cli:~/lab01$ cat buffy-sa-watch-rb.yml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: buffy-sa-watch-rb
  namespace: sunnydale
subjects:
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: watch-services-secrets
subjects:
- kind: ServiceAccount
  name: buffy-sa
  namespace: sunnydale

cloud_user@k8s-cli:~/lab01$ cat watch-services-secrets.yml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: watch-services-secrets
  namespace: sunnydale
rules:
- apiGroups: [""]
  resources: ["services", "secrets"]
  verbs: ["watch"]

Fix a Pod Configure to Use Incorrect SA

my basteling

cloud_user@k8s-cli:~/lab01-2$ cat kaput-lando.yml
apiVersion: v1
kind: ServiceAccount
metadata:
  creationTimestamp: "2022-11-23T17:23:42Z"
  name: lando
  namespace: bespin
  resourceVersion: "727"
  uid: 9027d6e7-ba08-4863-9a96-722c020c883a

cloud_user@k8s-cli:~/lab01-2$ cat lando-sa.yml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: lando-sa
  namespace: bespin
automountServiceAccountToken: false

cloud_user@k8s-cli:~/lab01-2$ cat lando.yml
apiVersion: v1
kind: Pod
metadata:
  name: lando
  namespace: bespin
spec:
  serviceAccountName: lando-sa
  containers:
  - name: busybox
    image: busybox:1.33.1
    command: ['sh', '-c', 'while true; do sleep 5; done']

Acloud.Guru Lab02 – Control Plane

broken control plane

edit on /var/lib/kubelet/config.yaml (not /etc/kubernetes/manifests/kube-apiserver.yaml as one could mean)

apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
...
 authorization:
    mode: Webhook

second question profiling

now we have /etc/kubernetes/manifests/kube-apiserver.yaml

spec:
  containers:
  - command:
    - kube-apiserver
...
    - --authorization-mode=Node,RBAC
    - --profiling=false

third question - etcd

/etc/kubernetes/etcd.yaml
spec:
  containers:
  - command:
    - etcd
    - --advertise-client-urls=https://10.0.1.101:2379
    - --cert-file=/etc/kubernetes/pki/etcd/server.crt
    - --client-cert-auth=true

Acloud.Gur Lab 03 – Network Policy

Default deny Network Policy

→ below actually wrong: no "Egress" needed, they only talk about ingress

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: mordor
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

create NetworkPolicy

use k -n mordor get pods mtdoom --show-labels  to show labels directly

for the second selector they say, we should add "namespaceSelector: {}" to allow traffic from all namespaces.

ensure for the second rule you also have the "-", so it an OR.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: mtdoom-np
  namespace: mordor
spec:
  podSelector:
    matchLabels:
      app: mtdoom
  policyTypes:
    - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          app: frodo
    - podSelector:
        matchLabels:
          app: sam
      namespaceSelector: {}
    ports:
    - port: 80
      protocol: TCP

Acloud.Guru Lab04 - Image Policy

configure in separate file: /etc/kubernetes/admission-control/imagepolicy.conf

{
   "imagePolicy": {
      "kubeConfigFile": "/etc/kubernetes/admission-control/imagepolicy_backend.kubeconfig",
      "allowTTL": 50,
      "denyTTL": 50,
      "retryBackoff": 500,
      "defaultAllow": false
   }
}

https://acg.trivy.k8s.webhook:8090/scan musst be added to /etc/kubernetes/admission-control/imagepolicy_backend.kubeconfig from kubeconfig file above

For enabling:

• Open the kube-apiserver manifest for editing:
  sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml
• In the --enable-admissions-plugin line under command, add our image policy webhook to the list:
  --enable-admissions-plugins=NodeRestriction,ImagePolicyWebhook

Acloud.Guru Lab 05 - Trivy

on control plane we could run it directly with:

kubectl get pods -n sunnydale -o jsonpath="{range .items[*]}{..image}{'\n'}{end}"  | cut -f1 -d' '  | xargs -I{} sh -c 'trivy image {} >{}.log'

egrep -l "CRITICAL: [1-9]|HIGH: [1-9]" *.log
  centos:7.log
  nginx:1.14.2.log

# unfortunately the correlation to the pod has to be done manually
k get pods -n sunnydale -o yaml | less

# first is xander
# second is cordelia

k -n sunnydale delete  pods cordelia xander --force

############## From Solution #############
# direct way also including correlation
kubectl get pods -n sunnydale --output=custom-columns="NAME:.metadata.name,IMAGE:.spec.containers[*].image"

Acloud.Guru Lab 06 - App Armor

AppArmor Profile enable

1. Login to worker
2. sudo apparmor_parser k8s-deny-write
3. sudo cp k8s-deny-write /etc/apparmor.d/ (to make it persistent)

Pod with AppArmor profile:

1. follow https://kubernetes.io/docs/tutorials/security/apparmor/#example
2. pod wit happarmor

   apiVersion: v1
   kind: Pod
   metadata:
   name: chewbacca
   namespace: kashyyyk
   annotations:
   # Tell Kubernetes to apply the AppArmor profile "k8s-apparmor-example-deny-write".
   # annotation must have container name as key and the name of the apparmor rule (in the apparmor file) as value
   container.apparmor.security.beta.kubernetes.io/busybox: localhost/k8s-deny-write
   spec:
   containers:
   - name: busybox
   image: busybox:1.33.1
   command: ['sh', '-c', 'while true; do echo hunter2 >> password.txt; sleep 5; done'

3. apply gives error

   k apply -f chewbacca.yml
   error: error validating "chewbacca.yml": error validating data: ValidationError(Pod.metadata): unknown field "container.apparmor.security.beta.kubernetes.io/busybox" in io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta; if you choose to ignore these errors, turn validation off with --validate=false

Acloud.Guru Lab 07 - Falco

Container to monitor defined in condition
monitor_rules.yml

- rule: something_spawned
  desc: Spawned process
  condition: container.name = "monitor" and evt.type = execve and evt.dir=<
  output: "%evt.datetime,%container.id,%container.image,%user.uid,%proc.name"
  priority: NOTICE

solution is %evt.time instead of %evt.datetime, but that only prints the time without date, what is something hateful

run rules:

cloud_user@k8s-worker1:~$ sudo falco -r monitor_rules.yml -M 45 > falco_output.log

cloud_user@k8s-worker1:~$ less falco_output.log
07:09:12.524848344: Notice 2022-11-24 07:09:12.524848344,393544c622a5,docker.io/library/busybox:1.33.1,0,cat
07:09:12.525215915: Notice 2022-11-24 07:09:12.525215915,393544c622a5,docker.io/library/busybox:1.33.1,0,sleep
Events detected: 2
Rule counts by severity:
   NOTICE: 2
Triggered rules by rule name:
   something_spawned: 2

Some falco rules used on Killer.sh:

# The explicit quotes are needed to avoid the - characters being
# interpreted by the filter expression.
- list: rpm_binaries
  items: [dnf, rpm, rpmkey, yum, '"75-system-updat"', rhsmcertd-worke, rhsmcertd, subscription-ma,
          repoquery, rpmkeys, rpmq, yum-cron, yum-config-mana, yum-debug-dump,
          abrt-action-sav, rpmdb_stat, microdnf, rhn_check, yumdb]

- list: openscap_rpm_binaries
  items: [probe_rpminfo, probe_rpmverify, probe_rpmverifyfile, probe_rpmverifypackage]

- macro: rpm_procs
  condition: (proc.name in (rpm_binaries, openscap_rpm_binaries) or proc.name in (salt-minion))

- list: deb_binaries
  items: [dpkg, dpkg-preconfigu, dpkg-reconfigur, dpkg-divert, apt, apt-get, aptitude,
    frontend, preinst, add-apt-reposit, apt-auto-remova, apt-key,
    apt-listchanges, unattended-upgr, apt-add-reposit, apt-config, apt-cache, apt.systemd.dai
    ]
- list: package_mgmt_binaries
  items: [rpm_binaries, deb_binaries, update-alternat, gem, pip, pip3, sane-utils.post, alternatives, chef-client, apk, snapd]

- macro: package_mgmt_procs
  condition: proc.name in (package_mgmt_binaries)

- macro: container
  condition: container.id != host

- macro: spawned_process
  condition: evt.type = execve and evt.dir=<

- rule: kill_process
  desc: kill_process
  #condition: spawned_process and container and package_mgmt_procs
  #condition: spawned_process and container and package_mgmt_procs and container.image contains "nginx"
  condition: evt.type = kill and container
  #condition: spawned_process and container and container.image = "nginx" and package_mgmt_binaries
  output: "%evt.time,%container.id,%container.name,%user.name -- %k8s.deployment.name, %k8s.pod.name"
  priority: WARNING

- rule: nginx_image_is_running
  desc: nginx image with apt
  #condition: spawned_process and container and package_mgmt_procs
  condition: spawned_process and container and package_mgmt_procs and container.image contains "nginx"
  #condition: spawned_process and container and container.image = "nginx" and package_mgmt_binaries
  output: "%evt.time,%container.id,%container.name,%user.name"
  priority: WARNING

Acloud.Guru Lab 08 - Audit Logging

Create rules, there are good examples at https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/#log-backend

/etc/kubernetes/audit-policy.yaml

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - "RequestReceived"
rules:
  - level: None
    resources:
    - group: ""
      resources: ["pods/log", "pods/status"]
  - level: RequestResponse
    resources:
    - group: ""
      resources: ["configmaps"]
  - level: Request
    resources:
    - group: ""
      resources: ["services", "pods"]
    namespaces: ["web"]
  - level: Metadata
    resources:
    - group: ""
      resources: ["secrets"]
  # A catch-all rule to log all other requests at the Metadata level.
  - level: Metadata

You could follow https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/#log-backend

Add values to /etc/kubernetes/manifests/kube-apiserver.yaml

spec:
  containers:
  - command:
    - kube-apiserver
    - --audit-policy-file=/etc/kubernetes/audit-policy.yaml
    - --audit-log-path=/var/log/kubernetes/audit.log
    - --audit-log-maxage=10
    - --audit-log-maxbackup=1

The volumes are already mounted, but would have been straight forward with the documentation.

Acloud.Guru Lab 09 - Secrets

jsonpath docu on k8s docs is cool

cloud_user@k8s-cli:~$ k get secrets -n larry curly
NAME    TYPE     DATA   AGE
curly   Opaque   2      71m
cloud_user@k8s-cli:~$ k get secrets -n larry curly -o yaml
apiVersion: v1
data:
  password: MTIzNDUK
  username: YWRtaW4K
kind: Secret
metadata:
  creationTimestamp: "2022-11-24T06:38:08Z"
  name: curly
  namespace: larry
  resourceVersion: "608"
  uid: ba44fc94-2159-4455-acfd-de69634dc3bf
type: Opaque

cloud_user@k8s-cli:~$ kubectl -n larry get secrets -o=jsonpath='{range .items[*]}{.metadata.name}{"\tusername="}{.data.username}{"\tpassword="}{.data.password}{"\n"}{end}'
curly   username=YWRtaW4K   password=MTIzNDUK

cloud_user@k8s-cli:~$ kubectl -n larry get secrets curly -o=jsonpath='{.data.username}' | base64 --decode
admin
cloud_user@k8s-cli:~$ kubectl -n larry get secrets curly -o=jsonpath='{.data.password}' | base64 --decode
12345

# save them directly:
cloud_user@k8s-cli:~$ kubectl -n larry get secrets curly -o=jsonpath='{.data.username}' | base64 --decode > username.txt
cloud_user@k8s-cli:~$ kubectl -n larry get secrets curly -o=jsonpath='{.data.password}' | base64 --decode > password.txt

you can convert to base in ViM direct and insert it to a secret :read !sh -c 'echo dbuser | base64'

moe.yml and secret-pod.yml

---
apiVersion: v1
kind: Secret
data:
  password: QTgzTWFlS296Cg==
  username: ZGJ1c2VyCg==
metadata:
  name: moe
  namespace: larry
type: Opaque

---
apiVersion: v1
kind: Pod
metadata:
  name: secret-pod
  namespace: larry
spec:
  containers:
  - name: busybox
    image: busybox:1.33.1
    command: ['sh', '-c', 'cat /etc/credentials/username; cat /etc/credentials/password; while true; do sleep 5; done']
    volumeMounts:
    - name: secret-volume
      readOnly: true
      mountPath: "/etc/credentials"
  volumes:
  - name: secret-volume
    secret:
      secretName: moe
cloud_user@k8s-cli:~$

verify

cloud_user@k8s-cli:~$ k -n larry exec secret-pod -- sh -c 'cat  /etc/credentials/*'
A83MaeKoz
dbuser

Acloud.Guru Lab 10 - Analyzing Dockerfile and YAML

Dockerfile fix

cloud_user@k8s-cli:~$ diff Dockerfile*
1c1
< FROM alpine:3.13.5
---
> FROM alpine:latest
6c6
< USER nobody
---
> USER root

YAML fixing

cloud_user@k8s-cli:~$ diff scooby-gang-deploy.yml*
20,21c20,21
<           privileged: false
<           runAsUser: 65534
---
>           privileged: true
>           runAsUser: 0

find mutable/immutable containers

# browse throu all pods in the namespace
k -n sunnydale get pods -o yaml | less

### looks like Pod "buffy" and "xander" ar mutable:
      securityContext:
        allowPrivilegeEscalation: false
        runAsUser: 0
...
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: false

# delete the bad pods

k -n sunnydale delete pods buffy xander --force

Acloud.Guru Lab 11 - gVisor

sandbox.yml

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: sandbox
handler: runsc

looks like runsc already is installed:

on control plane server
cloud_user@k8s-control:~$ sudo grep runsc -A1 /etc/containerd/config.toml
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc]
          runtime_type = "io.containerd.runsc.v1"

some notes to find out where to change the runtime:

cloud_user@k8s-control:~$ k -n sunnydale get pods
NAME                     READY   STATUS    RESTARTS   AGE
buffy-5c9f5f7d54-vqhbc   1/1     Running   0          143m
giles-5c9f5f7d54-s8nx7   1/1     Running   0          143m
spike-5c9f5f7d54-dvddg   1/1     Running   0          143m
cloud_user@k8s-control:~$ k -n sunnydale get all
NAME                         READY   STATUS    RESTARTS   AGE
pod/buffy-5c9f5f7d54-vqhbc   1/1     Running   0          143m
pod/giles-5c9f5f7d54-s8nx7   1/1     Running   0          143m
pod/spike-5c9f5f7d54-dvddg   1/1     Running   0          143m

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/buffy   1/1     1            1           143m
deployment.apps/giles   1/1     1            1           143m
deployment.apps/spike   1/1     1            1           143m

NAME                               DESIRED   CURRENT   READY   AGE
replicaset.apps/buffy-5c9f5f7d54   1         1         1       143m
replicaset.apps/giles-5c9f5f7d54   1         1         1       143m
replicaset.apps/spike-5c9f5f7d54   1         1         1       143m  

# check kernel running
cloud_user@k8s-control:~$ k -n sunnydale exec buffy-5c9f5f7d54-vqhbc -- uname -a
Linux buffy-5c9f5f7d54-vqhbc 5.13.0-1025-aws #27~20.04.1-Ubuntu SMP Thu May 19 15:17:13 UTC 2022 x86_64 GNU/Linux
cloud_user@k8s-control:~$ k -n sunnydale exec giles-5c9f5f7d54-s8nx7 -- uname -a
Linux giles-5c9f5f7d54-s8nx7 5.13.0-1025-aws #27~20.04.1-Ubuntu SMP Thu May 19 15:17:13 UTC 2022 x86_64 GNU/Linux
cloud_user@k8s-control:~$ k -n sunnydale exec spike-5c9f5f7d54-dvddg -- uname -a
Linux spike-5c9f5f7d54-dvddg 5.13.0-1025-aws #27~20.04.1-Ubuntu SMP Thu May 19 15:17:13 UTC 2022 x86_64 GNU/Linux

Edit the deployments and add spec to the container spce:

k -n sunnydale edit deployments.apps buffy

# edit the yaml at spec
 33     spec:
 34       runtimeClassName: sandbox
 35       containers:
 36       - command:

#### verify

kubectl get -n sunnydale pods -o=jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | xargs -I {} bash -c 'kubectl -n sunnydale exec {} -- uname -a'
Linux buffy-6ddc7d454-7vh7s 4.4.0 #1 SMP Sun Jan 10 15:06:54 PST 2016 x86_64 GNU/Linux
Linux giles-6ddc7d454-swp5w 4.4.0 #1 SMP Sun Jan 10 15:06:54 PST 2016 x86_64 GNU/Linux
Linux spike-6ddc7d454-9n74c 4.4.0 #1 SMP Sun Jan 10 15:06:54 PST 2016 x86_64 GNU/Linux

# with gVisor dmesg work, you could verify in one step with:
cloud_user@k8s-cli:~$ kubectl get -n sunnydale pods -o=jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | xargs -I {} bash -c 'echo {}; kubectl -n sunnydale exec {} -- sh -c "dmesg | head -2"'
buffy-6ddc7d454-7vh7s
[   0.000000] Starting gVisor...
[   0.166268] Generating random numbers by fair dice roll...
giles-6ddc7d454-swp5w
[   0.000000] Starting gVisor...
[   0.555445] Searching for socket adapter...
spike-6ddc7d454-9n74c
[   0.000000] Starting gVisor...
[   0.388231] Preparing for the zombie uprising...

Acloud.Guru Lab 12 - Pod Security Policy

https://v1-24.docs.kubernetes.io/docs/concepts/security/pod-security-policy/

PodSecurityPolicy
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: nopriv-psp
spec:
  privileged: false
  runAsUser:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  volumes:
  - '*'

Use the policy:

cloud_user@k8s-cli:~$ kubectl create sa -n hoth hoth-sa

## Cluster Role

cloud_user@k8s-cli:~$  cat use-nopriv-psp.yml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: use-nopriv-psp
rules:
- apiGroups: ['policy']
  resources: ['podsecuritypolicies']
  verbs:     ['use']
  resourceNames: ['nopriv-psp']

cloud_user@k8s-cli:~$ k create -f use-nopriv-psp.yml
clusterrole.rbac.authorization.k8s.io/use-nopriv-psp created

cloud_user@k8s-cli:~$ cat hoth-sa-use-nopriv-psp.yml

### Role Binding

cloud_user@k8s-cli:~$ cat hoth-sa-use-nopriv-psp.yml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: hoth-sa-use-nopriv-psp
roleRef:
  kind: ClusterRole
  name: use-nopriv-psp
  apiGroup: rbac.authorization.k8s.io
subjects:
# Authorize specific service accounts (not recommended):
- kind: ServiceAccount
  name: hoth-sa
  namespace: hoth

cloud_user@k8s-cli:~$ k create -f hoth-sa-use-nopriv-psp.yml
clusterrolebinding.rbac.authorization.k8s.io/hoth-sa-use-nopriv-psp create

### test
cloud_user@k8s-cli:~$ k create -f luke.yml
Error from server (Forbidden): error when creating "luke.yml": pods "luke" is forbidden: PodSecurityPolicy: unable to admit pod: [spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed]

killer.sh

there are some exerciese you have to use crictl, check it out on the docs