{"id":695,"date":"2022-01-15T22:14:44","date_gmt":"2022-01-15T20:14:44","guid":{"rendered":"https:\/\/blog.chloesoe.ch\/?p=695"},"modified":"2022-01-15T22:20:23","modified_gmt":"2022-01-15T20:20:23","slug":"cis-benchmarks-k8s","status":"publish","type":"post","link":"https:\/\/blog.chloesoe.ch\/?p=695","title":{"rendered":"CIS Benchmarks K8s"},"content":{"rendered":"

Overview CIS Benchmarks on K8s<\/h2>\n

From CKS there is a chapter about CIS Benchmarks with kube-bench.<\/p>\n

Benchmarks are coming from https:\/\/www.cisecurity.org\/benchmark\/kubernetes\/<\/a>. You have to go to "Download Latest CIS Benchmark" and register, then you'll get a link an can download all relevant benchmarks, for this how-to we used Fixes are based on "CIS Kubernetes V1.20 Benchmark - v1.0.0" from 2021-05-19.<\/p>\n

This documents what changes are needed, to fix all FAILs.<\/p>\n

\n

Get Benchmarks<\/h2>\n

First check https:\/\/github.com\/aquasecurity\/kube-bench\/blob\/main\/docs\/platforms.md<\/a> and take a note which benchmark version you want to check against.<\/p>\n

For my cluster running with K8s version 1.23.1 we use cis-1.20<\/code> version.<\/p>\n

wget -O kube-bench-control-plane.yaml https:\/\/raw.githubusercontent.com\/aquasecurity\/kube-bench\/main\/job-\nmaster.yaml\nwget -O kube-bench-node.yaml https:\/\/raw.githubusercontent.com\/aquasecurity\/kube-bench\/main\/job-node.yaml<\/code><\/pre>\n
Now you could change the startup command in the job yamls, to add the benchmark you want to use. You could also define --version<\/code> instead of --benchmark<\/code>, but it should be done automatically:<\/p>\n
      containers:\n        - name: kube-bench\n          image: aquasec\/kube-bench:latest\n          command: ["kube-bench", "run", "--benchmark", "cis-1.20", "--targets", "master"]<\/code><\/pre>\n
Now create the jobs:<\/p>\n
kubectl create -f kube-bench-control-plane.yaml\nkubectl create -f kube-bench-node.yaml\n\nkubectl get pods\n\nkubectl logs kube-bench-master-<RAND> > bench-master.log\nkubectl logs kube-bench-node-<RAND> > bench-worker.log<\/code><\/pre>\n
Control Plane<\/h3>\n
Get the output for master:<\/p>\n
$ grep FAIL bench-master.log \n[FAIL] 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated)\n[FAIL] 1.2.5 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated)\n[FAIL] 1.2.15 Ensure that the admission control plugin PodSecurityPolicy is set (Automated)\n[FAIL] 1.2.18 Ensure that the --insecure-port argument is set to 0 (Automated)\n[FAIL] 1.2.20 Ensure that the --profiling argument is set to false (Automated)\n[FAIL] 1.2.21 Ensure that the --audit-log-path argument is set (Automated)\n[FAIL] 1.2.22 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated)\n[FAIL] 1.2.23 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated)\n[FAIL] 1.2.24 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated)\n[FAIL] 1.3.2 Ensure that the --profiling argument is set to false (Automated)\n[FAIL] 1.4.1 Ensure that the --profiling argument is set to false (Automated)<\/code><\/pre>\n
Worker Nodes<\/h3>\n
Get the output of the worker:<\/p>\n
$ grep FAIL bench-worker.log \n[FAIL] 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated)<\/code><\/pre>\n
\n
Fixing Control Plane<\/h2>\n
Below documented the failed controls. <\/p>\n
Fixes are based on "CIS Kubernetes V1.20 Benchmark" v1.0.0 - 2021-05-19.<\/p>\n
Summary<\/h3>\n
After fixing, we have three open controls, which cannot be fixed right now:<\/p>\n
[FAIL] 1.2.5 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated)\n[FAIL] 1.2.15 Ensure that the admission control plugin PodSecurityPolicy is set (Automated)\n[FAIL] 1.2.21 Ensure that the --audit-log-path argument is set (Automated)<\/code><\/pre>\n
Reason see below in their respective sections.<\/p>\n
1.1.12<\/h3>\n
By default there is no etcd user.
\nFirst add user (from https:\/\/devopscube.com\/setup-etcd-cluster-linux\/<\/a>)<\/p>\n
groupadd -f -g 1501 etcd\nuseradd -c "etcd user" -d \/var\/lib\/etcd -s \/bin\/false -g etcd -u 1501 etcd\nchown etcd:etcd \/var\/lib\/etcd\/<\/code><\/pre>\n
1.2.5<\/h3>\n
\n
1.2.5 Follow the Kubernetes documentation and setup the TLS connection between the apiserver and kubelets. Then, edit the API server pod specification file \/etc\/kubernetes\/manifests\/kube-apiserver.yaml<\/code> on the master node and set the
\n--kubelet-certificate-authority<\/code> parameter to the path to the cert file for the  certificate authority.
\n--kubelet-certificate-authority=<ca-string><\/code><\/p>\n<\/blockquote>\n
The obvious solution to add - --kubelet-certificate-authority=\/etc\/kubernetes\/pki\/etcd\/ca.crt  to <\/code>\/etc\/kubernetes\/manifests\/kube-apiserver.yaml<\/code> does not really work. After rerunning the kube-bench job, the master pod will have theses logs:<\/p>\n
$ kubectl logs kube-bench-master-jhnp9 \nError from server: Get "https:\/\/213.167.224.157:10250\/containerLogs\/default\/kube-bench-master-jhnp9\/kube-bench": x509: cannot validate certificate for 213.167.224.157 because it doesn't contain any IP SANs<\/code><\/pre>\n
According to https:\/\/stackoverflow.com\/q\/63994701\/7311363<\/a> "you first need to make sure you got Kubelet authentication<\/a> and Kubelet authorization<\/a> enabled. After that you can follow the Kubernetes documentation and setup the TLS connection between the apiserver and kubelet<\/a>"<\/p>\n
Sounds interesting, to be revisited once all FAILs are fixed.<\/p>\n
Set it back to the default. <\/p>\n
1.2.15<\/h3>\n
deprecated soon<\/em><\/p>\n
https:\/\/kubernetes.io\/blog\/2021\/04\/06\/podsecuritypolicy-deprecation-past-present-and-future\/<\/a><\/p>\n
\n
1.2.15 Follow the documentation and create Pod Security Policy objects as per your environment.
\nThen, edit the API server pod specification file \/etc\/kubernetes\/manifests\/kube-apiserver.yaml on the master node and set the --enable-admission-plugins parameter to a value that includes PodSecurityPolicy:
\n--enable-admission-plugins=...,PodSecurityPolicy,...<\/code><\/code>
\nThen restart the API Server.<\/p>\n<\/blockquote>\n
Check ps -ef | grep kube-apiserver<\/code> and ensure PodSecurityPolicy<\/code> is included in --enable-admission-plugins<\/code>.<\/p>\n
If not add it to \/etc\/kubernetes\/manifests\/kube-apiserver.yaml<\/code><\/p>\n
spec:\n  containers:\n  - command:\n    - kube-apiserver\n    ...\n    - --enable-admission-plugins=NodeRestriction,PodSecurityPolicy<\/code><\/pre>\n
After setting that, the pod for the our CIS benchmark job cannot be started:<\/p>\n
  Type     Reason        Age                From            Message\n  ----     ------        ----               ----            -------\n  Warning  FailedCreate  27s (x3 over 57s)  job-controller  Error creating: pods "kube-bench-master-" is forbidden: PodSecurityPolicy: no providers available to validate pod request<\/code><\/pre>\n
checking for PodSecurityPolicies:<\/p>\n
$ kubectl get podsecuritypolicies.policy \nWarning: policy\/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+\nNo resources found<\/code><\/pre>\n
1.2.18<\/h3>\n
\n
1.2.18 Edit the API server pod specification file \/etc\/kubernetes\/manifests\/kube-apiserver.yaml on the master node and set the below parameter.
\n--insecure-port=0<\/code><\/p>\n<\/blockquote>\n
check<\/h4>\n
ps -ef | grep kube-apiserver<\/code><\/pre>\n
Verify that the --insecure-port<\/code> argument is set to 0 .<\/p>\n
fix<\/h4>\n
If not add it to \/etc\/kubernetes\/manifests\/kube-apiserver.yaml<\/code><\/p>\n
spec:\n  containers:\n  - command:\n    - kube-apiserver\n    ...\n    - --insecure-port=0<\/code><\/pre>\n
1.2.21 - 1.2.25 profiling and auditlog<\/h3>\n
Audit log is important to log security-relevant actions.<\/p>\n
Edit \/etc\/kubernetes\/manifests\/kube-apiserver.yaml<\/code><\/p>\n
spec:\n  containers:\n  - command:\n    - kube-apiserver\n...\n    - --audit-log-path=\/var\/log\/apiserver\/audit.log\n    - --audit-log-maxage=30\n    - --audit-log-maxbackup=10\n    - --audit-log-maxsize=100<\/code><\/pre>\n
WARNING:<\/strong>
\nIf you set audit-log-path, kubelet cannot start and will show error like:<\/p>\n
Jan 15 19:43:28 server.example.com kubelet[13157]: E0115 19:43:28.784974   13157 kubelet.go:2422] "Error getting node" err="node \\"server.example.com\\" not found"\nJan 15 19:43:28 server.example.com kubelet[13157]: E0115 19:43:28.886336   13157 kubelet.go:2422] "Error getting node" err="node \\"server.example.com\\" not found"<\/code><\/pre>\n
1.2.20, 1.3.2, 1.4.1 Disable Profiling<\/h3>\n
Disable profiling to reduce the potential attack surface. Profiling is used or the identification of specific performance bottlenecks and if we aren't active troubleshooting, we can disable it.<\/p>\n
We have to do this for:<\/p>\n