# Pentesting Kubernetes

**The main author of this page is** [**Jorge**](https://www.linkedin.com/in/jorge-belmonte-a924b616b/)

## kubesectips

Security tips for Kubernetes

* Part 1 - Architecture
* Part 2 - Vulnerabilities
* Part 3 - Hardening

## PART 1 - ARCHITECTURE & BASICS

### What does Kubernetes do?

* Allows running container/s in a container engine.
* Schedule allows containers mission efficient.
* Keep containers alive.
* Allows container communications.
* Allows deployment techniques.
* Handle volumes of information.

### Architecture:

![](https://sickrov.github.io/media/Screenshot-68.jpg)

* **Node**: operating system with pod or pods.
  * **Pod**: Wrapper around a container or multiple containers with. A pod should only contain one application \(so usually, a pod run just 1 container\). The pod is the way kubernetes abstracts the container technology running. 
    * **Service**: Each pod has 1 service attached, which is 1 **IP address**. It's goal is to maintain the communication between pods even if one dies and a new copy is run. It can be configured as internal or external. The service also actuates as a **load balancer when 2 pods are connected** to the same service.
  * **Kubelet**: Primary node agent. The component that establishes communication between node and kubectl, and only can run pods \(through API server\). The kubelet doesn’t manage containers that were not created by Kubernetes.
  * **Kube-proxy**: is the service in charge of the communications \(services\) between the apiserver and the node. The base is an IPtables for nodes. Most experienced users could install other kube-proxies from other vendors.
  * **Sidecar container**: Sidecar containers are the containers that should run along with the main container in the pod. This sidecar pattern extends and enhances the functionality of current containers without changing them. Nowadays, We know that we use container technology to wrap all the dependencies for the application to run anywhere. A container does only one thing and does that thing very well.
* **Master process:**
  * **Api Server:** Is the way the users and the pods use to communicate with the master process. Only authenticated request should be allowed.
  * **Scheduler**: Scheduling refers to making sure that Pods are matched to Nodes so that Kubelet can run them. It has enough intelligence to decide which node has more available resources the assign the new pod to it. Note that the scheduler doesn't start new pods, it just communicate with the Kubelet process running inside the node, which will launch the new pod.
  * **Kube Controller manager**: It checks resources like replica sets or deployments to check if, for example, the correct number of pods or nodes are running. In case a pod is missing, it will communicate with the scheduler to start a new one. It controls replication, tokens, and account services to the API.
  * **etcd**: Data storage, persistent, consistent, and distributed. Is Kubernetes’s database and the key-value storage where it keeps the complete state of the clusters \(each change is logged here\). Components like the Scheduler or the Controller manager depends on this date to know which changes have occurred \(available resourced of the nodes, number of pods running...\) 
* **Cloud controller manager**: Is the specific controller for flow controls and applications, i.e: if you have clusters in AWS or OpenStack.

Note that as the might be several nodes \(running several pods\), there might also be several master processes which their access to the Api server load balanced and their etcd synchronized.

#### Volumes:

When a pod creates data that shouldn't be lost when the pod disappear it should be stored in a physical volume. **Kubernetes allow to attach a volume to a pod to persist the data**. The volume can be in the local machine or in a remote storage.

#### Other configurations:

* **ConfigMap**: You can configure **URLs** to access services. The pod will obtain data from here to know how to communicate with the rest of the services \(pods\). Note that this is not the recommended place to save credentials!
* **Secret**: This is the place to **store secret data** like passwords, API keys... encoded in B64. The pod will be able to access this data to use the required credentials.
* **Deployments**: This is where the components to be run by kubernetes are indicated. A user usually won't work directly with pods, pods are abstracted in **ReplicaSets** \(number of same pods replicated\), which are run via deployments. Note that deployments are for **stateless** applications. The minimum configuration for a deployment is the name and the image to run.
* **StatefulSet**: This component is meant specifically for applications like **databases** which needs to **access the same storage**.
* **Ingress**: This is the configuration that is use to **expose the application publicly with an URL**. Note that this can also be done using external services, but this is the correct way to expose the application.

### How pods communicate with each other.

![](https://sickrov.github.io/media/Screenshot-67.jpg)

### PKI infrastructure - Certificate Authority CA:

![](https://sickrov.github.io/media/Screenshot-66.jpg)

* CA is the trusted root for all certificates inside the cluster.
* Allows components to validate to each other.
* All cluster certificates are signed by the CA.
* ETCd has its own certificate.
* types:
  * apiserver cert.
  * kubelet cert.
  * scheduler cert.

### Minikube

**Minikube** can be used to perform some **quick tests** on kubernetes without needing to deploy a whole kubernetes environment. It will run the **master and node processes in one machine**. Minikube will use virtualbox to run the node. See [**here how to install it**](https://minikube.sigs.k8s.io/docs/start/).

```text
$ minikube start
😄  minikube v1.19.0 on Ubuntu 20.04
✨  Automatically selected the virtualbox driver. Other choices: none, ssh
💿  Downloading VM boot image ...
    > minikube-v1.19.0.iso.sha256: 65 B / 65 B [-------------] 100.00% ? p/s 0s
    > minikube-v1.19.0.iso: 244.49 MiB / 244.49 MiB  100.00% 1.78 MiB p/s 2m17.
👍  Starting control plane node minikube in cluster minikube
💾  Downloading Kubernetes v1.20.2 preload ...
    > preloaded-images-k8s-v10-v1...: 491.71 MiB / 491.71 MiB  100.00% 2.59 MiB
🔥  Creating virtualbox VM (CPUs=2, Memory=3900MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.20.2 on Docker 20.10.4 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by defaul

$ minikube status
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

---- ONCE YOU HAVE A K8 SERVICE RUNNING WITH AN EXTERNAL SERVICE -----
$ minikube service mongo-express-service
(This will open your browser to access the service exposed port)

$ minikube delete
🔥  Deleting "minikube" in virtualbox ...
💀  Removed all traces of the "minikube" cluster
```

### Kubectl Basics

**`Kubectl`** is the command line tool fro kubernetes clusters. It communicates with the Api server of the master process to perform actions in kubernetes or to ask for data.

```bash
kubectl version #Get client and server version
kubectl get pod
kubectl get services
kubectl get deployment
kubectl get replicaset
kubectl get secret
kubectl get all

#kubectl create deployment <deployment-name> --image=<docker image>
kubectl create deployment nginx-deployment --image=nginx
#Access the configuration of the deployment and modify it
#kubectl edit deployment <deployment-name>
kubectl edit deployment nginx-deployment
#Get the logs of the pod for debbugging (the output of the docker container running)
#kubectl logs <replicaset-id/pod-id>
kubectl logs nginx-deployment-84cd76b964
#kubectl describe pod <pod-id>
kubectl describe pod mongo-depl-5fd6b7d4b4-kkt9q
#kubectl exec -it <pod-id> -- bash
kubectl exec -it mongo-depl-5fd6b7d4b4-kkt9q -- bash
#kubectl describe service <service-name>
kubectl describe service mongodb-service
#kubectl delete deployment <deployment-name>
kubectl delete deployment mongo-depl
#Deploy from config file
kubectl apply -f deployment.yml
```

### YAML configuration files

Each configuration file has 3 parts: **metadata**, **specification** \(what need to be launch\), **status** \(desired state\).  
Inside the specification of the deployment configuration file you can find the template defined with a new configuration structure defining the image to run:

![](../.gitbook/assets/image%20%28458%29.png)

#### Example of Deployment + Service declared in the same configuration file \(from [here](https://gitlab.com/nanuchi/youtube-tutorial-series/-/blob/master/demo-kubernetes-components/mongo.yaml)\)

As a service usually is related to one deployment it's possible to declare both in the same configuration file \(the service declared in this config is only accessible internally\):

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongodb-deployment
  labels:
    app: mongodb
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mongodb
  template:
    metadata:
      labels:
        app: mongodb
    spec:
      containers:
      - name: mongodb
        image: mongo
        ports:
        - containerPort: 27017
        env:
        - name: MONGO_INITDB_ROOT_USERNAME
          valueFrom:
            secretKeyRef:
              name: mongodb-secret
              key: mongo-root-username
        - name: MONGO_INITDB_ROOT_PASSWORD
          valueFrom: 
            secretKeyRef:
              name: mongodb-secret
              key: mongo-root-password
---
apiVersion: v1
kind: Service
metadata:
  name: mongodb-service
spec:
  selector:
    app: mongodb
  ports:
    - protocol: TCP
      port: 27017
      targetPort: 27017
```

#### Example of external service config

This service will be accessible externally \(check the `nodePort` and `type: LoadBlancer` attributes\):

```yaml
---
apiVersion: v1
kind: Service
metadata:
  name: mongo-express-service
spec:
  selector:
    app: mongo-express
  type: LoadBalancer
  ports:
    - protocol: TCP
      port: 8081
      targetPort: 8081
      nodePort: 30000
```

{% hint style="info" %}
This is useful for testing but for production you should have only internal services and an Ingress to expose the application.
{% endhint %}

#### Example of Ingress config file

```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dashboard-ingress
  namespace: kubernetes-dashboard
spec:
  rules:
  - host: dashboard.com
    http:
      paths:
      - backend:
          serviceName: kubernetes-dashboard
          servicePort: 80

```

#### Example of secrets config file

Note how the password are encoded in B64 \(which isn't secure!\)

```yaml
apiVersion: v1
kind: Secret
metadata:
    name: mongodb-secret
type: Opaque
data:
    mongo-root-username: dXNlcm5hbWU=
    mongo-root-password: cGFzc3dvcmQ=
```

#### Example of ConfigMap

A **ConfigMap** is the configuration that is given to the pods so they know how to locate and access other services. In this case, each pod will know that the name `mongodb-service` is the address of a pod that they can communicate with \(this pod will be executing a mongodb\):

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: mongodb-configmap
data:
  database_url: mongodb-service
```

Then, inside a **deployment config** this address can be specified in the following way so it's loaded inside the env of the pod:

```yaml
[...]
spec:
  [...]
  template:
    [...]
    spec:
      containers:
      - name: mongo-express
        image: mongo-express
        ports:
        - containerPort: 8081
        env:
        - name: ME_CONFIG_MONGODB_SERVER
          valueFrom: 
            configMapKeyRef:
              name: mongodb-configmap
              key: database_url
[...]
```

### Namespaces

Kubernetes supports **multiple virtual clusters** backed by the same physical cluster. These virtual clusters are called **namespaces**. These are intended for use in environments with many users spread across multiple teams, or projects. For clusters with a few to tens of users, you should not need to create or think about namespaces at all. You only should start using namespaces to have a better control and organization of each part of the application deployed in kubernetes.

Namespaces provide a scope for names. Names of resources need to be unique within a namespace, but not across namespaces. Namespaces cannot be nested inside one another and **each** Kubernetes **resource** can only be **in** **one** **namespace**.

There are 4 namespaces by default if you are using minikube:

```text
kubectl get namespace
NAME              STATUS   AGE
default           Active   1d
kube-node-lease   Active   1d
kube-public       Active   1d
kube-system       Active   1d
```

* **kube-system**: It's not meant or the users use and you shouldn't touch it. It's for master and kubectl processes.
* **kube-public**: Publicly accessible date. Contains a configmap which contains cluster information
* **kube-node-lease**: Determines the availability of a node
* **default**: The namespace the user will use to create resources

```bash
#Create namespace
kubectl create namespace my-namespace
```

## PART 2 - VULNERABILITIES and some fixes.

### Vulnerabilities - kubernetes secrets

A Secret is an object that contains a small amount of sensitive data such as a password, a token or a key. Such information might otherwise be put in a Pod specification or in an image. Users can create Secrets and the system also creates some Secrets. The name of a Secret object must be a valid **DNS subdomain name**.

Secrets can be things like:

* API, SSH Keys.
* OAuth tokens.
* Credentials, Passwords \(plain text or b64 + encryption\).
* Information or comments.
* Database connection code, strings… .

Secret types:

| Builtin Type | Usage |
| :--- | :--- |
| Opaque | arbitrary user-defined data |
| kubernetes.io/service-account-token | service account token |
| kubernetes.io/dockercfg | serialized ~/.dockercfg file |
| kubernetes.io/dockerconfigjson | serialized ~/.docker/config.json file |
| kubernetes.io/basic-auth | credentials for basic authentication |
| kubernetes.io/ssh-auth | credentials for SSH authentication |
| kubernetes.io/tls | data for a TLS client or server |
| bootstrap.kubernetes.io/token | bootstrap token data |

**How secrets works:**

[https://kubernetes.io/docs/concepts/configuration/secret/\#using-secrets-as-files-from-a-pod](https://kubernetes.io/docs/concepts/configuration/secret/#using-secrets-as-files-from-a-pod)

![](https://sickrov.github.io/media/Screenshot-164.jpg)

Create a secret, commands:

```text
kubectl create secret generic secret_01 --from-literal user=<user>
kubectl create secret generic secret_01 --from-literal password=<password>
kubectl run pod --image=nginx -oyaml --dry-run=client
kubectl run pod --image=nginx -oyaml --dry-run=client > <podName.yaml>
```

This is the generated file:

```text
apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
  - name: mypod
    image: redis
    volumeMounts:
    - name: <secret_01>
      mountPath: "/etc/<secret_01>"
      readOnly: true
  volumes:
  - name: <secret_01>
    secret:
      secretName: <secret_01>
      items:
      - key: username
        path: my-group/my-username
```

### Using Secrets as environment variables

If you want to use a secret in an environment variable to allow the rest of the pods to reference the same secret, you could use:

In the you could add the uncomment lines:

```text
#apiVersion: v1
#kind: Pod
#metadata:
#  name: secret-env-pod
#spec:
#  containers:
#  - name: mycontainer
#    image: redis
    env:
      - name: SECRET_USERNAME
        valueFrom:
          secretKeyRef:
            name: mysecret
            key: username
#     - name: SECRET_PASSWORD
#        valueFrom:
#          secretKeyRef:
#            name: mysecret
#            key: password
#  restartPolicy: Never
```

The result is:

```text
apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
  - name: mypod
    image: redis
    env:
      - name: PASSWORD
        valueFrom:
          secretKeyRef:
            name: <secret_02>
            key: <password>   
    volumeMounts:
    - name: <secret_01>
      mountPath: "/etc/<secret_01>"
      readOnly: true
  volumes:
  - name: <secret_01>
    secret:
      secretName: <secret_01>
      items:
      - key: username
        path: my-group/my-username
```

Save and:

```text
kubectl -f <podName.yaml> delete --force
kubectl -f <podName.yaml> create
```

or:

```text
kubectl -f <podName.yaml> replace --force
```

More info: [https://kubernetes.io/docs/concepts/configuration/secret/\#using-secrets-as-environment-variables](https://kubernetes.io/docs/concepts/configuration/secret/#using-secrets-as-environment-variables)

### Discover secrets in docker:

To get the id of the container.

```text
docker ps | grep <service>
```

Inspect:

```text
docker inspect <docker_id>
```

Check env \(environment variable section\) for secrets and you will see:

* Passwords.
* Ip’s.
* Ports.
* Paths.
* Others… .

If you want to copy:

```text
docker cp <docket_id>:/etc/<secret_01> <secret_01>
```

### Discover secrets in etcd: <a id="discover-secrets-in-etcd"></a>

Remember that etcd is a consistent and highly-available key-value store used as Kubernetes backing store for all cluster data. Let’s access to the secret in etcd:

```text
cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep etcd
```

You will see certs, keys and url’s were are located in the FS. Once you get it, you would be able to connect to etcd.

```text
ETCDCTL_API=3 etcdctl --cert <path to client.crt> --key <path to client.ket> --cacert <path to CA.cert> endpoint=[<ip:port>] health
i.e:
ETCDCTL_API=3 etcdctl --cert /etc/kubernetes/pki/apiserver-etcd-client.crt --key /etc/kubernetes/pki/apiserver-etcd-client.key --cacert /etc/kubernetes/pki/etcd/etcd/ca.cert endpoint=[127.0.0.1:1234] health
```

Once you achieve establish communication you would be able to get the secrets:

```text
ETCDCTL_API=3 etcdctl --cert <path to client.crt> --key <path to client.ket> --cacert <path to CA.cert> endpoint=[<ip:port>] get <path/to/secret>
i.e:
ETCDCTL_API=3 etcdctl --cert /etc/kubernetes/pki/apiserver-etcd-client.crt --key /etc/kubernetes/pki/apiserver-etcd-client.key --cacert /etc/kubernetes/pki/etcd/etcd/ca.cert endpoint=[127.0.0.1:1234] get /registry/secrets/default/secret_02
```

### Adding encryption to the ETCD <a id="adding-encryption-to-the-etcd"></a>

So, by default all the secrets are in plain text unless you apply an encryption layer: If the identity provider is empty with the default value = {} so the secrets are in plain text. [https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/)

**Encryption types**

\| Name \| Encryption \| Strength \| Speed \| Key Length \| Other Considerations \| \|-\|-\|-\|-\|-\|-\| \| identity \| None \| N/A \| N/A \| N/A \| Resources written as-is without encryption. When set as the first provider, the resource will be decrypted as new values are written. \| \| aescbc \| AES-CBC with PKCS\#7 padding \| Strongest \| Fast \| 32-byte \| The recommended choice for encryption at rest but may be slightly slower than secretbox. \| \| secretbox \| XSalsa20 and Poly1305 \| Strong \| Faster \| 32-byte \| A newer standard and may not be considered acceptable in environments that require high levels of review. \| \| aesgcm \| AES-GCM with random nonce \| Must be rotated every 200k writes \| Fastest \| 16, 24, or 32-byte \| Is not recommended for use except when an automated key rotation scheme is implemented. \| \| kms \| Uses envelope encryption scheme: Data is encrypted by data encryption keys \(DEKs\) using AES-CBC with PKCS\#7 padding, DEKs are encrypted by key encryption keys \(KEKs\) according to configuration in Key Management Service \(KMS\) \| Strongest \| Fast \| 32-bytes \| The recommended choice for using a third party tool for key management. Simplifies key rotation, with a new DEK generated for each encryption, and KEK rotation controlled by the user. \|

The secrets will be encrypted with the above algorithms and encoded by base64.

```text
kubectl get secrets --all-namespaces -o json | kubectl replace -f -
```

### **How to encrypt the ETCD**

Create a directory in /etc/kubernetes ; in this case you will name it as etcd, so you have:

```text
/etc/kubernetes/etcd
```

You create a yaml file with the configuration.

```text
vi <configFile.yaml>
```

You can copy the content of [https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/)

```text
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
    - secrets
    providers:
    - aescbc:
        keys:
        - name: key1
          secret: <your pass in b64>
    - identity: {}
```

Generate pass in b64 \(remember to use a pass character with lenght = 16 or = 24 or = 32\) :

```text
echo -n <password> | base64
```

You can see how the encryption provider is not setting.

After that, you have to edit the file /etc/kubernetes/manifest/kube-apiserver.yaml and add the following lines into the sections: And add the following line: spec:

```text
  containers:
  - command:
    - kube-apiserver
    - --encriyption-provider-config=/etc/kubernetes/etcd/<configFile.yaml>
```

Scroll down in the volumeMounts:

```text
- mountPath: /etc/kubernetes/etcd
    name: etcd
    readOnly: true
```

Scroll down in the volumeMounts to hostPath:

```text
- hostPath:
    path: /etc/kubernetes/etcd
    type: DirectoryOrCreate
  name: etcd
```

### Get information about the secrets.

```text
kubectl get secret
kubectl get secret <secret_name> -oyaml
ETCDCTL_API=3 etcdctl get /registry/secrets/default/secret1 [...] | hexdump -C
kubectl create secret generic test-secret --from-literal='username=my-app' --from-literal='password=45tRf$we34rR'
```

With root access:

```text
# kubectl get secret
kubectl get secret test-secret -oyaml
```

Do not forget to delete de secrets and re-create them again in order to apply the encryption layer.

### Final tips:

* Try not to keep secrets in the FS, get them from other places.
* Check out [https://www.vaultproject.io/](https://www.vaultproject.io/) for add more protection to your secrets.
* [https://kubernetes.io/docs/concepts/configuration/secret/\#risks](https://kubernetes.io/docs/concepts/configuration/secret/#risks)
* [https://docs.cyberark.com/Product-Doc/OnlineHelp/AAM-DAP/11.2/en/Content/Integrations/Kubernetes\_deployApplicationsConjur-k8s-Secrets.htm](https://docs.cyberark.com/Product-Doc/OnlineHelp/AAM-DAP/11.2/en/Content/Integrations/Kubernetes_deployApplicationsConjur-k8s-Secrets.htm)

### Vulnerabilities - Container runtime sandboxes <a id="vulnerabilities---container-runtime-sandboxes"></a>

How an attack with lateral movement and privesc could be done:

![](https://sickrov.github.io/media/Screenshot-161.jpg)

Getting inside the container:

```text
kubectl get node
kubectl run pod --image=<image_name>
kubectl exec pod -it -- bash
```

Once inside the container:

```text
root@pod01:/# uname -r
```

If you want to gather information you could use:

```text
strace uname -r
ltrace uname -r
```

When the attack achieves discovering the kernel version, he could run exploiting techniques to gather information or escalate into the OS.

For secure sandboxes:

* gVisor:

[https://github.com/google/gvisor](https://github.com/google/gvisor)

* Katakontainers:

[https://katacontainers.io/](https://katacontainers.io/)

### Vulnerabilities - OS <a id="vulnerabilities---os"></a>

Is mandatory to keep in mind to define privilege and access control for container / pod:

* userID’s and groupID’s.
* Privileged or unprivileged escalation runs.
* Linux.

More info at: [https://kubernetes.io/docs/tasks/configure-pod-container/security-context/](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/)

#### userID and groupID <a id="userid-and-groupid"></a>

```text
# kubectl run pod --image=busybox --command -oyaml --dry-run=client > <podName.yaml> -- sh -c 'sleep 1h'
# vi <podName>.yaml
```

Add the uncomment lines:

```text
#apiVersion: v1
#kind: Pod
#metadata:
#  name: security-context-demo
spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 3000
    fsGroup: 2000
#  volumes:
#  - name: sec-ctx-vol
#    emptyDir: {}
#  containers:
#  - name: sec-ctx-demo
#    image: busybox
#    command: [ "sh", "-c", "sleep 1h" ]
   securityContext:
    runAsNonRoot: true
#    volumeMounts:
#    - name: sec-ctx-vol
#      mountPath: /data/demo
#    securityContext:
#      allowPrivilegeEscalation: true
```

Save and:

```text
# kubectl -f <podName>.yaml delete --force
# kubectl -f <podName>.yaml create
```

Check permissions:

```text
# kubectl exec -it <podName> -- sh
```

#### How to disable privilege escalation: <a id="how-to-disable-privilege-escalation"></a>

```text
vi <podName>.yaml
```

Set this line to false

```text
      allowPrivilegeEscalation: false
```

Save and:

```text
kubectl -f <podName>.yaml delete --force
kubectl -f <podName>.yaml create
```

#### Modify PodSecurityPolicy <a id="modify-podsecuritypolicy"></a>

Pod security policies control the security policies about how a pod has to run. More info at: [https://kubernetes.io/docs/concepts/policy/pod-security-policy/](https://kubernetes.io/docs/concepts/policy/pod-security-policy/)

Edit the kube-apiserver.yaml file:

```text
vi /etc/kubernetes/manifests/kube-apiserver.yaml
```

Inside you add in

```text
- --enable-admission-plugins=NodeRestriction,PodSecurityPolicy
```

### Vulnerabilities - mTLS <a id="vulnerabilities---mtls"></a>

Mutual authentication, two-way, pod to pod.

![](https://sickrov.github.io/media/Screenshot-165.jpg)

More info at: [https://kubernetes.io/docs/tasks/configure-pod-container/security-context/](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/)

#### Create a sidecar proxy app <a id="create-a-sidecar-proxy-app"></a>

Create your .yaml

```text
kubectl run app --image=bash --comand -oyaml --dry-run=client > <appName.yaml> -- shj -c 'ping google.com'
```

Edit your .yaml and add the uncomment lines:

```text
#apiVersion: v1
#kind: Pod
#metadata:
#  name: security-context-demo
#spec:
#  securityContext:
#    runAsUser: 1000
#    runAsGroup: 3000
#    fsGroup: 2000
#  volumes:
#  - name: sec-ctx-vol
#    emptyDir: {}
#  containers:
#  - name: sec-ctx-demo
#    image: busybox
    command: [ "sh", "-c", "apt update && apt install iptables -y && iptables -L && sleep 1h" ]
    securityContext:
      capabilities:
        add: ["NET_ADMIN"]
 #   volumeMounts:
 #   - name: sec-ctx-vol
 #     mountPath: /data/demo
 #   securityContext:
 #     allowPrivilegeEscalation: true
```

See the logs of the proxy:

```text
kubectl logs app -C proxy
```

More info at: [https://kubernetes.io/docs/tasks/configure-pod-container/security-context/](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/)

## PART 3 - HARDENING.

### CLUSTER HARDENING - RBAC

[https://kubernetes.io/docs/reference/access-authn-authz/rbac/](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) **RBAC** = Role-based access control \(RBAC\) is a method of regulating access to a computer or network resources based on the roles of individual users within your organization. RBAC authorization uses the rbac.authorization.k8s.io API group to drive authorization decisions, allowing you to dynamically configure policies through the Kubernetes API

To enable RBAC, start the API server with the –authorization-mode flag set to a comma-separated list that includes RBAC; for example:

```text
kube-apiserver --authorization-mode=Example,RBAC --other-options --more-options
```

This is enabled by default. RBAC functions:

* Restrict the access to the resources to users or ServiceAccounts.
* An RBAC Role or ClusterRole contains rules that represent a set of permissions.
* Permissions are purely additive \(there are no “deny” rules\).
* RBAC works with Roles and Bindings

The principle of Least Privilege is the meaning of only access to data or information when is necessary for a legitimate purpose.

**Types of resources:**

[https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/)

#### **CONCEPT OF NAMESPACES:**

Kubernetes supports multiple virtual clusters backed by the same physical cluster. These virtual clusters are called **namespaces**. These are intended for use in environments with many users spread across multiple teams, or projects. For clusters with a few to tens of users, you should not need to create or think about namespaces at all. Start using namespaces when you need the features they provide.

Namespaces provide a scope for names. Names of resources need to be unique within a namespace, but not across namespaces. Namespaces cannot be nested inside one another and each Kubernetes resource can only be in one namespace.

**VIEWING NAMESPACES:**

You can list the current namespaces in a cluster using:

```text
kubectl get namespace
NAME              STATUS   AGE
default           Active   1d
kube-node-lease   Active   1d
kube-public       Active   1d
kube-system       Active   1d
```

#### **SETTING THE NAMESPACE PREFERENCE**

You can permanently save the namespace for all subsequent kubectl commands in that context.

```text
kubectl config set-context --current --namespace=<insert-namespace-name-here>
```

Not All Objects are in a Namespace. Most Kubernetes resources \(e.g. pods, services, replication controllers, and others\) are in some namespaces. However, namespace resources are not themselves in a namespace. And low-level resources, such as nodes and persistentVolumes, are not in any namespace.

To see which Kubernetes resources are and aren’t in a namespace:

**IN A NAMESPACE**

```text
kubectl api-resources --namespaced=true
```

**NOT IN A NAMESPACE**

```text
kubectl api-resources --namespaced=false
```

#### Difference between Role and ClusterRole: <a id="difference-between-role-and-clusterrole"></a>

**ROLE:**

RBAC allows setting different permissions for the same role with the independence of the namespace. Roles example:

```text
/api/v1/namespaces/{namespace}/pods/{name}/log

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: defaultGreen
  name: pod-and-pod-logs-reader
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["get", "list", "watch"]
```

Other example, same Role different nameSpace and permissions:

```text
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: defaultYellow
  name: pod-and-pod-logs-reader
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["watch"]
```

**CLUSTERROLE:**

A ClusterRole can be used to grant the same permissions as a Role. Because ClusterRoles are cluster-scoped, you can also use them to grant access to:

* cluster-scoped resources \(like nodes\).
* non-resource endpoints \(like /healthz\).
* namespaced resources \(like Pods\), across all namespaces.

For example you can use a ClusterRole to allow a particular user to run:

```text
kubectl get pods --all-namespaces
```

**CLUSTERROLE EXAMPLE:**

```text
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  # "namespace" omitted since ClusterRoles are not namespaced
  name: secret-reader
rules:
- apiGroups: [""]
  #
  # at the HTTP level, the name of the resource for accessing Secret
  # objects is "secrets"
  resources: ["secrets"]
  verbs: ["get", "watch", "list"]
```

**Role and ClusterRole Binding concept:**

A role binding grants the permissions defined in a role to a user or set of users. It holds a list of subjects \(users, groups, or service accounts\), and a reference to the role being granted. A RoleBinding grants permissions within a specific namespace whereas a ClusterRoleBinding grants that access cluster-wide.

A RoleBinding may reference any Role in the same namespace. Alternatively, a RoleBinding can reference a ClusterRole and bind that ClusterRole to the namespace of the RoleBinding. If you want to bind a ClusterRole to all the namespaces in your cluster, you use a ClusterRoleBinding.

RoleBinding example:

```text
apiVersion: rbac.authorization.k8s.io/v1
# This role binding allows "jane" to read pods in the "default" namespace.
# You need to already have a Role named "pod-reader" in that namespace.
kind: RoleBinding
metadata:
  name: read-pods
  namespace: default
subjects:
# You can specify more than one "subject"
- kind: User
  name: jane # "name" is case sensitive
  apiGroup: rbac.authorization.k8s.io
roleRef:
  # "roleRef" specifies the binding to a Role / ClusterRole
  kind: Role #this must be Role or ClusterRole
  name: pod-reader # this must match the name of the Role or ClusterRole you wish to bind to
  apiGroup: rbac.authorization.k8s.io
```

ClusterRoleBinding example:

```text
apiVersion: rbac.authorization.k8s.io/v1
# This cluster role binding allows anyone in the "manager" group to read secrets in any namespace.
kind: ClusterRoleBinding
metadata:
  name: read-secrets-global
subjects:
- kind: Group
  name: manager # Name is case sensitive
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: secret-reader
  apiGroup: rbac.authorization.k8s.io
```

Permissions are additive so if you have a clusterRole with “list” and “delete” secrets you can add it with a Role with “get”. So be aware and test always your roles and permissions and specify what is ALLOWED, because everything is DENIED.

### SERVICE ACCOUNTS HARDENING

**ACCOUNTS**

[https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin/](https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin/) Users:

* Accounts for “persons” who hold a certificate integrated with the Kubernetes Identity Management of cloud providers.
* There is no Kubernetes user resource.
* A user has a Key and a Cert.

**HOW IT WORKS:**

Openssl –&gt; CSR \(CertificateSigningRequest\) –&gt; CertificateSignedRequest –&gt; Kubernetes API &lt;– CA

Be aware of the certificates because there is no way to invalidate them, you have to wait until the expiration date reaches. So what could you do in case you have to restrict the access?

* Create a new CA and reissue all certificates.
* Remove all RBAC access

**ServiceAccounts:**

* Accounts for “machines”. Is managed by the kubernetes API.
* Namespaced.
* Can interact with the Kubernetes API.
* The “Default” SA is in every namespaced used by the PODS.

### KUBERNETES API HARDENING

API requests are always assigned to a User, ServiceAccount or Anonymous request. After the request must be authenticated. [https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-authentication-authorization/](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-authentication-authorization/)

**REQUEST PROCESS:**

User or K8s ServiceAccount –&gt; Authentication –&gt; Authorization –&gt; Admission Control.

TIPS:

* Close ports.
* Avoid Anonymous access.
* NodeRestriction; No access from specific nodes to the API.
  * [https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/\#noderestriction](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#noderestriction)
  * Basically prevents kubelets from adding/removing/updating labels with a node-restriction.kubernetes.io/ prefix. This label prefix is reserved for administrators to label their Node objects for workload isolation purposes, and kubelets will not be allowed to modify labels with that prefix.
  * And also, allows kubelets to add/remove/update these labels and label prefixes.
* Ensure with labels the secure workload isolation.
* Avoid specific pods from API access.
* Avoid ApiServer exposure to the internet.
* Avoid unauthorized access RBAC.
* ApiServer port with firewall and IP whitelisting.

### KUBERNETES CLUSTER HARDENING

Upgrade it frecuently, you will receive:

* Dependencies up to date.
* Bug and security patches.

Release cycles: Each 3 months there is a new minor release [https://kubernetes.io/docs/setup/release/version-skew-policy/](https://kubernetes.io/docs/setup/release/version-skew-policy/) 1.20.3 = 1\(Major\).20\(Minor\).3\(patch\)

**BEST WAY TO UPDATE OR UPGRADE A KUBERNETES CLUSTER:**

[https://kubernetes.io/docs/tasks/administer-cluster/cluster-upgrade/](https://kubernetes.io/docs/tasks/administer-cluster/cluster-upgrade/)

* Upgrade the Master Node components following this sequence:
  * etcd \(all instances\).
  * kube-apiserver \(all control plane hosts\).
  * kube-controller-manager.
  * kube-scheduler.
  * cloud controller manager, if you use one.
* Upgrade the Worker Node components such as kube-proxy, kubelet.