Blog: Current State: 2019 Third Party Security Audit of Kubernetes
Authors (in alphabetical order): Cailyn Edwards (Shopify), Pushkar Joglekar (VMware), Rey Lejano (SUSE) and Rory McCune (DataDog) We expect the brand new Third Party Security Audit of Kubernetes will be published later this month (Oct 2022). In preparation for that, let's look at the state of findings that were made public as part of the last third party security audit of 2019 that was based on Kubernetes v1.13.4. Motivation Craig Ingram has graciously attempted over the years to keep track of the status of the findings reported in the last audit in this issue: kubernetes/kubernetes#81146. This blog post will attempt to dive deeper into this, address any gaps in tracking and become a point in time summary of the state of the findings reported from 2019. This article should also help readers gain confidence through transparent communication, of work done by the community to address these findings and bubble up any findings that need help from community contributors. Current State The status of each issue / finding here is represented in a best effort manner. Authors do not claim to be 100% accurate on the status and welcome any corrections or feedback if the current state is not reflected accurately by commenting directly on the relevant issue. # Title Issue Status 1 hostPath PersistentVolumes enable PodSecurityPolicy bypass #81110 closed, addressed by kubernetes/website#15756 and kubernetes/kubernetes#109798 2 Kubernetes does not facilitate certificate revocation #81111 duplicate of #18982 and needs a KEP 3 HTTPS connections are not authenticated #81112 Largely left as an end user exercise in setting up the right configuration 4 TOCTOU when moving PID to manager's cgroup via kubelet #81113 Requires Node access for successful exploitation. Fix needed 5 Improperly patched directory traversal in kubectl cp #76788 closed, assigned CVE-2019-11249, fixed in #80436 6 Bearer tokens are revealed in logs #81114 closed, assigned CVE-2019-11250, fixed in #81330 7 Seccomp is disabled by default #81115 closed, addressed by #101943 8 Pervasive world-accessible file permissions #81116 #112384 ( in progress) 9 Environment variables expose sensitive data #81117 closed, addressed by #84992 and #84677 10 Use of InsecureIgnoreHostKey in SSH connections #81118 This feature was removed in v1.22: #102297 11 Use of InsecureSkipVerify and other TLS weaknesses #81119 Needs a KEP 12 kubeadm performs potentially-dangerous reset operations #81120 closed, fixed by #81495, #81494, and kubernetes/website#15881 13 Overflows when using strconv.Atoi and downcasting the result #81121 closed, fixed by #89120 14 kubelet can cause an Out of Memory error with a malicious manifest #81122 closed, fixed by #76518 15 kubectl can cause an Out Of Memory error with a malicious Pod specification #81123 Fix needed 16 Improper fetching of PIDs allows incorrect cgroup movement #81124 Fix needed 17 Directory traversal of host logs running kube-apiserver and kubelet #81125 closed, fixed by #87273 18 Non-constant time password comparison #81126 closed, fixed by #81152 19 Encryption recommendations not in accordance with best practices #81127 Work in Progress 20 Adding credentials to containers by default is unsafe #81128 Closed, fixed by #89193 21 kubelet liveness probes can be used to enumerate host network #81129 Needs a KEP 22 iSCSI volume storage cleartext secrets in logs #81130 closed, fixed by #81215 23 Hard coded credential paths #81131 closed, awaiting more evidence 24 Log rotation is not atomic #81132 Fix needed 25 Arbitrary file paths without bounding #81133 Fix needed. 26 Unsafe JSON construction #81134 Partially fixed 27 kubelet crash due to improperly handled errors #81135 Closed. Fixed by #81135 28 Legacy tokens do not expire #81136 closed, fixed as part of #70679 29 CoreDNS leaks internal cluster information across namespaces #81137 Closed, resolved with CoreDNS v1.6.2. #81137 (comment) 30 Services use questionable default functions #81138 Fix needed 31 Incorrect docker daemon process name in container manager #81139 closed, fixed by #81083 32 Use standard formats everywhere #81140 Needs a KEP 33 Superficial health check provides false sense of safety #81141 closed, fixed by #81319 34 Hardcoded use of insecure gRPC transport #81142 Needs a KEP 35 Incorrect handling of Retry-After #81143 closed, fixed by #91048 36 Incorrect isKernelPid check #81144 closed, fixed by #81086 37 Kubelet supports insecure TLS ciphersuites #81145 closed but fix needed for #91444 (see this comment) Inspired outcomes Apart from fixes to the specific issues, the 2019 third party security audit also motivated security focussed enhancements
Authors (in alphabetical order): Cailyn Edwards (Shopify), Pushkar Joglekar (VMware), Rey Lejano (SUSE) and Rory McCune (DataDog)
We expect the brand new Third Party Security Audit of Kubernetes will be published later this month (Oct 2022).
In preparation for that, let's look at the state of findings that were made public as part of the last third party security audit of 2019 that was based on Kubernetes v1.13.4.
Motivation
Craig Ingram has graciously attempted over the years to keep track of the status of the findings reported in the last audit in this issue: kubernetes/kubernetes#81146. This blog post will attempt to dive deeper into this, address any gaps in tracking and become a point in time summary of the state of the findings reported from 2019.
This article should also help readers gain confidence through transparent communication, of work done by the community to address these findings and bubble up any findings that need help from community contributors.
Current State
The status of each issue / finding here is represented in a best effort manner. Authors do not claim to be 100% accurate on the status and welcome any corrections or feedback if the current state is not reflected accurately by commenting directly on the relevant issue.
| # | Title | Issue | Status |
|---|---|---|---|
| 1 | hostPath PersistentVolumes enable PodSecurityPolicy bypass | #81110 | closed, addressed by kubernetes/website#15756 and kubernetes/kubernetes#109798 |
| 2 | Kubernetes does not facilitate certificate revocation | #81111 | duplicate of #18982 and needs a KEP |
| 3 | HTTPS connections are not authenticated | #81112 | Largely left as an end user exercise in setting up the right configuration |
| 4 | TOCTOU when moving PID to manager's cgroup via kubelet | #81113 | Requires Node access for successful exploitation. Fix needed |
| 5 | Improperly patched directory traversal in kubectl cp |
#76788 | closed, assigned CVE-2019-11249, fixed in #80436 |
| 6 | Bearer tokens are revealed in logs | #81114 | closed, assigned CVE-2019-11250, fixed in #81330 |
| 7 | Seccomp is disabled by default | #81115 | closed, addressed by #101943 |
| 8 | Pervasive world-accessible file permissions | #81116 | #112384 ( in progress) |
| 9 | Environment variables expose sensitive data | #81117 | closed, addressed by #84992 and #84677 |
| 10 | Use of InsecureIgnoreHostKey in SSH connections | #81118 | This feature was removed in v1.22: #102297 |
| 11 | Use of InsecureSkipVerify and other TLS weaknesses | #81119 | Needs a KEP |
| 12 | kubeadm performs potentially-dangerous reset operations |
#81120 | closed, fixed by #81495, #81494, and kubernetes/website#15881 |
| 13 | Overflows when using strconv.Atoi and downcasting the result | #81121 | closed, fixed by #89120 |
| 14 | kubelet can cause an Out of Memory error with a malicious manifest | #81122 | closed, fixed by #76518 |
| 15 | kubectl can cause an Out Of Memory error with a malicious Pod specification |
#81123 | Fix needed |
| 16 | Improper fetching of PIDs allows incorrect cgroup movement | #81124 | Fix needed |
| 17 | Directory traversal of host logs running kube-apiserver and kubelet | #81125 | closed, fixed by #87273 |
| 18 | Non-constant time password comparison | #81126 | closed, fixed by #81152 |
| 19 | Encryption recommendations not in accordance with best practices | #81127 | Work in Progress |
| 20 | Adding credentials to containers by default is unsafe | #81128 | Closed, fixed by #89193 |
| 21 | kubelet liveness probes can be used to enumerate host network | #81129 | Needs a KEP |
| 22 | iSCSI volume storage cleartext secrets in logs | #81130 | closed, fixed by #81215 |
| 23 | Hard coded credential paths | #81131 | closed, awaiting more evidence |
| 24 | Log rotation is not atomic | #81132 | Fix needed |
| 25 | Arbitrary file paths without bounding | #81133 | Fix needed. |
| 26 | Unsafe JSON construction | #81134 | Partially fixed |
| 27 | kubelet crash due to improperly handled errors | #81135 | Closed. Fixed by #81135 |
| 28 | Legacy tokens do not expire | #81136 | closed, fixed as part of #70679 |
| 29 | CoreDNS leaks internal cluster information across namespaces | #81137 | Closed, resolved with CoreDNS v1.6.2. #81137 (comment) |
| 30 | Services use questionable default functions | #81138 | Fix needed |
| 31 | Incorrect docker daemon process name in container manager | #81139 | closed, fixed by #81083 |
| 32 | Use standard formats everywhere | #81140 | Needs a KEP |
| 33 | Superficial health check provides false sense of safety | #81141 | closed, fixed by #81319 |
| 34 | Hardcoded use of insecure gRPC transport | #81142 | Needs a KEP |
| 35 | Incorrect handling of Retry-After |
#81143 | closed, fixed by #91048 |
| 36 | Incorrect isKernelPid check | #81144 | closed, fixed by #81086 |
| 37 | Kubelet supports insecure TLS ciphersuites | #81145 | closed but fix needed for #91444 (see this comment) |
Inspired outcomes
Apart from fixes to the specific issues, the 2019 third party security audit also motivated security focussed enhancements in the next few releases of Kubernetes. One such example is Kubernetes Enhancement Proposal (KEP) 1933 Defend Against Logging Secrets via Static Analysis to prevent exposing secrets to logs with Patrick Rhomberg driving the implementation. As a result of this KEP, go-flow-levee, a taint propagation analysis tool configured to detect logging of secrets, is executed in a script as a Prow presubmit job. This KEP was introduced in v1.20.0 as an alpha feature, then graduated to beta in v1.21.0, and graduated to stable in v1.23.0. As stable, the analysis runs as a blocking presubmit test. This KEP also helped resolve the following issues from the 2019 third party security audit:
- #81114 Bearer tokens are revealed in logs
- #81117 Environment variables expose sensitive data
- #81130 iSCSI volume storage cleartext secrets in logs
Remaining Work
Many of the 37 findings identified were fixed by work from our community members over the last 3 years. However, we still have some work left to do. Here's a breakdown of remaining work with rough estimates on time commitment, complexity and benefits to the ecosystem on fixing these pending issues.
| Title | Issue | Time Commitment | Complexity | Benefit to Ecosystem |
|---|---|---|---|---|
| Kubernetes does not facilitate certificate revocation | #81111 | High | High | Medium |
| Use of InsecureSkipVerify and other TLS weaknesses | #81119 | High | High | Medium |
kubectl can cause a local Out Of Memory error with a malicious Pod specification |
#81123 | Medium | Medium | Medium |
| Improper fetching of PIDs allows incorrect cgroup movement | #81124 | Medium | Medium | Medium |
| kubelet liveness probes can be used to enumerate host network | #81129 | High | High | Medium |
| API Server supports insecure TLS ciphersuites | #81145 | Medium | Medium | Low |
| TOCTOU when moving PID to manager's cgroup via kubelet | #81113 | Medium | Medium | Low |
| Log rotation is not atomic | #81132 | Medium | Medium | Low |
| Arbitrary file paths without bounding | #81133 | Medium | Medium | Low |
| Services use questionable default functions | #81138 | Medium | Medium | Low |
| Use standard formats everywhere | #81140 | High | High | Very Low |
| Hardcoded use of insecure gRPC transport | #81142 | High | High | Very Low |
To get started on fixing any of these findings that need help, please consider getting involved in Kubernetes SIG Security by joining our bi-weekly meetings or hanging out with us on our Slack Channel.
