![]() |
Tiexin GuoSenior DevOps Consultant, Amazon Web Services Author | 4th Coffee |
Over the past decade, the rise of microservice architectures, the DevOps culture, and popular tools like Docker and Kubernetes have made deploying applications as containers standard practice in the industry, making container security a top priority.
In this article, we will take a deep look at container security: what are the common container vulnerabilities, how to mitigate risks by container security scanning, and what popular container security tools and solutions are available to integrate into the SDLC in a DevSecOps way. Without further ado, let's get started.
Containers are created using container images, which act as templates. If the images have vulnerabilities, the containers will introduce those vulnerabilities into the production environment.
Container images have a collection of files in them: source code, but also configurations, binaries, and other dependencies, all of which could introduce vulnerabilities. So, what are the common container vulnerabilities? Here are a few:
Fortunately, most container vulnerabilities can be mitigated relatively easily via container security scanning, a process that analyzes images and containers for security issues. Continuous container security scanning, or continuous container security, if you will, is a critical part of DevSecOps.
How does container security scanning work? Typically, there are two different methods:
Next, let's examine these container vulnerabilities—secrets, misconfiguration, vulnerabilities from code, and dependencies—and how different approaches and tools can help mitigate these risks.
Misconfigurations and hard-coded secrets reduce security by making the application and potentially the entire infrastructure vulnerable to easy exploits. Attackers can gain access to critical systems and may even escape from within containers to the host machine.
So, at a bare minimum, we need to check for misconfigurations and ensure no hard-coded secrets are present in Docker images.
💡
Hard-coded secrets are fundamentally different from other vulnerabilities: any exposed secret can lead to a security breach, regardless of whether the container is unused or the image is old. A Docker image is constructed from stacked layers that form its current state. This layered structure is prone to leaks because a layer can hide secrets from previous layers, making them invisible in the final state but still present within the image. So using a scanner that can thoroughly examine each layer is crucial for identifying and mitigating these hidden secrets.
Now, let's have a look at a few popular tools that can detect hard-coded secrets and misconfiguration.
ggshield
(requires a GitGuardian account) is a CLI to find more than 350+ types of hardcoded secrets (specific and generic ones) and 70+ Infrastructure-as-Code security misconfigurations.
Simply install it with your package manager (for example, for macOS, you can install ggshield
using Homebrew: brew install gitguardian/tap/ggshield
), and then the ggshield secret scan docker
command is ready to be used to scan local docker images for secrets present in the image's creation process (Dockerfile and build arguments), and in the image's layers' filesystem.
Here's a demo of how it works:
ggshield
provides many more functionalities than simply scanning for secrets. For more details, refer to the cheat sheet below (click to get the PDF) or the official documentation here
Another option is SecretScanner
from Deepfence. It is a standalone, open-source tool that retrieves and searches container and host filesystems, matching the contents against approximately 140 secret types (specific only).
Using SecretScanner is simple: pull the docker image for it by running docker pull quay.io/deepfenceio/deepfence_secret_scanner_ce:2.2.0
, and we can simply do a docker run
to scan container images.
See it in action below:
Yet another option is trivy
, which is a versatile security scanner that can scan for both secrets (but only specific ones) and misconfiguration in container images, file systems, remote git repositories, and more (will touch on this a bit more in the next section). For now, let's focus on its capabilities of secrets and misconfiguration detection.
It can be installed using your package manager (for example, for macOS, you can also use Homebrew to install it: brew install trivy
), and you can use it to scan both Docker images and local file systems.
For example, to scan secrets in an image, run trivy image --scanners secret IMAGE_NAME
. Below is an example with a clean result:
$ trivy image --scanners secret ironcore864/go-hello-http:0.0.1
2024-06-23T20:02:49+08:00 INFO Secret scanning is enabled
2024-06-23T20:02:49+08:00 INFO If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2024-06-23T20:02:49+08:00 INFO Please see also https://aquasecurity.github.io/trivy/v0.52/docs/scanner/secret/#recommendation for faster secret detection
Trivy also provides built-in checks to detect misconfigurations in popular Infrastructure as Code files, such as Docker, Kubernetes, Terraform, CloudFormation, and more. You can even define your own custom checks.
Here's an example to scan a Dockerfile locally: Given the Dockerfile below (building a simple Golang app with Alpine as the base image):
FROM alpine
WORKDIR $GOPATH/src/github.com/ironcore864/go-hello-http
COPY . .
RUN apk add git
RUN go get ./... && CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o hello
CMD ["./hello"]
EXPOSE 8080/tcp
Run the following command to check for misconfigurations (as we can see, 4 are found):
$ trivy fs --scanners misconfig .
2024-06-23T20:05:56+08:00 INFO Misconfiguration scanning is enabled
2024-06-23T20:05:56+08:00 INFO Detected config files num=1
...
Failures: 4 (UNKNOWN: 0, LOW: 1, MEDIUM: 1, HIGH: 2, CRITICAL: 0)
MEDIUM: Specify a tag in the 'FROM' statement for image 'alpine'
When using a 'FROM' statement you should use a specific tag to avoid uncontrolled behavior when the image is updated.
See https://avd.aquasec.com/misconfig/ds001
...
HIGH: Specify at least 1 USER command in Dockerfile with non-root user as argument
Running containers with 'root' user can lead to a container escape situation. It is a best practice to run containers as non-root users, which can be done by adding a 'USER' statement to the Dockerfile.
See https://avd.aquasec.com/misconfig/ds002
...
HIGH: '--no-cache' is missed: apk add git
You should use 'apk add' with '--no-cache' to clean package cached data and reduce image size.
See https://avd.aquasec.com/misconfig/ds025
...
LOW: Add HEALTHCHECK instruction in your Dockerfile
You should add HEALTHCHECK instruction in your docker container images to perform the health check on running containers.
See https://avd.aquasec.com/misconfig/ds026
The previous section ensures that there are no hard-coded secrets and that the configurations are spot-on. Next, let's move on to the other files in the Docker image: OS packages, language-specific packages, dependencies, etc.
The aforementioned comprehensive scanner trivy
is one of the most popular open-source security scanners. It can also detect known vulnerabilities according to the versions of installed packages. This means that everything inside the image, whether called by your code or not, is scanned.
For example, let's write a simple Python Flask application and build a Docker image out of it, then scan it with trivy
.
A very simple Flask hello world app is created:
from flask import Flask
app = Flask(__name__)
@app.route("/")
def hello_world():
return "<p>Hello, World!</p>"
Then, we add some dependencies in requirements.txt
:
requests==2.19.1
cryptography==3.3.2
flask
As you noticed, some packages are added, but not even used by our Flask app code, simply as a test.
Then we create a Dockerfile to build it:
FROM python:3.9-slim-buster
WORKDIR /app
COPY ./requirements.txt /app
RUN pip install -r requirements.txt
COPY . .
EXPOSE 5000
ENV FLASK_APP=app.py
CMD ["flask", "run", "--host", "0.0.0.0"]
If you wish, you can clone the repository here to get the whole code above.
We build and tag it: docker build . -t ironcore864/python-insecure-app:0.0.1
, then scan it with trivy
:
$ trivy image --scanners vuln ironcore864/python-insecure-app:0.0.1
2024-06-23T20:50:02+08:00 INFO Vulnerability scanning is enabled
2024-06-23T20:50:03+08:00 INFO [python] License acquired from METADATA classifiers may be subject to additional terms name="pip" version="23.0.1"
2024-06-23T20:50:03+08:00 INFO [python] License acquired from METADATA classifiers may be subject to additional terms name="setuptools" version="58.1.0"
2024-06-23T20:50:03+08:00 INFO [python] License acquired from METADATA classifiers may be subject to additional terms name="MarkupSafe" version="2.1.5"
2024-06-23T20:50:03+08:00 INFO [python] License acquired from METADATA classifiers may be subject to additional terms name="certifi" version="2024.6.2"
2024-06-23T20:50:03+08:00 INFO [python] License acquired from METADATA classifiers may be subject to additional terms name="cffi" version="1.16.0"
2024-06-23T20:50:03+08:00 INFO [python] License acquired from METADATA classifiers may be subject to additional terms name="chardet" version="3.0.4"
2024-06-23T20:50:03+08:00 INFO [python] License acquired from METADATA classifiers may be subject to additional terms name="click" version="8.1.7"
2024-06-23T20:50:03+08:00 INFO [python] License acquired from METADATA classifiers may be subject to additional terms name="cryptography" version="3.3.2"
2024-06-23T20:50:03+08:00 INFO [python] License acquired from METADATA classifiers may be subject to additional terms name="idna" version="2.7"
2024-06-23T20:50:03+08:00 INFO [python] License acquired from METADATA classifiers may be subject to additional terms name="pycparser" version="2.22"
2024-06-23T20:50:03+08:00 INFO [python] License acquired from METADATA classifiers may be subject to additional terms name="requests" version="2.19.1"
2024-06-23T20:50:03+08:00 INFO [python] License acquired from METADATA classifiers may be subject to additional terms name="six" version="1.16.0"
2024-06-23T20:50:03+08:00 INFO [python] License acquired from METADATA classifiers may be subject to additional terms name="urllib3" version="1.23"
2024-06-23T20:50:03+08:00 INFO Detected OS family="debian" version="10.13"
2024-06-23T20:50:03+08:00 INFO [debian] Detecting vulnerabilities... os_version="10" pkg_num=93
2024-06-23T20:50:03+08:00 INFO Number of language-specific files num=1
2024-06-23T20:50:03+08:00 INFO [python-pkg] Detecting vulnerabilities...
ironcore864/python-insecure-app:0.0.1 (debian 10.13)
Total: 208 (UNKNOWN: 3, LOW: 92, MEDIUM: 61, HIGH: 50, CRITICAL: 2)
┌────────────────────────┬─────────────────────┬──────────┬──────────────┬─────────────────────────┬────────────────────────┬──────────────────────────────────────────────────────────────┐
│ Library │ Vulnerability │ Severity │ Status │ Installed Version │ Fixed Version │ Title │
├────────────────────────┼─────────────────────┼──────────┼──────────────┼─────────────────────────┼────────────────────────┼──────────────────────────────────────────────────────────────┤
│ apt │ CVE-2011-3374 │ LOW │ affected │ 1.8.2.3 │ │ It was found that apt-key in apt, all versions, do not │
│ │ │ │ │ │ │ correctly... │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2011-3374 │
├────────────────────────┼─────────────────────┤ │ ├─────────────────────────┼────────────────────────┼──────────────────────────────────────────────────────────────┤
│ bash │ CVE-2019-18276 │ │ │ 5.0-4 │ │ bash: when effective UID is not equal to its real UID the... │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2019-18276 │
│ ├─────────────────────┤ │ │ ├────────────────────────┼──────────────────────────────────────────────────────────────┤
│ │ TEMP-0841856-B18BAF │ │ │ │ │ [Privilege escalation possible to other user than root] │
│ │ │ │ │ │ │ https://security-tracker.debian.org/tracker/TEMP-0841856-B1- │
│ │ │ │ │ │ │ 8BAF │
├────────────────────────┼─────────────────────┼──────────┼──────────────┼─────────────────────────┼───────
... (truncated)
As you can see, multiple vulnerabilities are found, some of which are from the OS packages and some from the dependencies we added to it. Even though we do not call those packages directly with our code, since image scanning works by scanning the OS packages and dependencies—what's in the image—it finds them all.
Besides trivy
, there are a few other tools worth mentioning:
clair
: an open-source application for parsing image contents and reporting vulnerabilities affecting the contents, done via static analysis, not at runtime.grype
: a vulnerability scanner for container images and filesystems. It works with syft
, the powerful SBOM (software bill of materials) tool for container images and filesystems. For more information about SBOMs:Why you need an SBOM (Software Bill Of Materials)
SBOMs are security analysis artifacts becoming required by more companies due to internal policies and government regulation. If you sell or buy software, you should know the what, why, and how of the SBOM.
GitGuardian Blog – Code Security for the DevOps generationGreg Bulmash
Next, let's move on to a slightly different approach to vulnerability scanning: Software Composition Analysis (SCA), which automatically scans and detects vulnerabilities in components and third-party libraries.
While this might sound similar to Docker image scanning, there are key differences. SCA focuses on identifying vulnerabilities in the software dependencies and libraries used within an application, ensuring that all components are secure. In contrast, as we have seen, Docker image scanning examines the entire container image, including the operating system, application code, and configurations.
Let's have a look at one of the SCA tools snyk
to get a hands-on feeling about SCA. For macOS users, you can install snyk
by running:
brew tap snyk/tap
brew install snyk
If we scan the same app from the previous section:
git clone https://github.com/IronCore864/python-insecure-app.git
cd python-insecure-app
snyk test .
We should get results similar to:
$ snyk test .
Testing ....
Tested 15 dependencies for known issues, found 4 issues, 4 vulnerable paths.
Issues to fix by upgrading dependencies:
Upgrade [email protected] to [email protected] to fix
✗ Uncontrolled Resource Consumption [Low Severity][https://security.snyk.io/vuln/SNYK-PYTHON-CRYPTOGRAPHY-6913422] in [email protected]
introduced by [email protected]
Pin [email protected] to [email protected] to fix
✗ Cross-site Scripting (XSS) [Medium Severity][https://security.snyk.io/vuln/SNYK-PYTHON-JINJA2-6809379] in [email protected]
introduced by [email protected] > [email protected]
Pin [email protected] to [email protected] to fix
✗ Improper Removal of Sensitive Information Before Storage or Transfer (new) [Medium Severity][https://security.snyk.io/vuln/SNYK-PYTHON-URLLIB3-7267250] in [email protected]
introduced by [email protected] > [email protected]
Pin [email protected] to [email protected] to fix
✗ Remote Code Execution (RCE) [High Severity][https://security.snyk.io/vuln/SNYK-PYTHON-WERKZEUG-6808933] in [email protected]
introduced by [email protected] > [email protected]
...
As we can see, the issues detected by the Docker image scanning in the previous section were also detected by image scanning. Note, however, that image scanning found more issues.
Both Docker image scanning and SCA work by referring to known vulnerability databases, which explains the similar results.
However, whereas Docker image scanning scans everything present inside an image, including OS packages, SCA is only concerned about the dependencies declared in our project (in our case, defined in the requirements.txt
), which is a more specific scope.
At the end of the day, vulnerable dependencies (in our case, requests
and cryptography
) are reported by both Docker image scanning and SCA, but there are key differences:
Combining the above pros and cons of both Docker image scanning and SCA, we can see that they provide complementary values at different stages. They both have their place in your automated DevSecOps workflow.
9 Things to Consider When Choosing an SCA Tool
Software composition analysis is an essential part of application security. Here are the important factors to consider when selecting an SCA scanner to be sure it is well-suited to your needs.
GitGuardian Blog – Code Security for the DevOps generationGuest Expert
Besides snyk
, there is another open-source project osv-scanner
that's worth a look at: It's an SCA vulnerability scanner written in Go, which uses the data provided by osv.dev
For more information on osv-scanner
(and open-source security in general), read this blog:
Open-Source Software Security
Open-source software security is crucial in today’s cloud-native world. Learn about vulnerabilities, dependencies, and tools to improve security in this in-depth blog post.
GitGuardian Blog – Code Security for the DevOps generationGuest Expert
So far, we've covered both Docker image scanning and SCA, which happened at the code/image level, with which we can discover the known vulnerabilities and fix them before our apps get deployed. Next, let's cover another type of container security scanning: container runtime security.
Container runtime security, as the name suggests, happens at the runtime level: it takes proactive measures and controls to protect containerized applications during the runtime phase, detecting unexpected behaviour, configuration changes, and attacks in near real-time.
This sounds theoretical, so let's take a look at one concrete example with Falco, an open-source tool designed to detect and alert on abnormal behaviour and potential security threats in real-time.
To quickly try Falco out, prepare a K8s cluster and install it with helm:
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update
kubectl create namespace falco
helm install falco -n falco --set driver.kind=ebpf --set tty=true falcosecurity/falco \
--set falcosidekick.enabled=true \
--set falcosidekick.config.slack.webhookurl=$(base64 --decode <<< "aHR0cHM6Ly9ob29rcy5zbGFjay5jb20vc2VydmljZXMvVDA0QUhTRktMTTgvQjA1SzA3NkgyNlMvV2ZHRGQ5MFFDcENwNnFzNmFKNkV0dEg4") \
--set falcosidekick.config.slack.minimumpriority=notice \
--set falcosidekick.config.customfields="user:changeme"
Note: Be sure to change the last parameter from changeme to your name.
For a more detailed installation guide, see the official documentation here.
Once Falco is up and running, we can simulate suspicious actions to see if the so-called "runtime security" platform can capture them. For As a first example, let's start a container and get a shell. In the real world, an attacker with shell access means the container is already compromised, as the attacker could try all kinds of things.
Start a container by running: kubectl run alpine --image alpine -- sh -c "sleep infinity"
, then execute the uptimecommand
inside it:
kubectl exec -it alpine – sh -c "uptime"
(well, this is a harmless command, but you get the gist).
If we check Falco's output, we should get an alert:
22:09:57.827858816: Notice A shell was spawned in a container with an attached
...
cmdline=sh -c uptime pid=6461 terminal=34816
... (truncated)
Falco's base rules alert you if someone runs an interactive shell into a running container. For more standard rules, see the official documentation. Also, you can write and customize Falco rules by yourself to secure your environment.
In this blog, we've seen that there are several types of container vulnerabilities: hard-coded secrets, misconfigurations, vulnerabilities in the application code, and vulnerabilities in OS packages and dependencies.
Luckily, most of them can be detected by tools fairly easily:
ggshield
, SecretScanner
to scan for secretstrivy
to scan for misconfigurationstrivy
, clair
and grype
to scan Docker images for known CVEssnyk
and osv-scanner
to analyze software compositionBesides scanning code and images, we also can enhance our production environment by using container runtime security platforms such as Falco, which detects unexpected behaviour and attacks in real-time with predefined and customizable rules.
As a closing thought, it's worth pointing out that container security scanning has limitations: It's not great for detecting unknown vulnerabilities. If some vulnerabilities are not yet publicly disclosed, or if a new type of attacking behavior just emerged and it's not in the runtime security platform's rule list, they can't be detected.
So, although container security scanning is necessary to secure production environments, it's important to remember that it is one layer of security and not an end-game solution to mitigate all types of threats.
As a follow-up reading, you can refer to the Docker security cheat sheet, to help you understand how to run containers with the right security guardrails from the onset:
Docker Security Best Practices: Cheat Sheet
Containers are no security devices. That’s why we’ve curated a set of easily actionable recommendations to improve your Docker containers security. Check out the one-page cheat sheet.
GitGuardian Blog – Code Security for the DevOps generationThomas Segura
I hope you enjoyed this article. See you in the next piece!
*** This is a Security Bloggers Network syndicated blog from GitGuardian Blog - Code Security for the DevOps generation authored by Guest Expert. Read the original post at: https://blog.gitguardian.com/container-security-scanning-vulnerabilities-risks-and-tooling/