Source: Securelist – Kaspersky

Headline: Host-based logs, container-based threats: How to tell where an attack began

The risks associated with containerized environments

Although containers provide an isolated runtime environment for applications, this isolation is often overestimated. While containers encapsulate dependencies and ensure consistency, the fact that they share the host system’s kernel introduces security risks.

Based on our experience providing Compromise Assessment, SOC Consulting, and Incident Response services to our customers, we have repeatedly seen issues related to a lack of container visibility. Many organizations focus on monitoring containerized environments for operational health rather than security threats. Some lack the expertise to properly configure logging, while others rely on technology stacks that don’t support effective visibility of running containers.

Environments that suffer from such visibility issues are often challenging for threat hunters and incident responders because it can be difficult to clearly distinguish between processes running inside a container and those executed on the host itself. This ambiguity makes it difficult to determine the true origin of an attack and whether it started in a compromised container or directly on the host.

The aim of this blog post is to explain how to restore the execution chain inside a running container using only host-based execution logs, helping threat hunters and incident responders determine the root cause of a compromise.

How containers are created and operate

To effectively investigate security incidents and hunt for threats in containerized environments, it’s essential to understand how containers are created and how they operate. Unlike virtual machines, which run as separate operating systems, containers are isolated user-space environments that share the host OS kernel. They rely on namespaces, control groups (cgroups), union filesystems, Linux capabilities, and other Linux features for resource management and isolation.

Because of this architecture, every process inside a container technically runs on the host, but within a separate namespace. Threat hunters and incident responders typically rely on host-based execution logs to gain a retrospective view of executed processes and command-line arguments. This allows them to analyze networks that lack dedicated containerization environment monitoring solutions. However, some logging configurations may lack critical attributes such as namespaces, cgroups, or specific syscalls. In such cases, rather than relying solely on missing log attributes, we can bridge this visibility gap by understanding the process execution chain of a running container from a host perspective.

End users interact with command-line utilities, such as Docker CLI, kubectl and others, to create and manage their containers. On the backend, these utilities communicate with an engine that facilitates communication with a high-level container runtime, most commonly containerd or CRI-O. These high-level container runtimes leverage low-level container runtimes like runc (the most common) to do the heavy lifting of interacting with the Linux OS kernel. This interaction allocates cgroups, namespaces, and other Linux capabilities for creating and killing containers based on a bundle provided by the high-level runtime. The high-level runtime is, in its turn, based on user-provided arguments. The bundle is a self-contained directory that defines the configuration of a container according to the Open Container Initiative (OCI) Runtime Specification. It mainly consists of:

1. A rootfs directory that serves as the root filesystem for the container. It is created by extracting and combining the layers from a container image, typically using a union filesystem like OverlayFS.
2. A config.json file describing an OCI runtime configuration that specifies the necessary process, mounts, and other configurations necessary for creating the container.

It’s important to note which mode runc has been executed in, since it supports two modes: foreground mode and detached mode. The resulting process tree may vary depending on the chosen mode. In foreground mode, a long-running runc process remains in the foreground as a parent process for the container process, primarily to handle the stdio so the end user can interact with the running container.

In detached mode, however, there will be no long-running runc process. After creating the container, runc exits, leaving the caller process to take care of the stdio. In most cases, this is containerd or CRI-O. As we can see in the screenshot below, when we execute a detached container using runc, the runc process will create it and immediately exit. Hence, the parent process of the container is the host’s PID 1 (systemd process).

However, if we create a detached container using Docker CLI, for example, we’ll notice that the parent of the container process is a shim process, not PID 1!

In modern architectures, communication between high- and low-level container runtimes is proxied through a shim process. This allows containers to run independently of the high-level container runtime, ensuring the sustainability of the running container even if the high-level container runtime crashes or restarts. The shim process also manages the stdio of the container process so users can later attach to running containers via commands like docker exec -it , for example. The shim process can also redirect stdout and stderr to log files that users can later inspect either directly from the filesystem or via commands like kubectl logs -c .

When a detached container is created using Docker CLI, the high-level container runtime, for example, containerd, executes a shim process that calls runc as a low-level container runtime for the sole purpose of creating the container in detached mode. After that, runc immediately exits. To avoid orphan processes or reparenting to the PID 1, as in the case when we executed runc ourselves, the shim process explicitly sets itself as a subreaper to adopt the container processes after runc exits. A Linux subreaper process is a designated parent that takes care of orphaned child processes in its chain (instead of init), allowing it to manage and clean up its entire process tree.

This is implemented in the latest V2 shim and is the default in the modern containerd implementations.

When we check the help message of the containerd-shim-runc-v2 process, for example, we notice that it accepts the container ID as a command-line argument, and calls it the id of the task.

We can confirm this by checking the command-line arguments of the running containerd-shim-runc-v2 processes and comparing them with the running containers.

So far, we’ve successfully identified container processes from the host’s perspective. In modern architectures, one of the following processes will typically be seen as a predecessor process for the containerized processes:

• A shim process, in the case of detached mode; or
• A runc process, in the case of foreground (interactive) mode.

We can also use the command-line arguments of the shim process to determine which container the process belongs to.

Although tracking the child processes of the shim process can sometimes lead to easy wins, it is often not as easy as it sounds, especially when there are a lot of subprocesses between the shim process and the malicious process. In this case, we can take a bottom-to-top approach, pivoting from the malicious process, tracking its parents all the way up to the shim process to confirm that it was executed inside a running container. It then becomes a matter of choosing the process whose behavior we may need to check for malicious or suspicious activities.

Since containers typically run with minimal dependencies, attackers often rely on shell access to either execute commands directly, or install missing dependencies for their malware. This makes container shells a critical focus for detection. But how exactly do these shells behave? Let’s take a closer look at one of the key shell processes in containerized environments.

How do BusyBox and Alpine execute commands?

In this post, we focus on the behavior of BusyBox-based containers. We also included Alpine-based containers as an example of an image base that relies on BusyBox to implement many core Linux utilities, helping to keep the image lightweight. For the sake of demonstration, Alpine images that depend on other utilities are outside the scope of this post.

BusyBox provides minimalist replacements for many commonly used UNIX utilities, combining them into one small executable. This allows for the creation of lightweight containers with significantly reduced image sizes. But how does the BusyBox executable actually work?

BusyBox has its own implementation of system utilities, known as applets. Each applet is written in C and stored in the busybox/coreutils/ directory as part of the source code. For example, the UNIX cat utility has a custom implementation named cat.c. At runtime, BusyBox creates an applet table that maps applet names to their corresponding functions. This table is used to determine which applet to execute based on the command-line argument provided. This mechanism is defined in the appletlib.c file.

When an executed command calls an installed utility that is not a default applet, BusyBox relies on the PATH environment variable to determine the utility’s location. Once the path is identified, BusyBox spawns the utility as a child process of the BusyBox process itself. This dynamic execution mechanism is critical to understanding how command execution works within a BusyBox-based container.

Now that we have a clear understanding of how the BusyBox binary operates, let’s explore how it functions when running inside a container. What happens, for example, when you execute the sh command inside such containers?

In both BusyBox and Alpine containers, executing the sh command to access the shell doesn’t actually invoke a standalone binary called sh. Instead, the BusyBox binary itself is executed. In BusyBox containers we can verify that /bin/sh is replaced by BusyBox by comparing the inodes of /bin/sh and /bin/busybox using ls -li and confirm that both have the same inode number. We can also print their MD5 hash to see that they are the same, and by executing /bin/sh --help, we’ll see that the banner of BusyBox is the one that’s printed.

On the other hand, in the Alpine containers, /bin/sh is a symbolic link to /bin/busybox. This means that when you run the sh command, it actually executes the BusyBox executable referred to by the symbolic link. This can be confirmed by executing readlink -f /bin/sh and observing the output.

Hence, inside BusyBox- or Alpine-based containers, all shell commands are either executed directly by the BusyBox process or are launched as child processes under the BusyBox process. These processes run within isolated namespaces on the host operating system, providing the necessary containerization while still utilizing the shared kernel of the host.

From a threat hunting perspective, having a non-standard shell process for the host OS, like BusyBox in this case, should prompt further investigation. Why would a BusyBox shell process be running on a Debian or a RedHat OS? Combining this conclusion with the previous one allows us to confirm that the shell was executed inside a container when runc or shim is observed as the predecessor process to the BusyBox process. This knowledge can be applied not only to the BusyBox process but also to any other process executed inside a running container. This knowledge is crucial for effectively determining the origin of suspicious behavior while hunting for threats using the host execution logs.

Some security tools, such as Kaspersky Container Security, are designed to monitor container activity and detect suspicious behavior. Others, such as Auditd, provide enriched logging at the kernel level based on preconfigured rules that capture system calls, file access, and user activity. However, these rules are often not optimized for containerized environments, further complicating the distinction between host and container activity.

Investigation value

While investigating execution logs, threat hunters and incident responders might overlook some activities on Linux machines, thinking they are part of normal operations. However, the same activities performed inside a running container should raise suspicion. For example, installing utilities such as Docker CLI may be normal on the host, but not inside a container. Recently, in a Compromise Assessment project, we discovered a crypto mining campaign in which the threat actor installed Docker CLI inside a running container in order to easily communicate with dockerd APIs.

In this example, we detected the installation of Docker CLI inside a container by tracing the process chain. We then determined the origin of the executed command and confirmed the container in which the command was executed by checking the command-line argument of the shim process.

During another investigation, we detected an interesting event where the process name was systemd while the process executable path was /.redtail. To identify the origin of this process, we followed the same procedure of tracking the parent processes.

Another interesting fact we can leverage is that a Docker container is always created by a runc process as the low-level container runtime. The runc help message reveals the command-line arguments used to create, run or start a container.

Monitoring these events helps threat hunters and incident responders identify the ID of the subject container and detect any abnormal entrypoints. A container’s entrypoint is its main process and it will be the process spawned by runc. The screenshot below shows an example of the creation of a malicious container detected by hunting for entrypoints with suspicious command-line arguments. In this case, the command line contains a malicious base64-encoded command.

Conclusion

Containerized environments are now part of most organizations’ networks because of the deployment and dependency encapsulation feasibility they provide. However, they are usually overlooked by security teams and decision makers because of a common misunderstanding about container isolation. This results in undesirable situations when these containers are compromised, and the security team is not fully equipped with the knowledge or tools to help during response activities, or even to monitor or detect in the first place.

The approach discussed in this post is one of the procedures that we typically follow in our Compromise Assessment and Incident Response services when we need to hunt for threats in historical host execution logs with container visibility issues. However, in order to detect container-based threats in time, it is crucial to protect your systems with a solid containerization monitoring solution, such as Kaspersky Container Security.