Under the hood - How pipe works : ls | wc

Let's explore how pipe in a Unix-like terminal.

If you're a back-end developer, Linux system admin, or have ever played with a Linux system, you've likely used pipes (the "|" character). Pipes are a powerful feature in Linux and Unix-like systems that allow you to chain commands together, passing the output of one command as the input to the next. In this article, we'll take a deep dive into how pipes actually work under the hood.

How Bash Executes Commands

To fully grasp how pipes work, we first need to understand the process of how Bash executes commands. Let's take a look at what happens under the hood when you run a simple command like ls in Bash.

When you enter ls and press Enter, Bash performs a fork() system call. This system call creates an exact duplicate of the current Bash process, resulting in two identical processes: the parent process (the original Bash process) and the child process (the duplicate).

At this point, both the parent and child processes have an identical copy of everything, including a crucial data structure called the file descriptor table. Take a mental note of this table, as it plays a critical role in the functioning of pipes. We'll come back to it later.

After the fork(), the child process undergoes a significant change. It replaces its current process image with the ls command using the execv() system call. This means that the child process, which was initially a duplicate of Bash, now becomes an instance of the ls command.

The execv() call loads the ls command's code into the child process's memory and begins executing it. Meanwhile, the parent process (the original Bash process) typically waits for the child process to complete using the wait() system call.

File Descriptors: A Closer Look

File descriptors are a fundamental concept in Linux and Unix-like operating systems. In essence, a file descriptor is a unique identifier that you obtain when you open a file.

In Linux, everything is treated as a file, including regular files, directories, sockets, and devices. As a result, every open resource has an associated file descriptor, which is typically represented as a non-negative integer.

These file descriptors serve as indices into a process's file descriptor table. For example, if a file has a file descriptor of 1, it means that in the process's file descriptor table, the entry at index 1 contains a pointer to a file description object.

A file description object is a kernel-level data structure that holds all the metadata and state information associated with an open file.

Since file description objects exist at the kernel level, they are not specific to any individual process, allowing them to be shared among multiple processes.

Standard File Descriptors

In Unix-like operating systems, each process is initialized with three standard file descriptors:

1. File descriptor 0: Standard Input (stdin)

2. File descriptor 1: Standard Output (stdout)

3. File descriptor 2: Standard Error (stderr)

These file descriptors are consistently assigned across all processes, providing a standardized interface for basic I/O operations. When a new process is created, it inherits these standard file descriptors from its parent process.

By default, these standard file descriptors are usually connected to the terminal in interactive sessions. This means that stdin reads input from the terminal, while stdout and stderr write output to the terminal. This consistent arrangement allows for uniform handling of input and output across different programs and scripts.

Lets tie everything together

Let’s revisit the concept of forks. Imagine we create two forks, resulting in two child processes. Both of these child processes have identical file descriptor tables, which means each child can access the other's stdin and stdout. Given that these two child processes are aware of each other’s stdin and stdout, is it possible to facilitate communication between them? Can we employ some form of Inter-Process Communication (IPC) for this? If so, which IPC method should we choose?

Linux offers an Inter-Process Communication (IPC) mechanism called a pipe. When we create a pipe using the pipe() system call, it returns two file descriptors:

1. A file descriptor for writing to the pipe

2. A file descriptor for reading from the pipe

In the context of our scenario, where we want to establish communication between two child processes, a pipe is an ideal solution. The key is to create the pipe before calling fork() to create the child processes.

When we create the pipe before forking, the pipe's file descriptors are inherited by both child processes during the fork() operation. This means that each child process will have a copy of the pipe's read and write descriptors in its file descriptor table.

This inheritance of the pipe's file descriptors is crucial because it allows the child processes to communicate with each other.

By carefully managing these inherited file descriptors (as we will explore in the next section) prior to replacing the process image with execv, we can create a communication channel between the child processes, allowing them to collaborate as a pipeline.

To establish communication between the two child processes, we need to manipulate their file descriptors before replacing process image with execv:

1. In the first child process (program in left side of pipe), we assign the pipe's write descriptor to file descriptor 1 (stdout).

2. In the first second process (program in right side of pipe), we assign the pipe's read descriptor to file descriptor 0 (stdin).

By doing this, we effectively connect the standard output of the second process to the standard input of the first process. This allows the processes to communicate, with data flowing from the writing process to the reading process.

However, at this point, the two child processes are still just duplicates of their parent process (usually a Bash shell). To make these processes run the desired commands, we use the execv() system call. execv() replaces the current process image with a new process image. In our case, we use it to replace each child process with the intended command. For example, the first child process could be replaced with the wc command, while the second child process could be replaced with the ls command.

Let's consider a concrete example: running ls | wc in Bash. Here's what happens under the hood:

1. Bash creates a pipe and two child processes.

2. The child process intended to run ls replaces its file descriptor 1 (stdout) with the pipe's write descriptor.

3. The child process intended to run wc replaces its file descriptor 0 (stdin) with the pipe's read descriptor.

4. Each child process uses execv() to replace itself with the respective command (ls or wc).

After these steps, the ls command writes its output to the pipe ie to its stdout, and the wc command reads its input from the pipe ie its stdin. The pipe acts as a conduit, connecting the output of ls to the input of wc, allowing them to work together as a pipeline.

This is how Bash (and other shells) use pipes to connect the output of one command to the input of another, enabling powerful command chaining and data processing workflows.

Check out this code snippet to know how above discussed idea can be implemented!!