Your asynchronous backend service is a marvel of modern runtime engineering. Whether you write C# with Kestrel, Go with its netpoller, or Node.js on libuv, your service can comfortably handle 20,000 concurrent HTTP connections across four CPU cores without sweating.

Then, during a Friday afternoon traffic spike, the entire application collapses.

Your error tracking dashboard explodes with a cryptic message: System.IO.IOException: Too many open files or syscall: socket: too many open files. Incoming connections drop, your health check endpoint stops responding, and Kubernetes restarts the container pod.

When you inspect the metrics, CPU usage was hovering at a modest 25%, and memory consumption was nowhere near the container’s OOM threshold. The runtime didn’t crash because it ran out of computing power; it crashed because it ran out of File Descriptors.

To write resilient backend services on Linux, you have to understand the fundamental design principle that governs the operating system: In Linux, almost everything is a file.

The POSIX Abstraction: What Is a File Descriptor?

The Unix philosophy dictates that system resources should be exposed through a unified interface: a file path and an integer handle.

When your application opens a file on disk, Linux assigns a non-negative integer to track that open resource. That integer is a File Descriptor (FD).

Process File Descriptor Table

The kernel doesn’t just use file descriptors for files sitting on an ext4 or xfs filesystem. Linux uses file descriptors to manage:

• Disk files and directory handles
• Inbound and outbound TCP/UDP network sockets
• Unix domain sockets used for inter-process communication
• Pipes used to channel data between execution processes
• Hardware devices and pseudo-devices (/dev/null, /dev/urandom)
• Event notification handles (epoll, eventfd, timerfd).

Every single network socket your async service opens, whether it’s an incoming HTTP connection from a client, an outbound HTTP call to a microservice, a gRPC channel, or a pooled PostgreSQL connection, consumes an entry in your process’s File Descriptor table.

The Async Illusion vs Operating System Reality

Traditional multi-threaded server architectures mapped every incoming connection to an OS thread. If you wanted to handle 1,000 concurrent requests, you needed 1,000 threads. Because OS threads are expensive, consuming between 1MB and 8MB of stack memory each, your server usually runs out of RAM long before it runs out of file descriptors.

Async runtimes flipped this paradigm on its head.

By using non-blocking I/O multiplexers such as Linux’s epoll, modern runtimes decouple connection management from OS threads. A single worker thread can monitor thousands of active sockets simultaneously.

This efficiency creates a dangerous blind spot. Your async service can accept 10,000 concurrent TCP sockets with minimal memory overhead. But while async runtimes made OS threads cheap, they did not make File Descriptors infinite.

By default, many Linux distributions ship with a soft file descriptor limit of 1,024 per process.

If your high-throughput web service receives 1,000 concurrent requests, uses 20 database connections from a pool, opens three log files, and queries Redis, your application will hit 1,024 file descriptors in seconds.

The moment request 1,025 arrives, the Linux kernel refuses to allocate a new file descriptor and returns error code EMFILE (Too many open files).

Where the Limits Live (and How They Hide)

When a service throws an EMFILE or ENFILE error, diagnosing the issue requires understanding the three distinct layers of limits Linux enforces on file descriptors.

Linux File Descriptor Limits

1. Process Soft and Hard Limits (ulimit)

Linux tracks limits per process using “Soft” and “Hard” bounds:

• Soft Limit (ulimit -Sn): The actual limit enforced by the kernel for the current process. An unprivileged application can raise its soft limit to the ceiling set by the hard limit.
• Hard Limit (ulimit -Hn): The absolute ceiling set by system administrators or security policies (/etc/security/limits.conf). Only root processes can increase the hard limit.

2. The Systemd Override Trap

If you run your backend service as a systemd service on an enterprise Linux host (like RHEL or Ubuntu Server), systemd ignores /etc/security/limits.conf entirely.

Systemd services default to its own built-in limits unless explicitly overridden in the unit file:

[Unit]

Description=High Throughput Payment Service

[Service]

ExecStart=/usr/bin/dotnet /app/PaymentService.dll

# Systemd file descriptor limit configuration:

LimitNOFILE=65536

If you forget to declare LimitNOFILE in your systemd unit configuration, your service might boot with a restrictive limit of 1,024 FDs, regardless of your global host settings.

3. Container Runtime Limits (Docker & Kubernetes)

Containers inherit their initial file descriptor limits from the container engine runtime daemon (Docker/containerd). While modern Docker versions set a default soft limit of 1048576, misconfigured orchestrator specs or legacy security profiles (AppArmor/SELinux) can restrict container limits back down to 1,024.

You can inspect the exact live limits enforced on any running Linux process by reading its /proc virtual filesystem entry:

cat /proc/<PID>/limits | grep “Max open files”

Output:

Max open files 1024 4096 files

Anatomy of a Socket Leak in High-Throughput Services

Hitting file descriptor exhaustion isn’t always caused by a massive traffic surge. More often, it is caused by a socket leak, a slow, silent resource leak inside your application code.

A socket leak occurs when an application opens a network connection but fails to explicitly close it or return it to a managed connection pool.

The Unmanaged Client Anti-Pattern

Consider this common mistake in C#, Go, or Node.js services executing HTTP requests to external microservices:

// ANTI-PATTERN: Instantiating HttpClient inside a request path

public async Task<UserProfile> GetUserProfileAsync(string userId)

{

using (var client = new HttpClient())

{

var response = await client.GetAsync($”https://api.internal/users/{userId}”);

// Read response content…

}

}

Developers assume that wrapping HttpClient inside a using block disposes of the underlying network resources when the method exits.

It does not.

HttpClient implements IDisposable, but disposing it only frees local managed resources. The underlying OS socket is not closed immediately. Instead, the kernel transitions the socket into the TIME_WAIT state to ensure out-of-order network packets are safely received and discarded.

A socket can remain stuck in TIME_WAIT for up to two minutes (controlled by /proc/sys/net/ipv4/tcp_fin_timeout).

If your API executes 100 outbound requests per second using this pattern:

1. Every request opens a new file descriptor for a TCP socket.
2. Disposing of the object leaves the socket in TIME_WAIT.
3. Over two minutes, 12,000 file descriptors accumulate in the TIME_WAIT state.
4. Your application runs out of file descriptors and crashes.

The Unacknowledged CLOSE_WAIT State

Another common failure mode occurs when a remote service closes a TCP connection, but your local application fails to read the FIN packet or call close() on the socket handle.

The socket enters the CLOSE_WAIT state inside the Linux kernel network stack. The remote end is gone, but because your application holds an unclosed file handle, that file descriptor remains locked in your process table forever.

Forensic Debugging: Inspecting /proc and lsof

When a production Linux service starts throwing file descriptor errors, you don’t have to guess what is consuming your handles. You can use native Linux diagnostic tools to inspect the active process state.

Step 1: Count Active File Descriptors via /proc

In Linux, every active process exposes its open file descriptors as symbolic links inside /proc/<PID>/fd/.

To count the exact number of file descriptors currently held by a running process:

# Get process ID:

PID=$(pgrep -f “PaymentService”)

# Count open file descriptors:

ls -l /proc/${PID}/fd | wc -l

Step 2: Inspect What Your File Descriptors Point To

To list every resource your process is currently holding open:

ls -l /proc/${PID}/fd

Output snippet:

lrwx—— 1 app app 64 Sep 3 10:15 0 -> /dev/null

lrwx—— 1 app app 64 Sep 3 10:15 1 -> pipe:[482910]

lrwx—— 1 app app 64 Sep 3 10:15 2 -> pipe:[482911]

lr-x—— 1 app app 64 Sep 3 10:16 3 -> /app/appsettings.json

lrwx—— 1 app app 64 Sep 3 10:17 4 -> socket:[592014]

lrwx—— 1 app app 64 Sep 3 10:17 5 -> socket:[592015]

lrwx—— 1 app app 64 Sep 3 10:17 6 -> socket:[592016].

If you see hundreds of entries pointing to socket:[id], your process is leaking network connections.

Step 3: Deep Forensics with lsof

The List Open Files utility (lsof) provides detailed metadata about every file descriptor attached to a process, including socket IP addresses, port numbers, and TCP connection states.

To inspect network sockets for a specific process:

lsof -p ${PID} -i TCP

Output:

If lsof reveals hundreds of sockets stuck in CLOSE_WAIT targeting 10.0.3.12:http, you have instantly pinpointed the bug: your application is making calls to the HTTP service at 10.0.3.12, but your client code is failing to properly read or dispose of response streams when errors occur.

Hardening Infrastructure and Code

Fixing “Too many open files” errors requires a two-pronged approach: configuring your Linux environment to limit correctly and implementing defensive socket management in your application code.

1. Configure Production Limits

Set proper file descriptor limits across your deployment environments:

For Systemd Services (/etc/systemd/system/myservice.service):

[Service]

LimitNOFILE=65536

For Security Limits (/etc/security/limits.conf):

* soft nofile 65536

* hard nofile 65536.

For System-Wide Kernel Ceilings (/etc/sysctl.conf):

fs.file-max = 2097152

Apply with sysctl -p.

2. Enforce Socket Reuse and Connection Pooling

In application code, never instantiate unmanaged network clients on hot execution paths.

• In .NET (C#): Use IHttpClientFactory or long-lived static HttpClient instances paired with SocketsHttpHandler to enable automatic connection pooling and socket lifetime management.
• In Go: Configure http.Transport settings (MaxIdleConns and MaxIdleConnsPerHost) to ensure TCP connections are reused across HTTP requests rather than dropped.
• In Node.js: Configure custom http.Agent instances with keepAlive: true to prevent creating new sockets on every outbound request.

Respect the Underlying Operating System

Async runtimes give developers incredible power, allowing us to build high-concurrency systems that process tens of thousands of requests per second on minimal hardware.

However, abstractions stop at the operating system boundary. No matter how advanced your async execution framework is, your code runs on a Linux kernel that tracks every network socket, file handle, and pipeline as an integer inside a File Descriptor table.

When you monitor your file descriptor usage, configure explicit kernel limits, and manage connection lifecycles defensively, your services stop collapsing under heavy load — and you never have to deal with a Too many open files emergency again.

Master Linux Internals and System Engineering

Want to stop treating Linux as a black box and build a deep, intuitive understanding of process limits, networking, and system diagnostics?

Explore Hands-On: Learn Linux on Dometrain. Taught inside an interactive, zero-setup, in-browser terminal environment, this course guides you step-by-step through Linux process management, file permissions, shell tools, system diagnostics, and kernel internals with instant automated feedback on every exercise.