# My Website Went Down This Morning. Here's Exactly What Happened — And How I Fixed It.

> A real incident: cascading misconfigurations — a wrong Fail2ban regex, no swap space, and Nginx blocking without banning — brought down a server. Full breakdown and the 10-minute fix.

- Author: Rubén Rangel (https://rubenrangel.org)
- Published: 2026-03-17
- Updated: 2026-03-17
- Language: en
- Canonical: https://rubenrangel.org/en/blog/website-went-down-heres-exactly-what-happened

At 4:40 AM, my server stopped responding. No warning, no alert, no obvious reason. Just a website that was down and a Cloudflare notification about a domain expiring in 25 days — which had nothing to do with it.

This is a real incident that happened today. Here's the full breakdown.

---

## Step 1: Stop Guessing, Start Reading Logs

The first instinct when a server goes down is to restart everything and hope it works. That's the wrong move. If you don't know why it crashed, it will crash again.

```bash
sudo journalctl -xe --since "10 hours ago"
```

The logs showed the server had rebooted at 14:25 UTC. Not a service crash — a full system reboot. That narrowed it down immediately.

```bash
sudo tail -n 100 /var/log/nginx/error.log
```

That's where the real story was.

---

## Step 2: The Attack Nobody Noticed

Between 00:11 and 00:11:50 AM — just 40 seconds — a single IP from AWS Seoul (`3.38.148.216`) sent over 80 requests scanning for exposed `.env` files:

```
GET /FRONTEND/.env
GET /.env.production
GET /admin/.env.local
GET /laravel/.env.local
... (80+ variations)
```

Nginx blocked every single one with `access forbidden by rule`. The blocks worked. But here's the problem: **blocking a request with a 403 is not the same as banning the IP.**

- A 403 means your server still received the request, processed it, and responded.
- A proper IP ban means the connection is dropped at the firewall — the attacker gets nothing back.

Those 80+ requests saturated the PHP-FPM worker pool. By 4:40 AM, workers were timing out. The OOM killer stepped in, started terminating processes, and the server rebooted.

---

## Step 3: The Hidden Resource Crisis

```bash
free -h
```

```
Mem:    416Mi    192Mi    13Mi
Swap:     0B       0B      0B
```

**416MB RAM. Zero swap.**

A $3.50/month Lightsail nano instance running Nginx + PHP-FPM + a web application — with no swap space configured. When memory ran out, there was no fallback. The OOM killer had no choice.

This is a configuration mistake I've seen on dozens of servers. Trivial to fix, catastrophic to ignore.

---

## Step 4: Fail2ban Was Running But Doing Nothing

The server had Fail2ban installed and active — the tool that should automatically ban attacking IPs after a threshold of failed requests. I checked its status:

```bash
sudo fail2ban-client status nginx-req-limit
```

```
Currently banned: 0
Total banned:     0
```

Zero bans. During an 80-request attack.

The filter was configured to match:
```
limiting requests, excess:... by zone
```

But the actual Nginx log entries said:
```
access forbidden by rule
```

The regex never matched. Fail2ban watched the entire attack and did nothing — not because it wasn't running, but because the pattern was wrong. A subtle difference with significant consequences.

---

## The Fix: Three Things, 10 Minutes

**1. Add swap so the OOM killer has a safety net:**
```bash
sudo fallocate -l 1G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
```

**2. Fix the Fail2ban regex:**
```ini
[Definition]
failregex = access forbidden by rule, client: <HOST>
            limiting requests, excess:.* by zone.*client: <HOST>
ignoreregex =
```

**3. Verify it actually works:**
```bash
sudo fail2ban-regex /var/log/nginx/error.log /etc/fail2ban/filter.d/nginx-req-limit.conf
```

Result: **299 matches** from this morning's attack alone. Working as intended.

---

## What This Actually Demonstrates

This wasn't a catastrophic failure. It was a cascade of small misconfigurations:

- A security tool installed but misconfigured (Fail2ban with wrong regex)
- A resource limit with no fallback (no swap on a low-memory instance)
- A protection that worked technically but not operationally (Nginx 403 vs. IP ban)

Each one alone is manageable. Together, they brought down a server.

The skills that resolved this in under an hour — reading kernel and application logs, correlating events across time, distinguishing symptoms from root causes — are the same skills that prevent these issues from reaching production in the first place.

---

## The Question Worth Asking

When was the last time someone reviewed your server configuration, monitoring setup, and security tooling — not just to install them, but to verify they actually work under real conditions?

Installed ≠ configured. Configured ≠ tested. Tested ≠ working.

If you're running production infrastructure and don't have clear answers to those questions, that's worth a conversation.

**[Let's talk about your infrastructure.](https://rubenrangel.org/en/#contact)**

---

*Rubén Rangel — Senior Full-Stack Developer | 17+ years building and maintaining web applications in production | Security, monitoring, and incident response | Open to contract and full-time opportunities.*