My Website Went Down This Morning. Here's Exactly What Happened — And How I Fixed It.
At 4:40 AM, my server stopped responding. No warning, no alert, no obvious reason. Just a website that was down and a Cloudflare notification about a domain expiring in 25 days — which had nothing to do with it.
This is a real incident that happened today. Here's the full breakdown.
Step 1: Stop Guessing, Start Reading Logs
The first instinct when a server goes down is to restart everything and hope it works. That's the wrong move. If you don't know why it crashed, it will crash again.
sudo journalctl -xe --since "10 hours ago"
The logs showed the server had rebooted at 14:25 UTC. Not a service crash — a full system reboot. That narrowed it down immediately.
sudo tail -n 100 /var/log/nginx/error.log
That's where the real story was.
Step 2: The Attack Nobody Noticed
Between 00:11 and 00:11:50 AM — just 40 seconds — a single IP from AWS Seoul (3.38.148.216) sent over 80 requests scanning for exposed .env files:
GET /FRONTEND/.env
GET /.env.production
GET /admin/.env.local
GET /laravel/.env.local
... (80+ variations)
Nginx blocked every single one with access forbidden by rule. The blocks worked. But here's the problem: blocking a request with a 403 is not the same as banning the IP.
- A 403 means your server still received the request, processed it, and responded.
- A proper IP ban means the connection is dropped at the firewall — the attacker gets nothing back.
Those 80+ requests saturated the PHP-FPM worker pool. By 4:40 AM, workers were timing out. The OOM killer stepped in, started terminating processes, and the server rebooted.
Step 3: The Hidden Resource Crisis
free -h
Mem: 416Mi 192Mi 13Mi
Swap: 0B 0B 0B
416MB RAM. Zero swap.
A $3.50/month Lightsail nano instance running Nginx + PHP-FPM + a web application — with no swap space configured. When memory ran out, there was no fallback. The OOM killer had no choice.
This is a configuration mistake I've seen on dozens of servers. Trivial to fix, catastrophic to ignore.
Step 4: Fail2ban Was Running But Doing Nothing
The server had Fail2ban installed and active — the tool that should automatically ban attacking IPs after a threshold of failed requests. I checked its status:
sudo fail2ban-client status nginx-req-limit
Currently banned: 0
Total banned: 0
Zero bans. During an 80-request attack.
The filter was configured to match:
limiting requests, excess:... by zone
But the actual Nginx log entries said:
access forbidden by rule
The regex never matched. Fail2ban watched the entire attack and did nothing — not because it wasn't running, but because the pattern was wrong. A subtle difference with significant consequences.
The Fix: Three Things, 10 Minutes
1. Add swap so the OOM killer has a safety net:
sudo fallocate -l 1G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
2. Fix the Fail2ban regex:
[Definition]
failregex = access forbidden by rule, client: <HOST>
limiting requests, excess:.* by zone.*client: <HOST>
ignoreregex =
3. Verify it actually works:
sudo fail2ban-regex /var/log/nginx/error.log /etc/fail2ban/filter.d/nginx-req-limit.conf
Result: 299 matches from this morning's attack alone. Working as intended.
What This Actually Demonstrates
This wasn't a catastrophic failure. It was a cascade of small misconfigurations:
- A security tool installed but misconfigured (Fail2ban with wrong regex)
- A resource limit with no fallback (no swap on a low-memory instance)
- A protection that worked technically but not operationally (Nginx 403 vs. IP ban)
Each one alone is manageable. Together, they brought down a server.
The skills that resolved this in under an hour — reading kernel and application logs, correlating events across time, distinguishing symptoms from root causes — are the same skills that prevent these issues from reaching production in the first place.
The Question Worth Asking
When was the last time someone reviewed your server configuration, monitoring setup, and security tooling — not just to install them, but to verify they actually work under real conditions?
Installed ≠ configured. Configured ≠ tested. Tested ≠ working.
If you're running production infrastructure and don't have clear answers to those questions, that's worth a conversation.
Let's talk about your infrastructure.
Rubén Rangel — Senior Full-Stack Developer | 17+ years building and maintaining web applications in production | Security, monitoring, and incident response | Open to contract and full-time opportunities.