There's a principle I've followed throughout my 20+ years in system administration. It's so simple that it almost seems unnecessary to state, yet I've watched countless hours of troubleshooting wasted because it wasn't followed. This principle wasn't written in any sysadmin handbook—it was written around 300 BC, in Alexandria.
Euclid knew: One unknown, one solution. Two unknowns, infinite possibilities.
A Night I'll Never Forget
Years ago, one of our customers' e-commerce sites crashed. At 3 AM, naturally. When I jumped into the system, I discovered that the team had made two changes simultaneously the previous evening: they'd updated the router configuration AND upgraded the firmware.
Now the site was down, and we had two suspects. Was it the configuration change? Or an incompatibility introduced by the new firmware? There was no way to know. We couldn't roll back the firmware easily. And we weren't even sure if the config change was the culprit.
We burned three hours that night. If only one change had been made, the diagnosis would have taken three minutes.
The Mathematical Foundation
Euclid's principle from his Elements is elegantly simple. An equation with one unknown has a single solution:
x + 5 = 10 → x = 5
But the moment you add a second unknown, the possibilities become infinite:
x + y = 10 → (1,9), (2,8), (3,7), (4,6), (5,5)...
System administration works exactly the same way. When there's only one variable, finding the cause is trivial. The moment two things change simultaneously, causality fractures.
The Rule in Practice
This is now my iron-clad rule:
- Never make two changes at once. Config change and firmware update? Make them on different days.
- Test after each change. Complete your test before moving on. Don't postpone.
- Keep a changelog. Date, time, what changed, who did it, and the results.
- Especially at night, especially in emergencies—one change at a time. The urge to "quickly fix a couple things" is strongest when you're tired. Resist it.
Why This Is Difficult
Our instincts work against this principle. When we spot a problem, we want to fix "everything at once." Efficiency seems to demand doing two things while you're already in the system. And under time pressure, the urge to be "comprehensive" grows even stronger.
But this is illusory efficiency. The time you "save" by batching changes you'll pay back many times over when things go wrong.
The 2,300-Year-Old Wisdom
Euclid probably never saw a server room. But the mathematical principles he codified were describing universal truths about cause and effect.
One unknown, one solution. Two unknowns, endless possibilities.
This rule has saved me countless hours of troubleshooting over 20 years. The best part? It's free to apply. All it requires is a bit of patience and discipline.
The next time your fingers are itching to change two things at once, remember: Euclid wrote this down 2,300 years ago. Maybe there's a reason it still applies today.