A Deep Dive on Automation, Interpretation, and Why the Hard Part of Security Was Never the Code
The Comfortable Lie of “Objective” Systems
Every analyst who has stared at a SIEM dashboard at 2 a.m. knows the feeling: a wall of alerts, a CVSS score next to each one, and the quiet hope that the numbers will just tell you what to do. That hope is understandable. It’s also a lie we tell ourselves to make an unbearable workload feel manageable.
A CVSS score looks objective. It isn’t. It’s a frozen judgment call — made by a team, at a point in time, under a specific set of assumptions about what “critical” means for a hypothetical average system. The moment you apply it to your system, in your context, with your business’s tolerance for downtime, the “objective” number becomes an interpretation again. The math didn’t disappear. It just got outsourced to whoever wrote the scoring rubric, and their assumptions came along for the ride, invisible.
This is the pattern that shows up everywhere once you start looking for it: the technical layer is the easy part. The judgment layer sitting on top of it is the hard part — and it is irreducibly human.
The Pipeline That Hides the Judgment
Every automated security system — an IDS, a spam filter, an anomaly detector — runs the same basic pipeline:
Input → Output → Interpret → Decision → Action
Raw data comes in. The system produces a number: a probability, a classification, a flag. That number is not yet meaning. Someone or something has to interpret it — turn 0.92 into “this is malicious” — before a decision can be made, and before that decision becomes a real-world action: an account locked, a connection dropped, an alert paged to someone’s phone at 2 a.m.
The dangerous assumption is that this pipeline is a straight technical line, with judgment only showing up at the very end, when a human finally looks at a dashboard. In reality, judgment is smuggled in at every stage:
- Input — what data even gets collected reflects a decision about what’s worth watching.
- Output — the threshold that turns a raw score into a binary flag is a judgment call, usually made once, by one team, and rarely revisited.
- Interpret — the same output can mean different things depending on who’s reading it and what they already believe about the system’s risk profile.
- Decision — automated response rules (“auto-block after 3 failed logins”) encode someone’s prior guess about what’s reasonable.
- Action — irreversible, and the only stage where the cost of every judgment made upstream finally comes due.
Automation doesn’t remove judgment from this chain. It compresses it — pushing decisions that used to happen slowly, with a human in the loop, into rules written once and then executed thousands of times a second, unquestioned, until something breaks badly enough to force a second look.
Four Questions That Aren’t as Neutral as They Sound
Before any alert becomes an action, it passes through a filter that feels procedural but isn’t:
- Does this matter?
- Is this a priority?
- Is the response reasonable?
- Is the residual risk tolerable?
These read like a checklist. They function like one, in the sense that a triage engineer moves through them fast, almost automatically. But none of them has a formula. Each is answered through a lens that never shows up in the postmortem report:
Judgment, because no rule fully resolves ambiguous cases. Ethics, because “reasonable” always implies a value about who bears the cost of being wrong — the user locked out of their account, or the org exposed to one more minute of risk. Interpretation, because the same alert reads differently to a security engineer who’s seen this pattern turn into a breach before, versus one who hasn’t. Culture, because what counts as “tolerable” risk in a fast-moving startup is negligence in a hospital network. Communication, because a risk explained to a fellow analyst and the same risk explained to a board produce two different decisions, even when the underlying facts are identical.
Strip away the checklist framing and what’s actually happening is a small act of ethics, performed under time pressure, dozens of times a shift.
Why Frequency Isn’t Truth — and Why That Matters More as Systems Scale
There’s a specific failure mode that gets more dangerous, not less, as detection systems get faster: mistaking frequency for validity.
An alert that fires constantly starts to feel true by sheer repetition — the same mechanism behind the illusory truth effect, where a claim repeated often enough starts to feel correct regardless of accuracy. A false positive pattern that’s been auto-suppressed for six months isn’t validated by its silence; it’s just been trained into the blind spot. Majority behavior in a dataset isn’t inherently the correct baseline either — it’s just what’s common, and “common” is not synonymous with “safe” or “true.” Historically, plenty of majority positions in security (perimeter-only defense, “internal traffic is trusted”) turned out to be wrong, sometimes catastrophically.
Automation is exceptionally good at scaling repetition. It is not, by itself, good at scaling correction. A rule that encodes a wrong assumption doesn’t get less wrong by running ten thousand times a day — it just produces ten thousand wrong outcomes instead of one, and does it fast enough that the humans meant to catch the error never get the chance to look closely.
Rhetoric Didn’t Go Away — It Just Changed Targets
There’s a version of this argument that shows up constantly in security-adjacent AI work right now, and it’s worth naming directly: prompting is a form of rhetoric.
Ethos, pathos, and logos map onto prompt design almost exactly — establishing authority (“act as a senior analyst”), invoking stakes (“this is for a critical audit due tomorrow”), and structuring logical constraints all shape a model’s output the way a rhetorician shapes an audience’s judgment. Framing works the same way in both domains: how a request is phrased steers the response as surely as how a news story is framed steers a reader’s interpretation of identical facts.
Where this gets sharpest for security specifically is adversarial prompting — jailbreaking, prompt injection, social engineering aimed at a system instead of a person. These aren’t engineering exploits in the traditional sense. They’re rhetorical exploits, aimed at a system that was shaped by human-generated training data and inherited human-like susceptibilities to authority, urgency, and framing. A more capable model doesn’t make context, intent, and framing less important — it raises the stakes of getting them wrong, because the system now acts on that framing faster and at greater scale than a human target ever could.
The Thesis, Stated Plainly
Put all of this together and a single claim holds up under its own weight:
Technology scales judgment. It doesn’t replace it. The more powerful and automated the tools become, the more the outcome depends on the humanity behind the decisions to build them, deploy them, and interpret what they report.
A faster detection pipeline doesn’t reduce the number of judgment calls being made — it increases how many get made per second, and raises the cost of each one being wrong. A more persuasive language model doesn’t make rhetoric obsolete — it hands rhetorical leverage to anyone willing to use it, at a scale no single human speaker ever had. An “objective” score doesn’t remove ethics from a triage decision — it just hides the ethics inside a number that looks like it doesn’t need defending.
None of this is an argument against automation, or against building faster and more capable security tooling. It’s an argument against the comfortable fiction that better tools mean the human judgment layer matters less. It’s the opposite. The faster the system, the less room there is to catch a bad assumption before it becomes an action — which means the assumptions have to be good, examined, and owned by someone, before the system ever gets to make its ten thousandth decision per second.
The code was always the easy part. It still is.