Jev’s First Verdict: Correct in 5.6 Seconds

My TypeSafe invite arrived, so I read the Data Processing Addendum, created an API key, and ran my first real call to Jev. It flagged a fake AWS phishing email at probability 1.0 — and took 5.6 seconds to do it. Here's what I measured, and why I'm still not impressed.

My last post ended with everything installed and nothing to run it against. The invite came the next afternoon.

The email that let me in

The subject line was “TypeSafe AI: Your account is ready.” Before clicking anything, I ran the same check I’d just written a whole post about: the sender was team@updates.typesafe.ai, a subdomain of the real company, the footer listed a physical San Francisco address, and the email asked me to create an account rather than asking for a password or payment. It checked out.

Reading the terms before agreeing

Signup starts with the Terms of Service, Master Customer Agreement, Privacy Policy, and a Data Processing Addendum. I actually read the addendum, because it matters for what I plan to do next.

The parts worth knowing:

  • TypeSafe commits not to sell or share customer personal data as those terms are defined under CCPA, and not to retain or use it for any purpose other than providing the service.
  • They commit to notifying customers of a security incident within 72 hours.
  • Subprocessors are disclosed publicly, with advance notice before new ones are added.
  • Data may be transferred internationally under EU and UK data-transfer rules.
  • Retention is defined loosely: as long as necessary for the processing.

Nothing in it says they train on customer data, which was the thing I was looking for. Still, anything I send to Jev goes to their servers. Fake test emails are fine. Before I send real Wazuh log entries, I’ll strip internal IPs, hostnames, and usernames. That’s good practice with any third-party service, regardless of what the terms say.

What TypeSafe says Jev can’t do

The welcome page is refreshingly direct about limitations:

  • Not good at System 2 tasks. Jev handles fast, instinctive judgments, not slow multi-step reasoning.
  • Not trained on specialized domains. Relevant to me: it may not know security jargon deeply.
  • Not a generative chat model. You can’t chat with it. You define the shape of the answer, like writing multiple-choice questions.

They also concede that their biggest claim, frontier-level intelligence on System 1 tasks, is “the hardest claim to defend, and no one in the field has found a good way to prove it.”

That’s an invitation to test it.

Naming the API key

I created a key in the console and named it debian-jev-lab, after the machine and the project rather than after myself.

Every key I create would be “mine,” so naming it after me distinguishes nothing. Naming it after where it runs means that if a key leaks or a machine is compromised, I can revoke that one key without breaking anything else. Same reasoning as not reusing passwords.

I load it into my shell with read -s, so it never appears on screen or in my bash history, and the script reads it from the environment instead of having it hardcoded.

The test

phish_check.py sends one fake email to Jev:

URGENT: Your AWS account has been suspended. Verify your billing details within 24 hours at http://aws-verify-billing.co to avoid permanent deletion.

And asks three questions in a single call:

  • Choice: phishing, spam, or legit?
  • Score: how much pressure, from 0 to 2?
  • Noul (yes/no with probability): does it contain a link imitating a real brand?

The verdict

Category:         phishing
Probabilities:    {'legit': 0.0, 'spam': 0.0, 'phishing': 1.0}
Pressure score:   2.0 (0 to 2)
Suspicious link:  0.98

All three correct. The round trip took 5.6 seconds, which surprised me.

  • Category: phishing, at probability 1.0, with zero weight on the alternatives.
  • Pressure: 2.0, the top of my scale, “Threats or tight deadlines.” It read “within 24 hours” and “permanent deletion” exactly as a person would.
  • Suspicious link: 0.98, a 98% probability of yes. It caught that aws-verify-billing.co is imitating AWS.

I ran it four more times: 5.625s, 5.702s, 5.634s, 5.605s. Then I stripped it down to a single yes/no question instead of three, and got 5.604s. The number of questions makes no measurable difference, so the latency is a fixed cost per call, not per question. TypeSafe advertises under 100ms; that describes the model, not what a developer in Honolulu actually waits. Their own cookbook claims batching questions into one call is 9.6x faster, and my numbers support exactly that: three questions cost the same wall-clock time as one.

Why I’m not impressed yet

This email is textbook phishing: all caps, a threat, a 24-hour deadline, and a fake domain. Any classifier would flag it. A perfect score on an easy case tells you almost nothing about behavior on a hard one, and a confidence of exactly 1.0 is worth being a little suspicious of.

The real question isn’t whether Jev catches obvious phishing. It’s whether it can tell the difference between:

  1. A real AWS billing email. Should return legit.
  2. A legitimate marketing newsletter. Should return spam, not phishing.
  3. A genuine password-reset or security alert. Urgent, contains a link, asks you to act right now, and is completely real. This is where classifiers usually break, and it’s where false positives train people to ignore warnings.

That last case is the one I care about. A tool that flags everything urgent as phishing isn’t a security tool. It’s noise.

Next

I’ll run all three against Jev and publish whatever comes back, including the cases where it’s confidently wrong. Then I’ll try Wazuh alerts from my home lab, with the identifying details stripped.

Aloha from Honolulu. 🌺