Tune risk detection sensitivity, one policy at a time

A PII detection rule matches on shape, not on whether something is actually sensitive. That's why it flags an internal ID that happens to be IBAN-shaped, or an internal reference number that happens to look like a credit card, right alongside real PII. Those false positives erode trust in every finding the rule raises, and often the rule isn't wrong so much as mistuned for the traffic it's watching.

That mistuning comes down to a confidence threshold that has to serve every risk policy at once. Until now, Speakeasy's threshold was global, one minimum confidence shared across every policy, so tightening it for one meant tightening it for all of them.

Each risk policy now carries its own detection sensitivity, set where you build the policy.

What the sensitivity slider changes in the policy wizard

Risk policies now expose a per-policy minimum match confidence (the score a detection has to clear before it becomes a finding). In the policy wizard there's a new Sensitivity step with a slider. Drag it up to demand higher-confidence matches and cut noise, or drag it down to catch more and accept some false positives. The value is saved with the policy and loaded back when you edit it.

The global default is unchanged. Every policy still starts at a minimum confidence of 0.5. We tested moving it higher, but against a 100k-message production sample a 0.75 floor started dropping real findings, so 0.5 stays the right baseline for most rules.

What this release adds is the headroom to go higher per policy where it helps. Analyzers that match structured values like IBANs or phone numbers key off tight, well-shaped patterns, so raising their sensitivity trims noise without costing you true positives. The change is additive and backward compatible, so nothing re-tunes itself under you.

Why one global confidence threshold doesn't work for every risk policy

The same detector is the right amount of strict in one place and far too loud in another. A policy watching user prompts for leaked secrets wants to err toward catching things. A policy scanning a tool response that legitimately returns customer records wants to stay quiet unless it's sure. One global confidence threshold can't satisfy both, so it forces a compromise that's wrong for every policy at once.

Sensitivity is the precision dial for that problem. It sits alongside the scoping and exemption controls we wrote about earlier:

• Scope decides where a policy runs.
• Exemptions decide what to skip.
• Sensitivity decides how sure a match has to be before it counts.

Together they let you author a deliberately broad policy and then tune it down to signal without reaching for a blunter detector.

How to tune detection sensitivity against real traffic before going live

Sensitivity is most useful when you set it against evidence rather than instinct. Built-in detection rules can be run over a sample of text or a set of existing sessions before a policy goes live, so you can watch how a confidence threshold change moves the line between caught and ignored, then commit the number that keeps the policy quiet without making it weak.

How to set sensitivity on a risk policy

Open the policy wizard, build or edit a risk policy, and step to Sensitivity. Set the slider where the policy's job warrants (higher for precision, lower for recall) and save. Existing policies keep their current behavior until you choose to tune them.

Tuning detection sensitivity or confidence thresholds across a large agent footprint? Book time with our team and we'll walk through it with you.