- 01
Start from a question a trader asks
A study begins with a question, not with a signal to defend. The question names the event that matters; the signals are candidates for leading it, and several are tried.
Which crypto markets break out next? The event is a pump onset; five ideas were candidates for lighting before one.
- 02
Define the event before reading any signal
The event is something a trader could see on the bar it happens, confirmed by what follows. The confirmation reads later bars, so it is only ever used to label events — never inside a signal, where it would be look-ahead.
Onset: an hourly close above the previous 24 hours’ high on more than twice the market’s usual volume. Confirmed when the market gains 4% more within 6 hours.
- 03
Write each candidate as a plain rule
A rule is true or false on every bar, built from numbers the desk already shows. Every candidate is reported, including the ones that fail — a study that only shows its winner is an advertisement.
“Beating BTC by 1.5% over the last 6 hours, still under its 24-hour high.” True or false, every hour, every market.
- 04
Grade it against the events
For every event: was the rule true at any point in the lead window before the onset? That share is the catch rate. Then the other direction: of every bar the rule was true, how often did an event follow within the window? That is the precision, read beside how often an event follows any random bar. Precision over random is the lift.
Lead window 6 hours. The quiet-leader rule was lit before 66% of 124 onsets; an event followed a lit hour 6% of the time against 3% for a random hour — a 2.1× lift.
- 05
Measure chance, twice
A rule can catch half the events by being lit half the time. So the same rule is graded against two placebos: its own pattern taken from another market at the same hours, which keeps the daily rhythm and how markets move together; and the same market’s pattern moved to other days. Ninety-nine seeded draws of each give the range of what chance alone catches. A signal has to beat that range, not merely score well.
Chance caught 35% (other markets) and 49% (other days) of the onsets the quiet leader caught 66% of.
- 06
Grade it again on history it never saw
Any rule that passes is graded a second time on the 1,000 hours before the window it was chosen on. Several candidates were tried, so the best in-sample score is partly luck; a winner that fails out of sample was that luck.
On the earlier window the quiet leader caught 60% of 153 onsets against 31–44% by chance, a 2.5× lift.
- 07
Apply one verdict rule
Validated only when both the catch rate and the precision beat both chance checks under a false-discovery gate (Benjamini–Hochberg, q = 0.10), with at least 30 events across at least 5 markets. The same rule grades every signal, before and after it is published.
- 08
Publish all of it, with the desk
The numbers, the losers, the windows, the limits, and the desk that runs the rule live — so a reader can watch the signal on today’s board rather than take the page’s word for it.
What a verdict means
- Validated
- Beat both chance checks in sample and held on the window it never saw. A real tendency — where to look, not what to do.
- No edge
- Lit before events about as often as chance, or lit so much of the time that it says nothing.
- Negative
- Lit before fewer events than chance. Occasionally useful inverted; never published as a call.
- Accruing
- Fewer than 30 events on the window — too few to say either way.
What this method does not do
- Events are not trades. No entry, exit, sizing, fees or slippage are modelled; a validated signal says where to look, never what to do.
- One stretch of history per study, one venue — Hyperliquid — and the liquid crypto perpetuals on the board that day.
- Several candidates are tried in every study. The unseen window guards against keeping a lucky one; it cannot make a short window long.
- The chance checks keep the market’s rhythm but not every structure in it. A signal that beats both can still be the product of a regime that ends.
Questions
Why not a backtest with profit and loss?
A P&L curve bundles a signal with an entry, an exit, sizing and costs, so a bad rule with a lucky exit looks good and a good rule with a bad exit looks bad. Grading against events isolates the one question a signal can answer: does it lead the thing you care about more often than chance?
What counts as lit before an event?
The rule was true on at least one bar in the lead window before the onset — six hours in the breakout study — or on the onset bar itself. Lit after the onset does not count, and a rule that only ever lights after the move is graded as missing it.
How do you avoid fitting the rule to the data?
The event is defined first, every candidate is written before any grade is read, every candidate is reported, and anything that passes is graded again on history it never saw.
Can I run this on my own signal?
Yes. On the desk, describe the signal and the event it should lead, and vyx grades it the same way — chance checks beside it — in the chat, or over the API for your own AI.
Run the same test on your own idea
Describe a signal and the event it should lead, and the desk grades it this way — chance beside it — in the chat or from your own AI. Free, no install.
Open VYX