Inferring the rule from the outcome

The problem, in one sentence

There is a whole family of problems where the data hands you the outcome of a decision but not the criterion that produced it. A jury publishes its winner without its scoring grid. A pricing engine shows a price without its rule. A committee publishes the winner, the ranking and every amount — but the “method” field is empty, or filled in at random.

Mine was the third. Several thousand public decisions, each with its candidates, their amounts, and the winner’s name. The rule that produced that winner was recorded only intermittently, and when it was, it had often been copied from a form field that means nothing.

The question fits on one line: given the winner and the full set of bids, can you recover the rule? The answer is yes for the large majority of cases — provided you accept not answering for the rest.

Why you cannot simply read the answer

The first instinct is to look for the rule in the text: the record’s field, the justification sentence in the minutes. That is what my collector already did, with a pattern table.

It works when the source is honest. The trouble is that it isn’t always, and above all that it is silent when it does not know: the field carries a default value that looks like information.

A default value that looks like data is worse than an empty field. It raises no alert, it travels through the whole pipeline unopposed, and it ends up on a dashboard where someone reads it as a fact.

Hence the inversion: stop believing the label, and recompute it from what is not up for debate — amounts and a winner.

Eight observations, no assumptions

The starting point is the list of admissible bids for a given decision, and knowing which one won. On that list, eight quantities can be computed without assuming anything.

Are all the bids identical? A perfect tie between three or more candidates does not happen by chance: it means price was not what the competition was about — it is a percentage applied to a shared base.

Is the winner the cheapest? Is it the one closest to a computable reference price, from below? Does an admissible cheaper candidate exist that lost? — that last one is the most informative of the set: on its own it rules out any lowest-price rule.

And four weaker but useful tiebreakers: is the decision earlier than the regulatory reform, does the procedure label carry a hint, how many winners in the group, how dispersed are the amounts.

Preparation matters as much as the signals. Before computing anything you have to remove from the list whatever the rule itself excludes: administratively rejected bids, abnormally low ones, ones above a ceiling. A signal computed on the wrong population says nothing at all.

And everything is computed per lot, never globally: one decision can cover several lots awarded separately, and mixing them muddies every signal.

The tree

The tree is short, deliberately. Every node is an observation, not a heuristic — you can reopen any decision and say exactly which inequality produced it.

The important part is the “undetermined” exit. A single admissible candidate, or a two-way tie, support no conclusion. The temptation is to rule anyway, at low confidence. That is a mistake: a wrong label at 0.4 confidence gets read as a label, not as uncertainty.

no identified winner ──────────────────────────► UNDETERMINED (0.00)

the label explicitly names the rule ───────────► that rule (0.95)

n ≥ 3 and every amount equal ──────────────────► shared base (0.95)

an admissible CHEAPER candidate lost
        │
        ├── the winner is the closest to the
        │   reference price, from below ───────► reference-price rule
        │                                        (0.95 if n ≥ 3, 0.70 if n = 2)
        │
        └── it is not that either ─────────────► the decision turned on
                                                 something other than price (0.80)

the winner is the cheapest ────────────────────► price rule
                                                 (0.90 pre-reform,
                                                  0.70 post — ambiguous)

Confidence, written before the results

The confidence model is the part of the system that mattered most, and the part nobody talks about. It was written before looking at any results, so it could not be calibrated on the wish to be right.

Two principles govern it.

Confidence drops when an input is missing. Without an estimate, the reference price cannot be computed: the ceiling falls to 0.50 however clean the other signals are. Without the category, the abnormality threshold falls back to a default, and we take a notch off.

Confidence drops when two rules predict the same thing. The most frequent case is a winner who is simultaneously the cheapest and the closest to the reference price. Both rules yield the same winner: observation cannot separate them. The law settles it, observation does not — so the label is written, but confidence stays low and the case is flagged ambiguous.

Honest confidence costs you points on results tables and earns them on decisions. It is a trade you should make consciously, once, at the start.

The first pass fails

A hundred decisions, drawn from the most recent ones having a published winner, an estimate, and at least one priced admissible bid. No favourable selection: the entire population meeting those three conditions.

Result: 4 cases out of 100 above the confidence threshold. Seven plainly wrong classifications. A failure, in other words.

Two symmetric mistakes, and instructive ones.

A two-way tie mistaken for a rule. Two admissible candidates had submitted exactly the same amount, a round number. The “all amounts equal” signal fired, at 0.90 confidence. Two people landing on the same round number is a coincidence. Three is a rule. The signal had no minimum count.

An explicit label ignored. Six decisions whose procedure label literally named the rule were classified otherwise, because the tree only looked at the text as a last resort, as a weak hint. It was the other way round: when the source states the rule outright, that is the most direct signal in existence. It belongs ahead of the computed signals, not behind them.

The design error is the same in both: I had ordered the signals by elegance — computed first, text second, because computing is more satisfying than reading — instead of ordering them by evidential strength.

Four rules, and the second pass

Four corrections, each born from a specific case that had classified badly — none from a general idea about what “should” work.

R1 — the explicit label goes first and overrides everything else. R2 — a competition-type label combined with a winner who is not the cheapest is enough to conclude the decision did not turn on price. R3 — the tie signal now requires three candidates. R4 — the configuration where two rules predict the same winner has its confidence raised from 0.55 to 0.70, because the law does settle it.

The result, on exactly the same hundred cases:

Thirty-eight cases above threshold instead of four. All nine shared-base decisions correctly identified out of nine. No false positives. And mean confidence rising from 0.67 to 0.78 — without a single rule added to inflate a number.

The result I was not expecting

Seven decisions landed in a category I had not expected to see so often: the winner there is strictly dearer than an admissible candidate, and is not the closest to the reference price either. No price rule produces that winner. So the decision turned on something else — a technical score — while the source’s label said “on price”.

These are not engine errors. They are errors in the data, which the engine surfaced.

They count twice over, because another part of the product computed a “theoretical winner” from that label, for decisions whose outcome is not yet published. On those seven configurations it produced a wrong winner. Silently.

That is the least visible benefit of this kind of work, and often the most profitable: a model that recomputes a value becomes an audit of that value. Run both side by side for a while before replacing one with the other — the gap between them is a list of bugs.