Decision theory has a rich history, which is closely related to the concept of probability itself. Almost 300 years ago, Bernoulli thought about the topics we are discussing here, and he identified a paradox.

Here is an elegant summary of Bernoulli’s paradox from Jaynes, 2003, Section 13.2.

Imagine that we have a coin that is slightly more likely to be heads, or that p(heads)=0.51. One can bet that the coin comes up heads. We have that expected (profit) = p(heads)*bet + p(tails)(-bet) = 0.51*bet + 0.49(-bet) = 0.02*bet

Jaynes (and Bernoulli, in a slightly different way) noted that 0.02*bet is always greater than zero. To maximize expected profit, one should therefore always play this game.

However, could you imagine playing this game with 1 million dollars? You have almost a 50% chance of losing it. The paradox is in the disconnect between the strategy that gives the best expected profit and our common sense.

Whether playing this game is sensible is highly dependent on the money one has at their disposal. Bernoulli resolved the paradox by noting that we, in general, do not only want to maximize profit.

Rather, we want to maximize utility, which is related to profit, but is also a function of other things (for example, the available funds of the player).

In the threads so far, we have called utility “reward”, which can be conceivably formulated as R(bet, available funds) to resolve the paradox.

The general idea that a reward is a function of the outcomes (or, what we called “states”), not the outcome itself, is crucial here.

For example, in a medical setting one might not want to just maximize some outcome (years lived), but instead a function, R(years lived, quality of life).

References: Daniel Bernoulli. Exposition of a new theory on the measurement of risk. 1738. Jaynes, Edwin T. Probability theory: The logic of science. Cambridge university press, 2003.