Probability : an alternative and extension of logic
Computers have brought a new dimension to modeling. A model, once translated into a program and run on a computer, may be used to understand, measure, simulate, mimic, optimize, predict, and control. During the last fifty years science, industry, finance, medicine, entertainment, transport, and communication have been completely transformed by this revolution.
However, models and programs suffer from a fundamental flaw: incompleteness. Any model of a real phenomenon is incomplete. Hidden variables, not taken into account in the model, influence the phenomenon. The effect of the hidden variables is that the model and the phenomenon never have the exact same behaviors. Uncertainty is the direct and unavoidable consequence of incompleteness. A model may not foresee exactly the future observations of a phenomenon, as these observations are biased by the hidden variables, and may not predict the consequences of its decisions exactly.
In 1781, Sir William Herschell discovered Uranus, the seventh planet of the solar system. In 1846, Johann Galle observed for the first time, Neptune, the eighth planet. In the meantime, both Urbain Leverrier, a French astronomer, and John Adams, an English one, became interested in the “uncertain” trajectory of Uranus. The planet was not following exactly the trajectory that Newton’s theory of gravitation predicted. They both came to the conclusion that these irregularities should be the result of a hidden variable not taken into account by the model: the existence of an eighth planet. They even went much further, finding the most probable position of this eighth planet. The Berlin observatory received Leverrier’s prediction on September 23, 1846 and Galle observed Neptune the very same day!
Logic is both the mathematical foundation of rational reasoning and the fundamental principle of present day computing. However, logic, by essence, is restricted to problems where information is both complete and certain. An alternative mathematical framework and an alternative computing framework are both needed to deal with incompleteness and uncertainty.
Probability theory is this alternative mathematical framework. It is a model of rational reasoning in the presence of incompleteness and uncertainty. It is an extension of logic where both certain and uncertain information have their places.
James C. Maxwell stated this point synthetically:
The actual science of logic is conversant at present only with things either certain, impossible, or entirely doubtful, none of which (fortunately) we have to reason on.
Therefore the true logic for this world is the calculus of Probabilities, which takes account of the magnitude of the probability which is, or ought to be, in a reasonable man’s mind.
James C. Maxwell
Considering probability as a model of reasoning is called the subjectivist or Bayesian approach. It is opposed to the objectivist approach, which considers probability as a model of the world. This opposition is not only an epistemological controversy; it has many fundamental and practical consequences.
To model reasoning, you must take into account the preliminary knowledge of the subject who is doing the reasoning. This preliminary knowledge plays the same role as the axioms in logic. Starting from different preliminary knowledge may lead to different conclusions. Starting from wrong preliminary knowledge will lead to wrong conclusions even with perfectly correct reasoning. Reaching wrong conclusions following correct reasoning proves that the preliminary knowledge was wrong, offers the opportunity to correct it and eventually leads you to learning. Incompleteness is simply the irreducible gap between the preliminary knowledge and the phenomenon and uncertainty is a direct and measurable consequence of this imperfection.
In contrast, modeling the world by denying the existence of a “subject” and consequently rejecting preliminary knowledge leads to complicated situations and apparent paradoxes. This rejection implies that if the conclusions are wrong, either the reasoning could be wrong or the data could be aberrant, leaving no room for improvement or learning. Incompleteness does not mean anything without preliminary knowledge, and uncertainty and noise must be mysterious properties of the physical world.
The objectivist school has been dominant during the 20th century, but the subjectivist approach has a history as long as probability itself. It can be traced back to Jakob Bernoulli in 1713:
Uncertainty is not in things but in our head: uncertainty is a lack of knowledge.
Jakob Bernoulli, Ars Conjectandi (Bernouilli, 1713)
to the Marquis Simon de Laplace, one century later, in 1812:
Probability theory is nothing but common sense reduced to calculation.
Simon de Laplace, Théorie Analytique des Probabilités (Laplace, 1812)
to the already quoted James C. Maxwell in 1850 and to the visionary Henri Poincaré in 1902:
Randomness is just the measure of our ignorance.
To undertake any probability calculation, and even for this calculation to have a meaning, we have to admit, as a starting point, an hypothesis or a convention, that always comprises a certain amount of arbitrariness. In the choice of this convention, we can be guided only by the principle of sufficient reason.
From this point of view, every sciences would just be unconscious applications of the calculus of probabilities. Condemning this calculus would be condemning the whole science.
Henri Poincaré, La science et l’hypothèse (Poincaré, 1902)
and finally, by Edward T. Jaynes in his book Probability theory: the logic of science (Jaynes, 2003) where he brilliantly presents the subjectivist alternative and sets clearly and simply the basis of the approach:
By inference we mean simply: deductive reasoning whenever enough information is at hand to permit it; inductive or probabilistic reasoning when – as is almost invariably the case in real problems – all the necessary information is not available. Thus the topic of “Probability as Logic” is the optimal processing of uncertain and incomplete knowledge.
Edward T. Jaynes, Probability Theory: The Logic of Science (Jaynes,2003)