How can randomness be explained?
The procedure is known in principle. See:
H Grabert,
Projection Operator Techniques in Nonequilibrium
Statistical Mechanics,
Springer Tracts in Modern Physics, 1982.
This is, in fact, the best book about projection operator techniques that I know. It may not be perfect, but it is cleanly and clearly written, with all relevant details, and strongly focused on what is most important.
Grabert applies the procedure to various interesting special cases, and thus arrives at, for example, the Navier-Stokes equation, stochastic diffusion processes, master equations and quantum Markov processes a la Lindblad, all from the fundamental Liouville equation.
His presentation goes well beyond what is normally done, as it:
1. shows that the method is universal and yields all traditional equations (in my opinion it should therefore be as prominent in Physics education as Hamiltonian mechanics)
2. makes very clear how randomness arises (at least if one reads between the lines – and with the right eyes)
If one applies the same procedure to the measurement process (which seemingly no one before me has taken to the necessary level of universality) meaning one takes as relevant operators all functions of (commuting) pointer variables and all (non-commuting) operators that describe the quantum mechanical subsystem, one gets, in the Markov approximation, a quantum-classical jump diffusion process for the expectation values of the reduced variables as a reduced dynamics.
Thus one sees exactly how randomness arises: it is nothing more than the high frequency, un-modelled part of dynamics which is projected away, naturally not letting itself be completely eliminated from the world, but rather leaving traces of itself behind.
Probabilities are thus simply consequences of the selected description level.
If the universe is described with all of the details which are objectively to be found within it, then it is deterministic. However, if a subsystem is described as though it were alone in the universe (and that is what is practically always done), then the influence of the un-modelled part is admittedly nevertheless there, but can still only be approximately modelled using stochastic influences. That is why it appears random in this reduced description. And if it is conceived of quantitatively, it is precisely the Born rule that comes out of the projection operator analysis!
It is qualitatively not very different to rolling dice. If all forces are modelled, then the trajectory of the roll is determined; if only the score is modelled, traditional probabilities emerge. Only the numerical formulae are somewhat different.
The effective nonlinearity of the observable stochastic dynamics results from the projection operator formalism. Here there is naturally an unproven step of the kind that we always observe when an irreversible dynamics emerges out of a reversible dynamics. But since Boltzmann this has been considered well-understood, and there is a consensus to this effect amongst physicists today.
Arnold Neumaier (Arnold.Neumaier@univie.ac.at) A theoretical physics FAQ