Fitting Models to
Data
When an observation which is not predicted or explained by
the current a scientist’s current model of reality is discovered and resists all
attempts to correct it, the model must change to accommodate this new fact. In
most instances this can be done by simply extending the existing model without
any fundamental change to its existing structure. However, with sufficient
extension any model can be made to fit almost any observations, creating a
danger of ‘forcing’ the model to fit the data. Scientists detect this ‘forcing’
when proposed model extensions are not clearly logical and the justification is
inadequate or seems contrived. The key test for proposed extensions is their
ability to correctly predict similar situations to the original problematic
observation (the reason for this will be discussed later). An extension which
proves useful for many additional situations is adopted where as if the
extension continues to be insufficient it may trigger a proposal of a new model
of reality.
New models of reality come in two
major forms; those which challenge an existing model of reality by being
grossly consistent with past predictions & results while also opening up
new areas, and those which carve off a subset of reality into a new field of
study. The latter is typically triggered by an advance in technology or
technique which improves the resolution or scale with which observations can be
made. While the former are triggered by problematic observations which cannot
be explained or predicted using the existing model of reality.
New models are evaluated by
scientists before they are adopted or integrated into their personal model of
reality. The criteria used in this evaluation are the ability of the model to
explain existing observations (‘the facts’) as well as its ability to make
accurate predictions. Since each scientist has differing beliefs about the
importance of various observations to the understanding of the world based on
their personal model, each scientist will evaluate the new model slightly
differently. Kuhn correctly observes
that it typically takes 20 years for the majority of the scientific community
to adopt a new model of reality and often many opponents are never convinced
but rather retire in the interim. Because of the importance of prediction to
the evaluation of models senior scientists have more direct experience with the
prediction ability of the existing model; they tend to require more evidence of
a new model’s prediction ability before they will accept it. Scientific
research is slow so it often takes decades to accumulate sufficient validation
of a new model’s predictions to convince a majority of the scientists in a
field.
When models are extended or
devised, the scientist relies exclusively on existing data and models to inform
their thinking. Thus there is always a risk of over-fitting the data. Models
which over-fit the data will explain the existing data extremely well but will
fail to make accurate predictions because they are not modelling all of reality
due to biases in which data have been collected. Prediction ability is most
important when comparing models due to this problem. This over-fitting problem
also explains why the simplicity of a model is highly valued.
Value of Simplicity
“It is pointless to do
with more what can be done with fewer.”
― William of
Ockham (via: Wikiquote)
Above is just one
quote which represents the principle known as Occam’s Razor. This is highly
related to the issue of over-fitting discussed above, where a model of reality
is not informative or useful because it is only explaining the part of reality
we have already observed. Complicated models of reality can explain observed
data well just because it is complicated without providing understanding (or
predictability), because they effectively have a different explanation for each
subset of data, valuing simplicity avoids wasting too much time on these
models. In addition, excessive complexity can indicate the scientist is ‘forcing’
the model to fit the data.
Another reason for the value of
simplicity can be understood by its relationship with null hypotheses. Null
hypotheses are what are actually tested in most scientific experiments. They
are the model of what we expect if there is no pattern, no rules; so that we can
determine if our observations support our model or prediction. Similarly
observations cannot be said to support the additional elements in a more
complex model if a simpler model matches the observations just as well. Models
or model elements without support from observations cannot be considering
informative or meaningful.
Success of Science
In contrast to Kuhn, I shall close by explaining why science does
progress despite seeming to backtrack and flip-flop. Science is exploration. It wanders blindly into
the unknown feeling its way along. Individual ideas may wander back and forth or
even backtrack. However, with each new model of reality that is adopted more
and better predictions can be made without losing the explanation of
observations already made. As such each new model represents a closer
approximation to reality, a fixed target (given the assumptions outlined in
Part 1) we may never reach.
Part 1
Part 2
Part 3
Part 1
Part 2
Part 3
