Francesco Guala
The Methodology of Experimental Economics
Analytical
table of contents:
1 Introduction
This book
aims at showing that methodology is important and useful for experimental
economists, but also that philosophers of science can learn from experimental
economics. It is neither a handbook nor a textbook of experimental economics.
2. Inside
the laboratory
Experimental economists often complain that replication is
not valued enough in their discipline, but they fail to notice a crucial
distinction between mere repetition and replication. In this chapter I
introduce experimental economics to the novice by describing the replication of
an experimental phenomenon known as the ‘decay of overcontribution’ in public
goods games. Particularly important is the role of pilots and the extensive
checking for errors performed before, during, and after the experiment.
3. Hypothesis
testing
The Hypothetico-Deductive
(HD) model is a very popular, very simple and very general model of scientific
method. It can be used to highlight some basic logical problems of testing,
such as the Duhem-Quine problem: no hypothesis can be logically falsified by
the empirical evidence. As a consequence scientific reasoning must include an
inductive logic. In this chapter I also show what kind of hypotheses are
routinely tested by scientists, and introduce an important distinction between
‘data’ and ‘phenomena’.
The key to experimental
control is the controlled variation of one variable keeping the other
(background) conditions fixed. The rationale of variation can be explained
using a second important model of scientific method, the perfectly controlled
experimental design. This model is particularly important in experiments aimed
at testing causal hypotheses. Causes can be used to control or manipulate their
effects. Causal relations can be deterministic or probabilistic, and the
perfectly controlled experiment exemplifies a situation in which association
between variables reflect the underlying causal relations.
5. Prediction
Laboratory experimentation helps to tackle the Duhem-Quine
problem constructively, or to draw tight inductive inference from the evidence
to a given hypothesis. Much philosophical literature, however, has focused on
the wrong aspects of this inductive or evidential relation, by stressing the
importance of predictive success. In fact the crucial advantage of the experimental
method is that it allows the control of the background assumptions upon which
strong inductive inferences rest. This thesis is illustrated using the example
of preference reversal experiments.
6. Elimination
Bayesian confirmation theory stresses the importance of
the background, but for the wrong reasons. Scientists’ prior beliefs should not
be given too much weight in confirmation theory. What matters is whether the
background factors have been controlled by means of an effective experimental
design. The experimental method is best characterised as a procedure of
eliminative induction, where factors that may potentially disturb the inference
from the evidence to a hypothesis are checked one by one, until all sources of
error have been controlled for. Experiments on preference reversals provide
several examples of this strategy at work.
7. External validity
There is a trade-off between the internal validity of an experimental
result (whether a given laboratory phenomenon or mechanism has been correctly
identified) and its external validity (whether the results can be generalised
from the laboratory to the outside world). External validity is a genuine
problem and cannot be solved by metaphysical speculation or methodological
stipulation. It is an empirical issue that must be tackled and solved
empirically.
8. Economic engineering
The best example of successful external validity inference
is provided by cases of economic engineering, where a piece of the real world
is shaped so as to mirror the conditions of a laboratory experiment. I
illustrate this procedure using the early auctions of the Federal Communication
Commission as an example. The key external validity step is taken by comparing
field evidence with experimental evidence and using a so-called ‘no-miracle
argument’.
9. From the laboratory to the outside world
‘Radical localists’ argue that experimental results only
apply to laboratory circumstances, or to real-world circumstances that have
been engineered so as to resemble the lab. In reality, when experimenters
cannot shape the real world so as to fit the laboratory, they can try to shape
the laboratory so as to mimic the target system in the real world. Winner’s
curse experiments illustrate this principle at work. The inference from
experiment to the real world is a special kind of analogical argument, where
the inference is strengthened by making sure that the two systems are similar
in all relevant (causal) respects.
10. Experiments as mediators
Models and experiment share several important
characteristics. Both are systems that are created to aid scientists in their
investigations of a target system. They are ‘mediating tools’, an intermediary
step in the process connecting our speculations to the real world. Like models,
experiments can be closer to abstract theory or to application. The purpose of
an experiment is often to test the robustness of a phenomenon rather than its
applicability to a particular real--world situation.
11. On monetary incentives
The
debate on monetary incentives is used as an example to illustrate how
philosophical reasoning can help clarifying concrete problems arising from
scientific practice. I criticise the view that monetary incentives are a
necessary requirement for an adequate economic experiment, because different
experiments require different designs. There are no universal recipes in
science.