Francesco Guala

The Methodology of Experimental Economics

Cambridge University Press 2005



Analytical table of contents:


1          Introduction

This book aims at showing that methodology is important and useful for experimental economists, but also that philosophers of science can learn from experimental economics. It is neither a handbook nor a textbook of experimental economics.


Part one: Inferences within the experiment


2.         Inside the laboratory

Experimental economists often complain that replication is not valued enough in their discipline, but they fail to notice a crucial distinction between mere repetition and replication. In this chapter I introduce experimental economics to the novice by describing the replication of an experimental phenomenon known as the ‘decay of overcontribution’ in public goods games. Particularly important is the role of pilots and the extensive checking for errors performed before, during, and after the experiment.


3.         Hypothesis testing

The Hypothetico-Deductive (HD) model is a very popular, very simple and very general model of scientific method. It can be used to highlight some basic logical problems of testing, such as the Duhem-Quine problem: no hypothesis can be logically falsified by the empirical evidence. As a consequence scientific reasoning must include an inductive logic. In this chapter I also show what kind of hypotheses are routinely tested by scientists, and introduce an important distinction between ‘data’ and ‘phenomena’.


4.         Causation and experimental control

The key to experimental control is the controlled variation of one variable keeping the other (background) conditions fixed. The rationale of variation can be explained using a second important model of scientific method, the perfectly controlled experimental design. This model is particularly important in experiments aimed at testing causal hypotheses. Causes can be used to control or manipulate their effects. Causal relations can be deterministic or probabilistic, and the perfectly controlled experiment exemplifies a situation in which association between variables reflect the underlying causal relations.


5.         Prediction

Laboratory experimentation helps to tackle the Duhem-Quine problem constructively, or to draw tight inductive inference from the evidence to a given hypothesis. Much philosophical literature, however, has focused on the wrong aspects of this inductive or evidential relation, by stressing the importance of predictive success. In fact the crucial advantage of the experimental method is that it allows the control of the background assumptions upon which strong inductive inferences rest. This thesis is illustrated using the example of preference reversal experiments.


6.    Elimination

Bayesian confirmation theory stresses the importance of the background, but for the wrong reasons. Scientists’ prior beliefs should not be given too much weight in confirmation theory. What matters is whether the background factors have been controlled by means of an effective experimental design. The experimental method is best characterised as a procedure of eliminative induction, where factors that may potentially disturb the inference from the evidence to a hypothesis are checked one by one, until all sources of error have been controlled for. Experiments on preference reversals provide several examples of this strategy at work.


Part two: Inferences from the experiment


7.    External validity

There is a trade-off between the internal validity of an experimental result (whether a given laboratory phenomenon or mechanism has been correctly identified) and its external validity (whether the results can be generalised from the laboratory to the outside world). External validity is a genuine problem and cannot be solved by metaphysical speculation or methodological stipulation. It is an empirical issue that must be tackled and solved empirically.


8.    Economic engineering

The best example of successful external validity inference is provided by cases of economic engineering, where a piece of the real world is shaped so as to mirror the conditions of a laboratory experiment. I illustrate this procedure using the early auctions of the Federal Communication Commission as an example. The key external validity step is taken by comparing field evidence with experimental evidence and using a so-called ‘no-miracle argument’.


9.      From the laboratory to the outside world

‘Radical localists’ argue that experimental results only apply to laboratory circumstances, or to real-world circumstances that have been engineered so as to resemble the lab. In reality, when experimenters cannot shape the real world so as to fit the laboratory, they can try to shape the laboratory so as to mimic the target system in the real world. Winner’s curse experiments illustrate this principle at work. The inference from experiment to the real world is a special kind of analogical argument, where the inference is strengthened by making sure that the two systems are similar in all relevant (causal) respects.


10.      Experiments as mediators

Models and experiment share several important characteristics. Both are systems that are created to aid scientists in their investigations of a target system. They are ‘mediating tools’, an intermediary step in the process connecting our speculations to the real world. Like models, experiments can be closer to abstract theory or to application. The purpose of an experiment is often to test the robustness of a phenomenon rather than its applicability to a particular real--world situation.


11.      On monetary incentives

The debate on monetary incentives is used as an example to illustrate how philosophical reasoning can help clarifying concrete problems arising from scientific practice. I criticise the view that monetary incentives are a necessary requirement for an adequate economic experiment, because different experiments require different designs. There are no universal recipes in science.