Most studies on marine reserves lack rigor Tripcony. For instance studies of increased fish numbers in a reserve do not usually take in to account the following:
Burdens of evidence and the benefits of marine reserves:
putting Descartes before des horse?
"Detection of recovery of fish density in marine reserves often suffers from lack of rigour in the design of field surveys (Hurlbert 1984; Stewart-Oaten
et al. 1986; Underwood 1990, 1993). As Underwood (1990) pointed out, studies lacking replication cannot be logically interpreted. In the marine reserve context there are many reasons why researchers might have limits on their
sampling designs. However, a critical evaluation of the experimental designs employed by many published studies brought to light the following problems with replication and lack of control sites:
(1) insufficient sample replication (for example only one site sampled inside and outside a reserve, or no control sites sampled at all);
(2) spatial confounding (for example all control sites located only at one end of the reserve, so that comparisons are confounded by unknown location effects);
(3) lack of temporal replication (most studies consist of surveys done at only one time);
(4) lack of replication at the reserve level limiting the generality of results (although in many cases this reflects the number of reserves available); and
(5) non-random placement of reserves, i.e. often reserves are sited to include ‘special’ or unique features, which causes difficulties in selecting valid control sites (this is obviously no fault of the researchers).
To date, there are no well-designed studies that avoid the above problems as well as possessing a time series of ‘before’ and ‘after’ data. However, some might be used as examples of attempts to fulfil good design criteria (Table 1). In addition, the power to detect effects can be affected by the choice of sampling method (Willis et al. 2000), especially when the target species are large carnivores that can exhibit fishing-related behavioural plasticity between sites (Cole 1994; Jennings & Polunin 1995; Kulbicki 1998)
Your description of the spillover effect is also addressed by this paper:
Furthermore, the proliferation of models and reviews has resulted in model
assumptions evolving into accepted paradigms, a case of ‘What everybody says must be true’ (Simpson 1993).
The speculative conclusion that marine reserves will be effective management tools can be obtained from simple behavioural and demographic assumptions. These include:
(1) Where movement range of individuals is small relative to the size of the reserve, those individuals are spatially isolated from fishing mortality, and density within the reserve will be higher than in comparable fished areas.
(2) Elevated densities within the reserve will result in net emigration of biomass from the reserve to fished areas, either by random diffusion (Beverton & Holt 1957) or density-dependent processes (specifically ‘spillover’) (Kramer & Chapman 1999).
(3) Unfished populations of fishes are composed of relatively larger individuals, which have greater fecundity, and hence reserves will act as more productive sources of gametes than comparable fished areas.
The magnitude of the effect may also be speculated on in some cases. For example, if adult fish are sedentary then it could be postulated that density in reserves will increase to carrying capacity (see Hastings & Botsford 1999).
While such speculations are intuitive, they often appear in the literature as logically true assertions.
However, these deceptively reasonable speculations are each dependent on underlying assumptions about behaviour, ecology and the fishery. It is logically true that preventing fishing in particular areas will eliminate direct fishing mortality and stop the destruction of habitat caused by contact fishing gears (Collie et al. 2000). However, it is imprudent to make untested assertions about the primary consequences of reserve protection on fish population dynamics, and then to extrapolate those effects to fishery-level predictions. Typical predictions of fishery enhancement could be invalidated for a number of reasons, including displaced fishing effort around the reserve
boundary (Parrish 1999), recruitment limitation (Doherty & Fowler 1994), self-recruitment rather than larval export (Leis 2002), irreversible changes in species assemblages, and any number of unknown causes due to the underlying complexity of the ecosystem. Without empirical substantiation, predictions of fishery enhancement are deductions based on circumstantial evidence and ancillary information. Furthermore, even if model assumptions are logically correct, it is not sufficient to test only for the existence of reserve effects. Of real relevance is the magnitude of an effect and the certainty (or lack thereof ) that surrounds estimates of it.