Inadequacies in Battelle Research

by Carl  Shipley
June 6, 2011

Subject: Comments on SMP Process – specifically inadequacies in Battelle research

Honorable Commissioners and SMP Coordinator,

I have attended many of the SMP meetings and have tried to research the shoreline science that county planners appear to be using in developing regulations. It is clear that much of the rationale for the Kitsap SMP process comes from a single study of Kitsap beaches carried out by the Battelle Research Institute.

This study, the “East Kitsap County Habitat Assessment and Restoration Framework,” claimed to find a weak negative relationship between shoreline development and nearshore environmental health. The full report is available online (a link is included below).

I have read this study carefully, as well at the Battelle study of Bainbridge Island beaches. I have a Ph.D. from UCLA in Experimental Psychology for which I took numerous classes in statistical analysis and experimental design. I can assure you that the experimental design used in the Battelle study is totally inadequate to support the claim that proposed stressors such as bulkheads, docks, stairs to the beach, and the like have clear negative effects on nearshore environmental health.

This claim is made on page ii of the Battelle East Kitsap study: “The validation indicates that improving processes at the site and drift cell scale will improve ecological functions.”  and on page 40, which makes the very strong claim that:  “moving a stressor score from 0.4 to 0.2 would predictably result in improvement in functionality from the existing low-moderate score to high moderate or high functionality score.”

These claims are clearly guiding much of the thinking of county planners but a careful reading of the Battelle report reveals that the claims are based on scientific methodology that is so flawed it would not be accepted in even an introductory college course in statistics or experimental design.

The Battelle East Kitsap validation study is seriously flawed for at least the following reasons:

  1. An inadequate sample size was used. Ultimately, a sample of 14 beaches (termed NAUs in the report – Nearshore Assessment Units) was used for the functional validation study of the effect of proposed stressors on environmental health. This sample was taken from a population of over 1000 beach areas in the Kitsap assessment (this figure includes the Battelle research on both East and West Kitsap beaches since the study of West Kitsap beaches did not contain any validation work and simply referenced the East Kitsap study).
  2. Sample size was really smaller than 14 because the validation study is intended to document the relationship between stressors and health for all beach types in the county – that is to say, for sites as different as high bluffs, rocky shores, salt marshes, estuaries, and broad sandy beaches. The study acknowledges that physical processes affecting environmental health differ for these different each types. Yet, for most beach types (termed geomorphologies in the study), only a single example was included in the validation study and for some beach types, no example was included. Only one beach type, the “transport” beach, was represented by more than one sample and this type was represented by only 7 cases.
  3. Beaches selected for inclusion in the validation study were not randomly sampled — in fact, the word “random” does not appear in the report. This is an inexcusable breach of the most basic tenets of experimental design. When people rought this issue up to R.Thom, the senior author on the study, he claimed that the beaches were selected using “stratified random” methodology. This is simply not true. Stratified random sampling means the experimental design requires that some subcategories of a variable be sampled in specific amounts – for example, in a study of voters, a stratified design might specify that equal numbers of men and women voters would be sampled. However, in this kind of design samples are still taken randomly from within a stratified subcategory. This was not done in the East Kitsap validation study. Only one beach type was sampled more than once for the validation study and the beaches in this small group of 7 study areas were clearly not chosen randomly – many of them are right next to each other near Point No Point and all are in North Kitsap.
  4. The study contains no formal statistical analysis of the data – no hypothesis testing, even though the authors claim to have confirmed a hypothesis. By hypothesis testing I mean the standard experimental practice of quantifying the degree to which a study’s results could be expected to occur purely by chance. The reason the authors of the Battelle study did not use this standard approach is that they did not use standard scientific methodology – random sampling, adequate sample size, any kind of standard statistical analysis.
  5. Establishing a functional relationship of the type asserted in the statement (p. 40) that a change in stressor score will “predictably result in improvement in functionality” cannot be done on the basis of the correlational approach used in the study. The kind of functional relationship implied by the assertion of “predictable” change can only be established by experimentation in which confounding variables are controlled or eliminated and systematic changes in one variable are shown to reliably produce changes in a second variable.
  6. The approach of the east Kitsap validation study was to look at stressors to see if they were related to measures of environmental health. This is similar to the approach used in the Battelle Bainbridge study, which also used a number of measures of environmental health in an attempt to determine if health was affected by proposed “stressors” such as bulkheads. One would assume that in the east Kitsap validation, the Battelle team would use the same standard measures of environmental health that were used in the Battelle Bainbridge assessment – but this was not the case.

On Kitsap, Battelle omitted most of the measures of health used in the Bainbridge study including standard metrics of obvious value, such as the presence or absence of fish spawning areas. While the authors state that new measures they used in the East Kitsap assessment are standard measures of health this does not appear to be the case. In fact some of these new measures of health appear to actually be measures of development and, thus, are clearly faulty.

For example, there is a measure of environmental health called “vegetation.” The measure is a composite of scoring 4 things:

  1. the percent of the NAU with vegetation in the backshore (backshore being defined as area within 75 meters of the shoreline),
  2. whether the start of vegetation is within 10 meters of the shoreline,
  3. the composition of the vegetation (low scores for landscaping or grass),
  4. estimate of the percent of overhanging vegetation on the shoreline.

Notice that part of this measure is based on whether there is a lawn or other landscaping anywhere close to the shore of the NAU. If so, the NAU gets a score of one (out of five where five is best) for environmental health (see the score sheet for this is in Appendix F – page F-5). What this means is that if there is landscaping on the shoreline, the measuring system will record harm to the nearshore. What this works out to is that development equals harm to nearshore environmental health by definition.

If you measure health in this way, why do a study at all? Why not just say development equals harm to the environment and be done with it? Well, actually, that is pretty much what the Battelle authors did. The county cannot accept this as real science. Similarly, if vegetation is cleared back from the shoreline on a percentage of the NAU (also a proxy measurement of the presence of development) then the NAU gets a low score for environmental health. These indicators don’t have anything to do with nearshore – i.e. in the water – environmental health, unless, of course, one takes as a given the idea that landscaping near the shoreline harms nearshore health.

It is not the case that these flawed measures are minor and unimportant oversights. In fact, the study doesn’t use many measures of environmental health for most beach types so these flawed measures are important. For the main beach type, a “sedimentation, depositional, or transport” beach, the category that is sampled 7 times in the study of 14 beaches, there are only 4 measures of environmental health”

  1. presence of driftwood on the shore,
  2. eelgrass beds in the water,
  3. vegetation near the shoreline and
  4. beach wrack on the shore.

Of these, only eelgrass is a measure that was used in the Bainbridge study. Clearly, if even one of four measures of environmental health is seriously flawed it’s a major issue for the assessment.

I do not mean to be overly harsh in saying this but, in all honesty, as someone who had taught basic experimental design, if an undergraduate student at UCLA proposed this kind of research as acceptable science he or she would fail and if a graduate student proposed this kind of methodology he or she would be asked to leave the program. For the county to use this kind of mock science as a basis of establishing important public policy is clearly wrong.

I realize that you may not bring formal training in experimental design to your role as a commissioner but one does not have to have formal training to understand that random sampling is a foundation of experimental research or to nderstand that tiny samples and unusual metrics, of the kind used in the Kitsap beach research, do not provide an adequate basis for developing important shoreline regulations. I believe the environmental movement takes it as obvious that structures such as small residential bulkheads or small docks must be harmful to the environment.

But, in fact, there is little evidence that this is true, especially for structures in relatively sheltered waters such as we have in many parts of Kitsap County. A very detailed study of Bainbridge Island beaches failed to find clear evidence that bulkheads had any negative effect on common measures of nearshore environmental health such as eelgrass beds and fish spawning areas.

Every year tens of thousands of salmon return to Chico creek, swimming past some of the most highly developed shoreline in the county in the process. The county has done excellent work in upgrading habitat in Chico creek and the results are highly commendable. The county should be putting scarce resources into that kind of effort instead of pouring hundreds of thousands of dollars – perhaps millions – into regulations of highly questionable value based on science that is clearly flawed.

Sincerely,
Carl Shipley
17434 Nordic Cove Ln NW
Poulsbo, Wa.

Links to the studies

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s