At the request of Council, the ISRP reviewed the proposed statement of work to develop Protocols for the Inventory and Monitoring of Fish, Wildlife, and their Habitats in the Pacific Northwest by David Johnson, Washington Department of Fish and Wildlife.
Background
The need to fill data gaps and develop a better system for collecting data and making them available was well documented in the ISRP report, "Review of Databases Funded through the Columbia River Basin Fish and Wildlife Program". The ISRP has frequently commented that the process of monitoring and evaluation and the resulting data are inadequate for many projects. If data are collected, they are often not reported or not collected in a systematic manner. Where data are available, it is often uneven or poorly documented. ISRP comments on monitoring and evaluation in the recent report on the Upper and Middle Snake, Columbia Cascade, and other provinces reinforce these observations (ISRP 2002-2).
The Council curtailed funding for PATH based on ISRP review comments, but reserved $900,000 from the discontinued PATH project to support efforts to develop better data and data systems for the region. The development of a basinwide research plan is already underway and has been submitted to the ISRP for review.
Other initiatives are under various stages of development. One of these initiatives requires the Council to work with the region to "establish guidelines appropriate for the collection and reporting of data in the Columbia River Basin." This is the purpose of the David Johnson proposal. Complementary to this, Council also initiated a joint effort with NMFS to work cooperatively to develop standardized measurement and reporting protocols and design and implement a basinwide data management system based on ISRP reports and recommendations. Recently, the Council and NMFS agreed to an MOA regarding a collaborative effort to develop a single data system. One of the first joint efforts will focus on adopting standard measurement protocols for both environmental and biological variables.
ISRP Review Comments
The ISRP is impressed with the need for this project and with the potential that the Johnson proposal may have toward developing better data and data systems for the region. Johnson’s work to this point is well done and appears very promising for the region (e.g., Johnson et al. 2001, “Inventory and Monitoring of Salmon Habitat in the Pacific Northwest”. Standardization of data collection and sharing methods will feed logically (and profitably) into the forthcoming Subbasin Planning efforts and the next round of provincial reviews.
The review material for the ISRP consisted of a 3-page Briefing Paper and Statement of Work (SOW) by David Johnson building on his 2001 inventory and monitoring report. This project proposes to provide the region with two manuals that include data collection protocols for counting fish and their habitats. The author plans to work with CBFWA to arrive at consensus for a final subset of protocols for both habitat assessment and fish counting for consistently gathering data across the Pacific Northwest. After public and scientific review, a final set of recommended protocols would be submitted to the Council for adoption. Final products would include a 3-ring binder of complete protocols and electronic versions to provide guidance and ready reference to meta-data. The author has plans to eventually include wildlife habitat, wildlife counting, and estuary-near shore inventory and monitoring.
Unfortunately, in its present form, the Statement of Work (SOW) from Johnson does not provide adequate technical information and detail to allow the ISRP to evaluate the soundness of the proposed work. The SOW lacks appropriate detail on methods, descriptions of how various sampling protocols would be evaluated, explanations of methods or rationales for how certain protocols would come to be recommended over others (likely not by consensus, as the SOW suggests), and any description of the importance of sampling design and methods, such as probabilistic sampling (see Sampling Design comments below).
Perhaps the lack of detail in the presentation is a function of the abbreviated SOW form. However, basing funding decisions on a Statement of Work form seems premature. This worthwhile idea should be written as a full technical proposal using the standard Council-BPA form, which will allow scientific and technical review. Statements-of-Work are the work-plans that follow the funding of a proposal.
Issue of Sampling Design
An additional missing piece in this proposal is review, evaluation, and recommendations of top-down probabilistic sampling procedures that in conjunction with common data collection procedures can provide statistical inferences at different scales: the Columbia Basin, Subbasins (e.g., the Salmon, John Day, or Methow Subbasins), watersheds (e.g., Beaver Creek in the Methow), and finally individual project study areas (e.g., the Oxbow Ranch acquisition in the John Day). The essential features of such a sampling procedure are:
- There is a minimum sampling effort using the same site selection and data collection procedures that allow statistical inferences to the entire (or major parts of the) Columbia Basin.
- The sampling procedures can be intensified to provide statistical inferences to Subbasins (e.g., the John Day Subbasin).
- The sampling procedures can be further intensified to provide statistical inferences to small watersheds (e.g., Beaver Creek in the Methow Subbasin), and finally;
- The sampling procedures can be further intensified to provide statistical inferences to individual properties (e.g. the Oxbow Ranch in the John Day Subbasin).
If a sampling procedure has the above properties then data collected at lower levels can be pooled to improve inferences at the higher levels with better precision. Other desirable properties of an overall sampling procedure include: 1) the ability to rotate sample sites out and add new ones as time passes and they “wear out”, i.e. are not longer representative of the intended universe, 2) there is little stratification and no strata boundaries will change with time, etc.
There are two models that should be reviewed: 1) the so called “Oregon Plan” for estimation of juvenile coho abundance (i.e., the EMAP stream sampling procedures with primary contacts Kelly Moore, ODFW, and Don Stevens, OSU, and references on the website), and 2) the Northern California juvenile coho estimation methods developed by Overton and McDonald (1998) and McDonald and Nielson (2001). There are similar terrestrial sampling procedures, e.g., the National Resources Inventory and Forest Health Monitoring.
Importance of the Review Process
While the ISRP is quite supportive of efforts to develop better data and data systems for the region, we are concerned that the proposed project by Johnson has not benefited from the independent scientific scrutiny that occurs routinely and necessarily for other projects in the basin. The 1996 amendment to the Power Act, describes the role of the ISRP in project review and outlines specific review criteria for projects. The amended Act directs the ISRP to review projects in the context of the Council’s program and in regard to whether they:
- are based on sound science principles;
- benefit fish and wildlife;
- have clearly defined objectives and outcomes; and
- have provisions for monitoring and evaluation of results.
It is especially important that a project, such as this one, that is intended to set broad protocols for the region be clearly and fully presented and reviewed before being implemented. There are additional reasons to support development of a full technical proposal and review by the ISRP prior to project implementation. First, and most simply, is that the project might benefit and be improved by comments from the reviewers. Second, the review process and the public scrutiny that would occur might broaden the constituency and support for the project.
Notwithstanding the above comments about the review process, the ISRP understands that the work proposed by this project represents a critical need in the basin, and as such, support for the general goal of developing better data and data systems for the region may warrant an accelerated approach outside of the provincial review process.
One possible avenue for funding and review of the project outside the provincial review process, that has been used successfully before, is through an RFP (Request for Proposal) process. Previously, and with good result, the Council has solicited work on mainstem habitats and on chinook and steelhead population structure via RFP’s. Johnson’s 2001 report Inventory and Monitoring of Salmon Habitat in the Pacific Northwest might be used as a Statement of Qualifications to justify sole source solicitation for the RFP to Johnson. If this avenue were pursued, it would demand that a full proposal be prepared for ISRP review and comment. After the proposal was reviewed and a response generated, then a Statement of Work would be prepared, and contracting finalized. A second element of the RFP approach that is attractive is that the final product could also receive ISRP review to ensure that the terms of the RFP were realized.
Broader Inferences
Another element deserving consideration as part of the data standardization effort, is the long-standing ISRP concern of how activities such as this proposed project fit into broad-scale overviews and assessments of overall recovery efforts. Without slowing or compromising the effort to standardize data collection and distribution protocols (as represented by this proposal), serious attention also needs to be paid to systematically determining, as an entire package,
- What are the information needs for the high priority decisions in the basin?
- What kinds of data need to be collected to address these information needs?
- What spatial and temporal resolution is required to address these information needs?
- What level of precision in the estimates is required to address these data needs?
- How many samples, when and where, must be taken to deliver the required precision at the correct spatial and temporal scale?
- How should sampling sites be chosen so that the data are representative of whatever they are supposed to represent in space and time?
- What measurement methods should be used so that everyone's measurements will be compatible when the data are analyzed?
- Where will the data be stored and how can the data and metadata be accessed?
Conclusion
In summary, the ISRP is strongly in favor of the Council's efforts to initiate and advance the development of standardized protocol. However, the statement of work submitted for review does not provide adequate technical information and detail to allow the ISRP to evaluate the soundness of the proposed work. We believe that the Council should go through a more complete proposal review process with the likely result a better project with broader constituency and support. Due to the importance of standardization, an accelerated solicitation and review route outside of the provincial review process seems justified.
References
ISRP. 2002. Preliminary Review of Fiscal Year 2003 Proposals for the Upper and Middle Snake, Columbia Cascade, and Lower Columbia and Estuary Provinces. March 1, 2002. ISRP Report 2002-2 to the Northwest Power Planning Council, Portland, Oregon.
McDonald, T.L. and R. Nielson. 2001. Statistical power of abundance surveys to estimate status and detect trend in juvenile coho populations. Western EcoSystems Technology, Inc., 2002 Central Ave, Cheyenne, WY 82001. Draft final report prepared for Humboldt State University Foundation’s Forest Science Project, 1 Harpst St., Arcata, CA 95521.
Overton, S.W. and T.L. McDonald. 1998. Regional Estimation of juvenile coho abundance in streams. Final Report, West Technical Report #98-5, 2003 Central Ave., Cheyenne, WY 82001.
