INTRODUCTION
ISRP COMMENTS ON THE DRAFT RESEARCH PLAN
Background of the Research Plan
Best Scientific Information
Research Plan Development
Audience
The Meaning of Research (Module 1, sections 2-4)
The Research Proposal Process (Module 1, sections 5-6)
Research Results and Evaluation (Module 1, sections 7-11)
Program Accountability
Module 2: Short-Term and Long-Term Research Agendas
Short-term research agenda
Long-term research agenda
CONCLUSION
INTRODUCTION
In a January 22, 2002 memo, Council staff requested that the ISRP review the Draft Research Plan for Fish and Wildlife in the Columbia River Basin (1/22/02 version). The memo included several questions directed to the ISRP review. These questions are answered indirectly in the following review comments, which recommend a substantial reorganization of the draft plan.
The Northwest Power Planning Council's 2000 Fish and Wildlife Program (FWP) calls for the establishment of a basin-wide research plan that will identify key uncertainties and research required for their resolution. The plan should identify major research topics, including ocean research, and establish priorities for research funding. The program calls for research strategies that will: 1) identify and resolve key uncertainties for the program; 2) monitor, evaluate, and apply results; and 3) make information from the FWP readily available.
The research plan is to be coordinated with the mainstem and subbasin plans and should contain mechanisms to ensure public access to research results. Additionally, the program calls for projects to review the current state of the science in key research areas. It identifies the Independent Scientific Advisory Board as the body charged with developing a series of reports to survey past research and summarize the state of the science in key areas.
The draft document "Research Plan for Fish and Wildlife in the Columbia River Basin" is the response to the FWP's call for a research plan.
ISRP COMMENTS ON THE DRAFT RESEARCH PLAN
The draft research plan is a laudable attempt to provide useful information to a wide variety of audiences. Some audiences do not understand the nature of research, and for this audience the draft attempts to define research. Other audiences do not understand the workings of Council and BPA research funding or the components of an acceptable proposal. Still other audiences want assurance that research expenditures are cost-effective. The current draft attempts to fulfill each of these diverse roles, yet with a brevity that befits an overview document. However, to provide useful information to each of the audiences, the draft plan should be substantially revised. The ISRP recommends several revisions in this review document.
Background of the Research Plan
The document should make it clear that the call for a research plan originates in the FWP rather than the Northwest Power Planning and Conservation Act. The draft document contains a quote from the Act which implies that it charges the Council to develop a research plan, when in fact it charges the Council with preparation of a fish and wildlife program "...based, on and supported by, the best available scientific knowledge." Without providing some background information on the efforts made by the Council to identify and examine uncertainties and to provide consistent scientific standards for FWP projects through structures such as the Technical Working Groups (TWG), Scientific Review Group (SRG), Independent Scientific Group (ISG), Independent Scientific Advisory Board (ISAB), and ISRP, it is misleading to proceed with statements that read:
"Every year the Council observes this mandate by recommending the funding of numerous research projects as vehicles to gather such necessary scientific knowledge. This large — and expensive — family of projects, however, is not implemented consistently and lacks administrative organization and scientific standards. These deficiencies compromise the credibility and accountability of the research effort contemplated under the Program."
Best Scientific Information
In developing its FWP, the Council has made an effort to proceed on the basis of the "best available scientific information." Initially this meant assuming that the agencies and tribes were in possession of the best available scientific information, but the Council encountered disagreement among the various parties as to what actions should be associated with that information.
The Council has dealt with this problem in different ways as the FWP has evolved. The first approach was the formation of Technical Working Groups (TWGs), consisting of representatives of the agencies and tribes, and some third parties. The TWGs were charged with summarizing existing information and identifying FWP research needs in areas such as hatcheries, fish disease and habitat. The Council later addressed the need to satisfy the "best available scientific information" provisions of the Act through the establishment of the Scientific Review Group (SRG), Independent Science Group (ISG), Independent Scientific Advisory Board (ISAB), Independent Scientific Review Panel (ISRP) and Independent Economic Advisory Board (IEAB) as independent review bodies.
Research Plan Development
Because the draft provides a limited review of the Council's actions with respect to development of a research plan, it contains a number of incorrect assumptions and a critical attitude toward the Council's FWP that is unwarranted.
Specifically, the citation to the Fish and Wildlife Program in the paragraph entitled "significance" should be enlarged to prevent misinterpretation. The referenced section of the FWP established the Council's Fish Committee and spelled out some of that committee's responsibilities. Taken in context, the appointment of Technical Working Groups and their development of five-year research plans, which included assessments of past research and identification of data needs, derived from this section of the FWP.
Additionally, Section 206 of the 1987 FWP contains what could appropriately be called the Council's research plan for salmon and steelhead. Section 206 called upon BPA to fund research in specified areas of emphasis over the ensuing five years. It also requested that BPA fund the Technical Working Groups, whose responsibilities were to develop the five-year work plans in those areas of emphasis. The work plans were to be approved by the Council, thus becoming Council plans.
These efforts should receive more detailed acknowledgement in the research plan while retaining the central point that despite the efforts of the fish and wildlife agencies, tribes, TWGs, the ISG, ISAB, and ISRP, after 20 years the Council does not have a stand-alone research plan that identifies areas of uncertainty and needed research.
Audience
The draft research plan covers a number of issues related to FWP research that are targeted at different audiences. The draft suffers from the attempt to communicate with too wide a variety of audiences, each of which has different degrees of familiarity with research in general and the Fish and Wildlife Program in particular. Additionally, the comprehensive guidance on research administration, practice, and assimilation provided in this document either duplicates information provided elsewhere or is so general as to not be operationally useful.
The communication objectives and target audiences of the document need more specific definition. The ISRP recommends that the range of audiences be defined, that communication objectives for each audience be clarified, and that separate documents be developed for each audience.
If there is a need to provide new Council members with an overview of research and its conduct under the FWP, a short white paper summarizing some of the material provided in Module 1 (sections 1-6) could be prepared. An emphasis should be placed on short, direct and clear descriptions that provide the needed information within the operational context of the FWP.
Another example of a useful paper would be a short semi-technical description of the research selection and funding process in the FWP. That would be targeted to new Council members, new advisors, and new investigators participating in the provincial reviews. Details such as the contents of a proposal should not be included in this document, however, as this and other solicitation details are available on the web.
Similarly, there could be a need for a paper on the distinction between monitoring and evaluation and research. Such a paper could also describe the continuum between basic and applied research. However, excessive focus on definitions should be avoided, as it can generate unnecessary debate that diverts attention from the need for research that answers questions relevant to the FWP. Research relevant to the FWP is research that addresses basin-wide uncertainties.
The Meaning of Research (Module 1, sections 2-4)
The research plan should not attempt to provide guidance for research performance and administration. There should be no need to define research or the scientific method. This relates to the need to define specific audiences and target short pieces directly to those audiences.
The plan's definition of research as hypothesis testing and cause-effect definition is too restrictive and does not reflect current practice. For example this definition would exclude studies involving the collection of data to better quantify important known relationships, or improvements in statistical methods that could be of value in analyzing data from the Columbia River Basin. Estimation, pattern recognition, observation and categorization are equally valid types of research (taxonomy, geomorphology, landscape/GIS work, etc.). A dictionary definition of research is "study and investigation, especially to discover new facts."
The Research Proposal Process (Module 1, sections 5-6)
Information in sections 5-6 is already provided in the solicitation for proposals. As the document states, proposals must conform to the instructions provided for each specific funding category, so they need not be repeated here.
The bullets in section 6 (proposal review, selection, and implementation) are but a few of many tips that might be given. They are not really "guiding principles."
Research Results and Evaluation (Module 1, sections 7-11)
These sections propose methods that would introduce closure and accountability into projects by having back-end reviews of completed projects. The ISRP strongly supports the objective of achieving more accountability for project accomplishments, but questions whether annual workshops and creation of a new science advisory group are an effective way to accomplish this accountability. The workshops may duplicate other project presentations, and are likely to be able to accomplish only a superficial reporting of project results.
Alternative approaches that could be considered include:
- Broader scale analysis that applies information from several projects to address a particular question. For example, a separate project to conduct an Ecosystem Diagnosis and Treatment (EDT) analysis using data generated from individual projects.
- Synthesis reports of work completed in a particular area, such as the Giorgi report, Mainstem Passage Strategies in the Columbia River System: Transportation, Spill, and Flow Augmentation.
- Workshops structured around single topics driven by critical questions, e.g. transportation effects, and projects synthesized to address that topic. Workshops and symposia tied to topics are a good way to bring the region up to date on specific subjects. All workshops should have published proceedings so that workshop material is circulated throughout the region.
- Expanded provincial review presentations.
- Publication of an annual hard copy index of project reports and abstracts citing web locations. In the past, annual project reports were distributed in hard copy form but are now only available in electronic form. While this approach generates cost-savings, it also creates costs in that many people are not aware of report availability on the web.
- The State of the Science Review, an ISAB function, enhanced by the contribution of joint ISAB/ISRP membership.
The ISRP supports the review of completed projects and agrees that some entity needs to ensure that raw data produced during a project are documented and made available and that project reports are distributed.
However, the proposed creation of a new committee (SRC) may not be the most cost-effective way to accomplish post-project review. The SRC would be given significant responsibilities to compile information on each completed research project and perform an evaluation. The formation of the SRC would add another layer of bureaucracy that is likely to place significant demands on fiscal resources. Other mechanisms are already in place to accomplish the functions described for the SRC. The project contracting process should ensure accountability of projects to their stated objectives as well as the dissemination of information. Symposia can serve the function of presenting and synthesizing research results beyond established electronic reporting practices.
Program Accountability
Other issues of accountability apply more generally beyond individual projects to the entire FWP.
The ISRP supports the development of mechanisms to ensure that projects reach completion. Presently, many projects do not end at the completion of the performance period but instead continue to add new objectives to extend the performance period. The ISRP recommends that a mechanism such as a required specification of an end date for proposal objectives and tasks be introduced to force projects to end at their intended time.
The mechanisms to achieve accountability and closure may not be the same for all projects. "Infrastructure" projects, for example, may be an exception to the requirement for an end date. Particular needs could be addressed through separate project categories.
Module 2: Short-Term and Long-Term Research Agendas
Short-term research agenda
This section lists research projects that comprise Council's current research portfolio. The existing table of projects is valuable in its listing of the number and funding of projects under various categories of research. However, neither the number of projects or money spent in a given area indicates whether the relevant questions are being answered or the proportion of program spending dedicated to that area. The project list would be more informative if it were accompanied by a pie chart showing funding amounts in each of the major research categories (e.g. hatcheries, habitat, passage). A pie chart, while not indicating the quality of research conducted under various categories, does provide an overview of the distribution of FWP funding across categories.
A pie chart would also reveal areas not funded by the FWP. For example, hydropower affects many aspects of the riverine environment that could be the subject of research leading to better mitigation. We try to help fish passage, but we know very little about how fish migrate. What cues are they using? Are there cues that would aid safe passage opportunities?
Many research planning workshops have been conducted but few of their recommendations are included in the FWP. Targeted solicitations were developed as a remedy to this problem, but that approach was discontinued. Many research areas outlined in the ISG's Return to the River (1996) are still not on the research agenda.
There is little point in developing a short-term research agenda that is no more than a list of ongoing research that will soon be replaced by subbasin plans. The primary focus of the Council's research plan should be larger-scale analyses that are based on findings of individual projects. The plan should serve as a basinwide framework that guides the content of research in the subbasin plans.
Long-term research agenda
The development of a long-term research agenda is a major undertaking. Developing an agenda requires serious thought about its content, objectives, and its potential contribution to adaptive management. A long-term research agenda should be built around the need to address key uncertainties.
There are both top-down and bottom-up approaches to the development of the agenda. The design objective is to ensure that both approaches meet in the middle with an accepted plan. The Subbasin plans embody the bottom-up approach, and will contain input from a wide range of stakeholders. The FWP provides the top-down approach to the identification of program-wide research needs and applications. The FWP provides upper-level guidance for long-term research and can guide the Council in evaluating and choosing among large-scale initiatives. Sub-basin planners will look to the FWP for guidance on the structure of plans and the types of research that should be pursued.
The 2000 FWP states that a meeting of fish and wildlife agencies, tribes and hydrosystem operating agencies should be convened regularly to identify key uncertainties about the operation of the hydrosystem and associated mainstem mitigation activities. This list of key uncertainties will provide the basis for targeted calls for proposals. The 2000 FWP also tasks the ISAB with advising on priorities for conservation and recovery efforts, including research, monitoring and evaluation within NMFS recovery plans. Additionally, the Action Agencies (BPA, U.S. Army Corps of Engineers, U.S. Bureau of Reclamation) call, in their Annual Implementation Work Plan (AIWP), for a 2002 workshop to identify research needs related to habitat sampling and surveys.
The ISAB, ISRP and IEAB could be useful resources to help the Program identify key uncertainties as well as basin-wide questions that could be answered through a set of sample experiments across subbasins. The ISRP recently identified a need for the Council to make more effective use of the IEAB in comparing proposals for their relative costs and benefits to fish and wildlife.
One mechanism to develop combined ISAB, ISRP and IEAB identification and advice on critical uncertainties and major areas of emphasis would be to conduct a retreat for members of these three bodies. Participants could be assigned to prepare for the workshop by reviewing work accomplished and recommendations made in particular subject areas. The table provided in the draft research plan and Return to the River (2000) are good starting points for the discussion.
The draft long-term research agenda has a tendency to downplay research that has general benefit. Temperature effects, for example, are not mentioned anywhere, but thermal requirements (including latent effects of non-lethal high-temperature exposures) affect fish populations everywhere. Valid and well-researched temperature criteria are needed in every province. Research is also needed on nutrient transport, habitat preference/selection, and methods such as DNA techniques for detection of pathogens in water. The draft could contain more references to this sort of research.
CONCLUSION
The Fish and Wildlife Program has a clear need for a long-term basinwide research plan. But rather than defining research and compiling a list of projects that may meet that definition, the plan should instead include a prioritized list of uncertainties, mechanisms for funding research to address the uncertainties (e.g. RFPs), mechanisms for identifying when a topic has been adequately researched, and structures to identify new uncertainties in an iterative manner.
ISRP reviews have highlighted the need for a basinwide research plan that identifies priority RFPs. The purpose of the plan would be to evaluate the effectiveness of major FWP initiatives, to assist in choosing among areas of research, to identify needed shifts in research emphasis, and to identify new areas of research. The FWP needs a research plan that looks at overarching questions and assists in making decisions about relative importance among projects.
The most effective way for the Council to proceed with its research plan is incrementally, as it has done. The five-year interval specified in the 1987 FWP may be the most appropriate for Council purposes, since the Council is committed to revising the FWP periodically. The areas of emphasis in the 1987 FWP as well as specific questions raised by the Technical Work Groups need to be updated to account for ensuing experience.
For development of a useful research plan, the Council should identify key research areas such as the hatchery initiative and its associated supplementation issue. Other areas already identified as important include mainstem passage and habitat improvement. Research questions related to these and other areas can be identified through a number of alternative approaches: ISAB reviews; combined ISAB/ISRP/IEAB assessments; specific projects such as those done by Pratt and Chapman, PATH, Giorgi et al., and the ISG; and proposals from outside sources.
In conclusion, the ISRP recommends that separate documents be prepared (or referenced on the web) for specific audiences on processes related to FWP research. The research plan could better be focused on the information uncertainties affecting the mission of the FWP and how, from a top-down perspective, these uncertainties might be addressed by funded research projects over the next 5 years. Specific strategies for integrating with the bottom-up Subbasin Plans should be laid out in order to ensure compatibility.
Report by: |
Charles C. Coutant
Daniel Goodman
Susan S. Hanna | Nancy Huntly
Lyman McDonald
Brian Riddell | William Smoker
Richard R. Whitney
Richard N. Williams |
