At the Council’s January 21st, 2005 request, the ISRP and ISAB reviewed the "All-H Analyzer" (AHA) model that Council staff is proposing to use as part of a larger exercise in the Fish and Wildlife Program to establish draft numerical objectives for anadromous fishes, including natural returns, hatchery escapement, and harvest at the subbasin, province, and basin levels. Subbasin Plans were recently found to be deficient in the integration of hatchery and natural production with habitat, hydro, and harvest goals for anadromous fishes.[1] A goal of the use of the AHA model in planning is to facilitate consideration of the balance of hatchery and natural production in relation to habitat actions, out-of-subbasin harvest, and hydrosystem constraints.The Council asked five general and three supplemental review questions concerning the AHA. To help the ISRP and ISAB answer these questions, lead scientists from Puget Sound’s Hatchery Scientific Review Group (HSRG) presented the model to us, explained how the model was used in Puget Sound for informing hatchery management, and described the development of the fitness equation in the model. Although appreciated and informative, the presentation raised many unanswered questions about the model’s structure and the mechanics of its use. In addition to the presentation, the AHA model itself was made available, and several reviewers explored its behavior. The ISRP&AB also reviewed the available background documentation pertaining to the model, including HSRG technical papers and some of the scientific literature that the fitness equation is based on, in particular a recent paper by ISAB ex-officio member Mike Ford (Ford. 2000. Cons. Biol. 16: 815-825). It is important to note, however, that the AHA model is still in development and much of the background material needed for review and application is NOT available. In other words, this ISRP/ISAB "review" of the AHA is essentially a review of the intentions or potential in the development and projected use of the model and not a review of the reliability or accuracy of the model itself.
Before addressing the questions, the ISRP&AB emphasize that there is a clear need for quantitative analysis, including disciplined use of analytical and exploratory modeling, to improve fish and wildlife management in the Columbia River Basin, particularly the integration of natural and hatchery production with habitat actions at the subbasin level. We strongly agree with the Council staff's observation that a major problem within the Columbia Basin is the lack of clearly articulated objectives integrated across the four Hs at the subbasin, province, and basin levels. Without these objectives, it is difficult to prioritize project implementation and monitoring activities. A key to developing these objectives is a comprehensive, integrated analysis of habitat, hatchery, hydrosystem and harvest actions. In our review of subbasin plans, we specifically noted that most plans did not adequately describe how hatchery programs would be integrated with existing natural production, habitat improvements, and future rehabilitation activities. Many plans also lacked stated measurable objectives for natural returns, hatchery escapement, and harvest. An attractive feature of the AHA model is that it is one of the first software products developed explicitly for planning purposes in the region that attempts to incorporate some representation of a relationship between hatchery and natural production and fitness of the wild stock.
The vocabulary referring to the AHA model as a "tool" (see the wording of Question 4 specifically) merits special attention, because of the unrealistic expectations that this labeling might convey. This language derives from "decision support tools," developed generally as "expert systems." These are artificial intelligence products, intended to capture existing hard scientific information and expert opinion in a technical area and make these accessible, via a "user friendly" interface, to users who themselves are not expert in that technical area. In practice, the goal of providing an interface that is truly easy to use is seldom attained, so a cadre of technicians who are adept with the interface often are needed to serve as facilitators between the software product and its intended audience. But even so, the actual deployment of the software in real applications is generally in the hands of a user team that is not expert in the underlying scientific area that is supposed to be supplied by the software itself. The quality of the predictions (or recommendations) generated by the model will depend on: the quality and quantity of the real data (measurements) input to it by the user, the accuracy of the user-supplied input "guesstimates" (numbers not based formally on statistical evaluation of actual data), and the hard scientific information and expert opinion encapsulated in the software.
In the present instance — predicting salmon productivity in the presence of wild/hatchery interactions — the available library of actual data is very sparse, case-specific data that users can supply also will be very sparse, the predictive power of the available pertinent hard science is limited by data gaps, and there is not a consensus of expert opinion. The ISAB has reviewed this state of the science in our supplementation review.[2] For these reasons, an expert system predicting salmon productivity in the presence of wild/hatchery interactions will have unpredictable performance, and probably low reliability. At best, such a system will offer a useful way to organize assumptions and quantify the implications of adopting those assumptions, provided the tool carefully documents its own assumptions, as well as the user-supplied inputs, along with its output of predictions. If the tool is adequately documented, and adequately documents inputs as part of its output, the proper use of the tool is to generate hypothesesthat should be tested, rather than accepted at face value. The ISRP and ISAB have explained this perspective in comments on another expert system, EDT, during the course of the model synthesis review.[3] The real substantive benefits from use of a properly documented tool to generate hypotheses will not accrue until the results are obtained from experiments that are correctly designed to deliver the needed resolution. The design and implementation of experiments of this sort, and on this scale, have historically proven to be a very difficult challenge within the Columbia River Basin. The ISAB has already commented, in its supplementation review, on the insufficiency of the existing record of experimentation with supplementation, despite the widespread institutional acknowledgement that supplementation is experimental.
Review Summary
The AHA model attempts to provide a convenient interface to an underlying model for evaluating the "expected" consequences of a hatchery program, given certain assumptions and quantitative relationships (e.g., habitat capacity), but, unfortunately, those assumptions and equations are incompletely documented. Also, some reasonable scoping of the model's sensitivities is needed.
The ISRP and ISAB encourage the Council and region to pursue the effort to set objectives and integrate across the Hs, but caution against using the AHA model or any other unvalidated single model to generate specific objectives, numerical or otherwise, or to propose recovery goals for anadromous fish. As we understand the AHA model, it was designed to evaluate how well a particular hatchery program in Puget Sound or Coastal Washington, conforms to the HSRG's "integrated program" criteria. It is essentially a deterministic model that provides an estimate of the expected long-term equilibrium numbers and ratios of hatchery and wild fish, given a certain set of incompletely documented assumptions. It would be inappropriate to use this model to decide what natural and hatchery production objectives and recovery criteria should be in the first place. It is not a good idea to encourage people to apply an undocumented model, where details of the structure or operations of the model remain hidden from the user. Acceptance of the meaning of an unknown process is more an act of faith than a sound application of science.
Recommendation. The AHA model should not be used to aid in the development of draft numerical objectives for anadromous fishes, including natural returns, hatchery escapement, and harvest at the subbasin, province, and Columbia Basin levels until it is properly documented and validated in a substantive review.
In terms of an overall approach to setting integrated objectives and establishing priorities, the ISRP and ISAB recommend that two or more modeling approaches be developed, and that a serious effort be dedicated to validating the respective models against real data, diagnosing and reconciling whatever differences emerge between models, and conducting deliberate and rigorous experiments to resolve empirically the uncertainties about parameter values or model form that prove to be important to the predictions. We understand that the SHIRAZ model, developed at the University of Washington, has similar objectives to the AHA model as an integrated tool and was used in the Snohomish River Basin for regional recovery planning. In the absence of a review of that model, we do not know whether or not it is fully enough developed to allow immediate application in the Columbia Basin.
Recommendation. Two or more models should be explored in the process of developing tools for evaluation of numerical objectives for anadromous fishes in relation to natural returns, hatchery escapement, and harvest at the subbasin, province, and Columbia Basin levels.
There are some inherent differences between ecosystems in Puget Sound and Coastal Washington and ecosystems in the Columbia River Basin. Some additional components (e.g., more information about the hydrosystem) need to be added to the AHA model before it is fully applicable in the Columbia River Basin. Adult anadromous fishes need to be allowed to return over multiple years, as do chinook and steelhead. The model needs to incorporate different routes of passage and survival parameters through the Columbia River Basin hydropower system. Allowance should be made for variation in additional critical input parameters, such as productivity, capacity, and harvest rate. Assigning single values to input parameters fails to account for both uncertainty about the parameter and natural variation in the parameter.
Recommendation. The AHA model should be tailored to meet unique ecosystem conditions in the Columbia Basin.
Answers to Council Questions
Question 1) Does AHA provide a useful approach to developing regional objectives by integrating hatchery programs and harvest goals with subbasin plans?
It is a little misleading to call this model in its current form an "All-4H analyzer". The hydro part is essentially missing. Different routes of passage and survival parameters through the hydrosystem (e.g., transportation or inriver with and without spill) are not included. The habitat part consists only of a pair of capacity and productivity parameters. Values for these must be generated outside of the model. When applying AHA in a "4H" context, we assume there must be some combination of use of AHA and EDT, for example.
The AHA model is intended to generate ranges and means for the numbers of anadromous fish released from hatcheries, of returning natural fish, of hatchery fish, and of harvested fish under constraints imposed by limitations on the genetic interaction of hatchery and natural fish. The ISRP and ISAB do not believe these numbers should be used as explicit objectives for subbasin or province plans. When effects of the missing "H," i.e., hydro are added, the basic fields (productivity, capacity, hatchery production, harvest, fitness, etc.) needed to facilitate useful discussion of all-H integration will be present in the tool, but the reliability of the model predictions is unknown. Specifically, we cannot be confident in the reliability of the numbers generated by this "beta-test" version of AHA, and we are uncertain about the sensitivity of AHA to assumptions made in choosing values of its numerous input variables.
Some subbasin plans already specify numerical objectives for adult fish returns. Once the AHA model is validated and its output compared with that of other models, it may prove useful as an exploratory or discussion tool that might lead to refinement or reassessment of fisheries objectives for hatchery and naturally produced anadromous fish at the subbasin level. More importantly, the model also could be used to suggest critical experiments, which, if they were conducted properly, would quantify the model's reliability (and possibly lead to improvements in modeling capability and contribute to our scientific understanding of the factors controlling salmon productivity and role of hatchery/wild interactions).
Questions 2 and 3) What cautions or caveats would you give for interpreting the outputs of the AHA exercise? Do you have suggestions for improving the approach?
The model is still in development and some components that would be needed to effectively use and validate the tool in the Columbia River Basin are not yet incorporated. The ISRP&AB offer several suggestions to improve the AHA model. Documentation of the equations, and the data which serve as a basis for these equations, is needed to properly review the model. In addition, reviewers would need to meet with the modelers to discuss various results from several pilot model runs.
Based on our immediate impressions, it should be possible to incorporate the issues in the next three bullets in the AHA model relatively quickly:
- Analyze species whose adults return over multiple years, e.g., chinook. Currently AHA is a discrete generation model. Most hatcheries culture chinook, and it's not clear what effect this simplification in AHA has on its output for chinook.
- Incorporate different routes and survival parameters through the hydrosystem.
- Allow for variation in additional critical input parameters, such as productivity, capacity, and harvest rate. Although the current version of the model allows for variation in ocean survival based on a single published dataset, other critical parameters are only considered as fixed constants. Assigning single values to input parameters fails to account for both uncertainty about the parameter and natural variation in the parameter.
One possible improvement in the model would be to allow the option of inputting parameters as the mean and variance of an assumed distribution, with random selection of parameter values for simulating natural variation in the system. For example the model treats habitat productivity as a constant, whereas it is known that productivity varies from year to year as the environment varies, e.g., with climate events such as floods and droughts. Using long-term habitat averages in the model exacerbates the risk of missing the effect of short-term natural disturbances, which could result in the undesirable situation of adding too many hatchery fish when natural production is low. Harvest rates are not likely to be constant either. The implications of this shortcoming are significant. For example, we could use the AHA to create a scenario employing reasonable estimates of capacity and productivity, in which a small population would be stable indefinitely. If this abundance, capacity, and productivity were employed in a Population Viability Analysis model, however, the population would very likely go extinct because of random variation in the demographics of small populations.
The AHA model is undergoing a natural development from consideration of a few relatively simple concepts to incorporation of more options to mimic the complex reality of multiple natural systems. In the longer term, as the model matures, it would be useful to develop the methods and ability to:
- Incorporate the interaction of multiple species and their hatchery programs within a subbasin or with straying from other subbasins. Currently, the AHA model does not consider interactions among species or the effect of hatchery fish straying from one subbasin to another. Apparently, the HSRG encountered this difficulty in the Puget Sound effort and developed principles to guide analyses on a case-by-case basis.
- Aggregate objectives from subbasins to provinces and to the Columbia River Basin as a whole.
- Incorporate metapopulation genetics that consider straying among multiple populations that may differ in productivity and abundance. Straying not only provides gene flow but also can have important demographic impacts on component populations.
- Consider fitness changes due to human and natural causes, in addition to incorporating variation in demographic parameters discussed above. Fitness of hatchery and naturally spawning stocks is assumed to be different, but constant for each type of population. Human actions such as size selective harvest and habitat loss can alter fitness of both hatchery and naturally spawning populations.
- Conduct a sensitivity analysis that provides documentation of confidence in the data and parameter values used in the model. The data requirements of the model are substantial. Most of the data required by the model for specific populations may not be available at this time. We are concerned that objectives and protocols would be established based largely on multiple speculative parameter estimates that are not grounded in concrete empirical data. This practice has resulted in serious criticisms of EDT within the region. How representative are the data used? Which parameters are the most reliable, which most speculative, how does this uncertainty play out? What are the confidence intervals of parameters input to and of outputs from the model? For example, EDT outputs may be used for estimates of productivity and capacity in the AHA model. During our subbasin plan review, it was very apparent that the quality of and subsequent confidence in EDT outputs were significantly affected by the source and quality of the data available, the expertise of the modelers, and the time and resources available to do the analysis. One suggestion for capturing the users' confidence in data inputs would be to add a sensitivity component to input fields that would shade the field according to strength or degree of belief; e.g., red means general opinion, yellow means data from nearby watersheds were used, green means the model used empirical data from the area in question.
- Conduct an analysis of the sensitivity of model results to various input parameters; i.e., which input parameters are most influential and which have little effect on model predictions.
- Test model results by comparison with real data for actual streams. Can the AHA model replicate conditions close to where we are today? Could alternative models fit the actual data equally well? Would these alternative models lead to different predictions from the AHA if applied to other scenarios after this common calibration? It is our understanding that the HSRG carried out a calibration exploration with the AHA model in their application, but the presentations and materials we were provided did not include these exercises.
- Add a Ricker curve to the model to compare with the Beverton-Holt curves currently used in the model, because Ricker curves include certain density-dependent functions not present in the Beverton-Holt curves.
A subset of ISRP&AB members spent some time exploring the model and offer their observations on its use:
- The model can produce irrational results with certain combinations of input parameters. In one use, the model generated cells with "X" divided by zero, an "undefined" number. In the summary output, this was recorded as years with large returns of fish occurring a negative proportion of years.
- Using the model in its present form almost certainly would require that one of its programmers be present. Although the simplicity of the Excel spreadsheet approach permits easy access to using the program, questions arise about the functions that underlie many of the cells that can be manipulated by a user. The model would need to be used with caution, and by individuals knowledgeable about the computation of cell values. Regardless of how the model is used, we recommend that the model equations be fully documented and provided to the user.
- Under the natural component tab, it appears that the egg-to-smolt survival is adjusted by a formula as a function of the smolt-to-adult survival to accommodate a fixed productivity that was set on the population tab. There will be situations where the egg-to-smolt and smolt-to-adult survivals will be positively correlated, i.e., one will not compensate for the other. For example, in drought years, poor conditions can exist for egg-to-smolt survival and for downstream migration.
- There are numerous cells in the AHA model into which users can enter numbers that don't seem to affect the output of the model. As an example, in the natural and hatchery component tabs, changing the number in the initial population cell doesn't seem to have an effect on outputs on the population tab.
- Toggling the fitness function on and off did not change the output very much. But altering the fitness function itself had a large effect on the output. Additional time would be required to fully explore and understand how these features lead to the observed changes in the models predictions.
Question 4) Does the AHA model provide a useful approach to incorporating assumptions about fitness loss?
With the model still in a "beta" developmental state, and undocumented, the ISRP and ISAB reviewers could not examine the assumptions in detail with the presenters or in the supporting documentation and literature. The model does not provide an exhaustive menu of plausible alternatives, and dealing with the uncertainty about assigned parameters is left to the user. The model user is required to assign a difference to the optimum phenotype in a hatchery and in nature and to identify at what life history stage this difference is manifest. At this time, these assignments are based on professional judgment. There are other considerations of genetic change, such as Goodman (In Press, CJFAS) and Lynch and O'Hely (2001; this paper is cited in the model). There needs to be more explanation of how the fitness equation is employed throughout the population model; nevertheless, it is a laudable and important development of the AHA model that fitness impacts of hatchery fish on wild fish are explicitly considered in a planning effort. Any model used for this exercise should include a fitness component.
Question 5) Do alternative tools exist that would be better?
We suggest that the Council consider that this should not be an either/or choice. For numerous reasons we recommend that several alternative models be developed and used for this exercise. When people use a single model they may be tempted to place unwarranted faith in the numerical results, and they may not scrutinize the model itself. Given the current scientific uncertainty about some of the key relationships considered by the AHA, it is almost inevitable that another model would produce different results than the first model, and this would motivate greater scrutiny of each model's predictions and assumptions. In general, use of alternative models highlights differences in their outputs and helps users understand the consequences of each model's predictions and assumptions.
Models are useful, but practitioners should be aware of the assumptions inherent in the models. It is dangerous to use models without this awareness. We recommend that both proponents and skeptics be involved in modeling exercises, and that the range of views as to veracity of the results be documented
In terms of alternative tools, we are aware that Dr. Ray Hilborn at the University of Washington has developed an integrated (multiple Hs) model, called SHIRAZ, but we haven't studied it. There are, in the literature, many alternative models that could serve as a basis for alternative tools, or incorporated as options into a single tool.
Supplemental Questions
Questions 1 and 2: Did the initial applications of the AHA model to the Yakima and Kalama Subbasins provide measurable objectives for natural returns, hatchery escapement and harvest needs? Are these measurable objectives scientifically sound? Are they of sufficient quality to be adopted into the Council's program?
The Kalama and Yakima analyses are in draft form and aren't available for our review. An evaluation of those analyses, when available, would likely take more time than allotted for this review. Moreover, other reviewer concerns identified above about the model would make such an exercise unproductive at this time.
Based on our brief experience with the AHA model, we recommend that the output from the AHA not serve as a standalone source for establishing numerical hatchery production objectives. Output of these numbers is just a first step in developing objectives, i.e., one element to consider. There are other scientific considerations that are not currently incorporated in the model, such as those described earlier. In addition, we recommend that the Council look at results from at least two alternative models; e.g., test the AHA model, in conjunction with another model, on a subset of subbasins such as the Yakima and Kalama to see how the model outputs compare.
Question 3. Is Mobrand the only firm that can provide this type and quality of analysis?
See our comments on Question 5 above.
[1]ISRP and ISAB 2004-13. Scientific Review of Subbasin Plans for the Columbia River Basin Fish and Wildlife Program. For the Columbia River Basin Indian Tribes, NOAA Fisheries, and Northwest Power and Conservation Council.
[2]ISAB 2003-3: Review of Salmon and Steelhead Supplementation.
[3]ISAB 2001-1. Model Synthesis Report: An Analysis of Decision Support Tools Used in Columbia River Basin Salmon Management. Northwest Power Planning Council.
