Background
At the October 21, 2005 request of the Council, the ISRP reviewed the Idaho Supplementation Studies’ (ISS) document, “Updated Study Design and Statistical Analysis of Idaho Supplementation Studies.” This document is the latest reviewed in an iterative evaluation process of the ISS program that began with the proposals reviewed in the Fiscal Year 2002 Mountain Snake Province Review. In the provincial review, the ISRP recommended “not fundable until the ISRP concerns are adequately addressed.” The ISRP commented that the ISS’s experimental design had not been adhered to and the proposed experimental design was not adequately defined. The ISRP recommended that the ISS sponsors provide a table specifying timelines for termination of the treatments on a stream-by-stream basis. The ISRP further commented that future commitment to treatment durations, particularly to the portion of the study design in which supplementation ceases, would enable the project sponsors to analyze the effects of the treatments. In the past, the project sponsors had not maintained the agreed upon control streams; consequently, the ISRP recommended that the ISS sponsors develop a certified statistical design. The Council recommended that the ISS sponsors address the ISRP and Council concerns.
In 2003, the ISRP reviewed the ISS sponsors’ response to the ISRP and Council concerns (see ISRP 2003-8). The ISRP found that the ISS sponsors made meaningful progress in developing protocol to specifically address ISRP concerns about the staggered timetable, partial treatment streams, statistical power, and statistical analysis in the face of straying by conventional hatchery fish into control and treatment streams. The ISRP, however, raised several outstanding concerns that the Council recommended the ISS sponsors needed to address for continued funding (Council memo to the ISS sponsors dated July 9, 2003). The document under review (IDFG Report Number 05-35) was developed to address these concerns.
Specifically, the ISS sponsors needed to:
- develop a final design for Phase III,
- expand carcass collection to all ISS study streams to better estimate the effects on production by hatchery strays,
- evaluate DNA-based assessment in ISS treatment and control streams to further identify parental contribution from the three classes of adults in Phase III analysis.
The ISS includes the following projects:
- 1989-098-00, Idaho Supplementation Studies (IDFG, Salmon Subbasin).
- 1989-098-01, Evaluate Supplementation Studies in Idaho Rivers (USFWS, Clearwater Subbasin).
- 1989-098-02, Evaluate Salmon Supplementation Studies in Idaho Rivers (NPT, Salmon Subbasin).
- 1989-098-03, Salmon Supplementation Studies in Idaho (SBT, Salmon Subbasin)
- 1990-055-00, Steelhead Supplementation Studies in Idaho Rivers (IDFG/IOSC, Clearwater Subbasin)
- 1996-043-00, Johnson Creek Artificial Propagation Enhancement Project (NPT, Salmon Subbasin).
Review Summary
The sponsors of the Idaho Supplementation Studies have made a serious effort to address recommendations and comments from past reviews in this response/update. They have worked hard to assess the quality and quantity of existing data and to conduct some preliminary analyses. They are cautious (almost cryptic) about the strength of the conclusions they will be able to draw at the end of Phase III. The ISRP is similarly cautious about whether the analysis of the final data will convincingly answer the questions posed by the ISS. With that caveat, the ISRP is more optimistic now than in the past, that this project can make a meaningful contribution to evaluating the efficacy of supplementation. The ISS demonstrates the difficulty of maintaining an experimental design that encompasses whole watersheds and multiple years/generations because of external and unforeseen events. In bringing the ISS to conclusion, the project sponsors must direct rigorous attention to the details of the final data collections in the field – including redd counts, juvenile abundance, adult escapement, and carcass counts – to provide for analysis and interpretation.
The ANOVA and Regression modeling of the data from the Phase III data collections appears appropriate. The ISRP recommends that the sponsors pay particular attention to the response variables (whether they use redds/km or juvenile or adult abundance) and covariates in these analyses. In the update the response variable (dependent variable, redds/km) is presented on the grounds that it represents the most complete data set. The analysis of redds/km needs to be executed in a manner that demonstrates that it is a meaningful metric for evaluating supplementation. In the update, the experimental design to assess the response variable is not explicitly linked to the objectives of the ISS or the questions posed and ought to be.
The carcass collections that the sponsors conducted in 2003, and indicate they will conduct for the remainder of the monitoring, are essential for the analysis and evaluation of the effectiveness of the supplementation conducted by the ISS. In this ISS update, a table (Table 3.1) is presented summarizing the carcass recoveries from 32 streams in 2003. The table and accompanying text are insufficient for the ISRP to evaluate the adequacy of the carcass recoveries to provide data to serve as a covariate in the final ISS analysis. The purpose of the carcass collection is to establish the proportions of natural, ISS Supplementation, and General Production adults that are spawning in a particular stream. These proportions will serve as a covariate to account for different sources of adults producing redds in the ANOVA and Regression modeling. The ISRP recommends that the ISS sponsors review the data and field protocols used in 2003 to ensure that the estimates have sufficient precision.
The intention and mechanism to collect tissues for a DNA-based parentage and assortative mating assessment is adequate. The eventual genetic analysis of parents and subsequent progeny produced in ISS treatment and reference streams will undoubtedly provide important insights into the efficacy of supplementation. It is likely that both the genetic markers (microsatellites or single nucleotide polymorphisms) and analytical tools and approaches will advance during the duration of the study; therefore, it is not necessary to commit to a particular method at this time. Rather, it is important at this time to put in place the elements to conduct the investigation (sampling and tissue collection protocols), and establish participation of molecular and statistical geneticists to assist with the eventual genotyping, data analysis, and interpretation.
Review Comments
1. Final Design for Phase III
The treatment measures and the response variables used in the final analysis need to reflect the study objectives and research questions.Section one of the update identifies study objectives and research questions. The first objective is “Monitor and evaluate the effects of supplementation on presmolt and smolt numbers and spawning escapements of naturally produced Chinook salmon. The first research question is: “Does supplementation of existing Chinook salmon populations enhance natural production?” The primary response variables available to assess this question include the number of redds, juvenile abundance, and adult escapement (counts of adult fish) between the various treatment and control/reference streams. It is not clear from the response that the ISS sponsors have addressed the subtleties the analysis requires.
Based on the objective of measuring the effect of supplementation on the number of naturally produced presmolt, smolt, and adult Chinook salmon, the ISS is required to evaluate the numerical response of natural juvenile and adult abundance to natural spawning by ISS hatchery adults, i.e., juvenile and adult recruitment in the next generation. Simply establishing an increase in the number of spawning females (natural plus hatchery) as reflected in redds/km as a consequence of releasing hatchery-origin smolts is not sufficient. For this reason unadjusted redd numbers, or a standardized redds/km is not a sufficient response variable. The redd counts need to be partitioned into those attributable to natural spawning adults and those attributable to ISS-hatchery adults, in addition to the recognized covariate General Production-hatchery (GP hatchery) stray.
From the experimental design it is logical to analyze two questions, and they will employ different sources of treatment and response data. The first question is what is the effect of hatchery smolt releases on the number of adult salmon returning from the ocean? For this question the treatment is the number of smolts released, or perhaps the proportion of smolts leaving a stream that were of hatchery-origin. The response variable is the number of returning adult salmon that are of hatchery-origin. The second question is what is the effect of the spawning of these returning hatchery-origin adults on numbers of natural juveniles and adults in the next generation? For this question the treatment is the numbers or proportion of spawning fish that are of hatchery origin. The response variable is the number of natural juvenile progeny and the number of subsequent natural adults returning from the ocean.
Redds/km is a standardized measure of adult female salmon abundance. In the case of the ISS, these redds/km can be produced by natural-origin salmon or by hatchery-origin salmon. As the ISS progresses from Phase I, to Phase II, to Phase III, the source of the females that contribute to redds/km and how they are interpreted in evaluating the ISS will change. In an ideal stream where there was no GP hatchery straying, the redds/km in Phase I would represent only natural-origin salmon, in early phase II it would represent natural-origin and ISS-hatchery-origin salmon, in late Phase II the natural-origin component will represent natural fish that had natural parents and natural fish that had hatchery parents in the previous generation, plus there will be an ISS-hatchery-origin contribution. This partitioning will extend into early Phase III. In late Phase III there will be no more ISS-hatchery-origin contribution, but there will be natural production owing to the wild spawning of hatchery fish in the previous generation. If Phase III is extended for enough generations, the system will return to a state where all the redds are produced by females that had natural-origin parents even though some will have had hatchery-origin fish in their pedigree several generations earlier.
It appears that the ISS sponsors are devising a way to correct for GP spawning, as a source of redds/km, but they are not devising a way to partition the remainder of the redd production into the component sources. This does not provide an evaluation of the critical points of supplementation. From the long history of artificial production in the Pacific Northwest we can almost be certain that adding hatchery smolts will increase total adult returns and probably also redds. Evidence for this from the ISS analysis to date is the significant positive relationship between GP straying and redds/km. It would be very disappointing if all the ISS accomplished was confirmation that adding hatchery smolts to a stream increased the total adult abundance and redds/km. The question posed by supplementation, as identified by the ISS sponsors, is whether supplementation can increase the abundance of natural-origin juveniles and adults. What we want to evaluate is the redds/km from natural-origin adults in late Phase II and early Phase III – when these natural-origin adults represent the progeny of ISS-hatchery-origin adults spawning in the wild.
The ISS statistical committee needs to address this more detailed partitioning and evaluation of their response variable.Using both the mixed model ANOVA and regression analysis for confirmatory and exploratory analyses is justified. The manner in which statistical blocking will be considered in the analyses is not clearly identified.
For the regression analysis we caution that using Phase I redd production to represent baseline stream productivity may not be advisable. In Phase I carcass data is largely lacking so the partitioning into components during that period is not possible. The ISS sponsors acknowledge this on page 24 in their discussion of the regression analysis:
1. Any regression model that incorporates GP hatchery straying cannot include Phase I data as straying numbers were not collected before 1995.
However, they argue in point 2:
2. Phase I redd production can be used, however, to indicate baseline stream productivity and as such appear as an independent variable in the analysis.
The ISRP is unconvinced on this point. The lack of data on the proportion of hatchery fish in a spawning mixture and the lack of data on the reproductive performance of those hatchery fish has been identified as limiting the extinction risk assessments for Columbia River Basin salmon by NOAA-Fisheries and the TRTs and gathering data on that problem was (is?) part of the RPA (182) for the Hydrosystem BiOp. It is entirely possible that a stream could have lots of redds built by hatchery adults that produced almost no subsequent adult offspring, and the metric redds/km would not reflect the inherent productivity of that stream.
Finally, the response variables are components of productivity, and they contribute to the study at a most fundamental and understandable level. While not presented as such, there is presumably some capability to examine the "character" or shape of the responses as well. For example, some assessment and comparison of these variables across seasons will be of primary interest to determine whether the full complement of run subunits are being rebuilt (or conversely compromised) or if the performance of one or a few such subunits are camouflaging the performance of others.
Questions 3 and 8 are intriguing in that they take a risk assessment perspective and are probably answerable only in a very general sense. Specifically, the sponsors recognize that "acceptable decrease" in Q3 is undefined (pp. 5 & 6). At first glance, this appears to be a bet-hedging question, in that they are quite prepared for a decrease in natural productivity from supplementation-augmentation, but trying to decide how much can be lived with. As such, this is a policy question as much as a scientific one. Question 8 is similar in tone and perhaps unanswerable.
The discussion following Q3 suggests that a solution to determining if supplementation reduces natural productivity below acceptable levels would be to identify any detectable change in productivity. Without considering the probability of detecting meaningful change (power of the test) there is a good chance that inadequate data will be collected to identify when a change has occurred.
Related to the issue of statistical power is a concern in detecting change in adult escapement when addressing Q1 and Q2 because of the limited number of identified control streams in tables 1.1 and 1.2.
2. Carcass Collection
Identifying and assigning adult fish to natural production, ISS-hatchery production, and general production (GP hatchery) based on carcass recoveries, for the purpose of accounting for straying from non-supplemental hatchery releases and partitioning redd production into natural and ISS-hatchery sources is necessary for complete analysis of the ISS. Straying from other hatcheries (non-supplemental releases) may greatly confound results if these cannot be filtered out of the analysis. The stray rates for some of the streams are astonishingly high. According to the response (p 25), “carcasses are critical for determining GP hatchery stray rates into treatment and control streams. Thorough carcass sampling should be conducted in as many treatment and control streams as possible.” It is not stated whether carcass collection continued in 2004-05. Did it? The ISRP recommends that such collection indeed be conducted throughout Phase III.
Table 3.1 (page 23) presents the carcass recoveries from 32 streams in 2003.The carcass data that were collected in 2003 appear to be a valuable addition to the program. The table and accompanying text, however, are insufficient for the ISRP to evaluate the adequacy of the carcass recoveries to provide data to serve as a covariate in the final ISS analysis. No details are given on how the carcass surveys were designed to assess the veracity of the estimates. This is needed before an assessment of the scientific adequacy can be made. Some streams (Eldorado Creek, Big Flat Creek, Pete King Creek) had no carcass recoveries at all, and others had very few. Did these same locations have redds? If they did have redds, how will the redds/km be partitioned into the various sources? How much sampling variation is there in the other streams? What is needed, that is absent from the presentation, is evidence of the precision and bias in the estimates of carcasses assigned to different categories.
Furthermore, the use of the straying covariate in the appendices depends on the assumption that the relationship between the response (e.g., migrants, total survival) and the covariate is linear and that the slope of this linear relationship is the same for all stream groups. It is not clear if these assumptions have been verified. An additional concern is the limited number of carcass recoveries in some study streams (Table 3.1). The reliability of these estimates of hatchery straying based on these data will influence the results of an analysis using hatchery straying as a covariate.
The ISRP recommends that the ISS sponsors review the data and field protocols used in 2003 to ensure that the estimates have sufficient precision.
Finally, p.22, last paragraph, line 6: there is a suggestion here that pre-spawn mortality may be a factor to be considered? Is it significant enough to use as a response variable? It is a metric that might tie with some habitat features such as temperature.
3. DNA-based Assessment
The intention and mechanism to collect tissues for a DNA-based parentage and assortative mating assessment is adequate. The eventual genetic analysis of parents and subsequent progeny produced in ISS treatment and reference streams has the potential to provide important insights into the efficacy of supplementation. It is likely that both the genetic markers (microsatellites or single nucleotide polymorphisms) and analytical tools and approaches will advance during the duration of the study; therefore, it is not necessary to commit to a particular method at this time. It is important, however, to put in place the elements to conduct the investigation (sampling and tissue collection protocols), and establish participation of molecular and statistical geneticists to assist with the eventual genotyping, data analysis, and interpretation.
The section on the genetic analysis appeared focused on the difficulty identifying progeny produced by small proportions of GP hatchery fish. This should not preclude the usefulness of this approach. If the parents are in such small proportion, uncertainty in those particular estimates may need to be accepted. The information derived from locations where GP hatchery strays and ISS hatchery adults are spawning in appreciable proportions will remain insightful. In the tests of assortative mating and parentage, the sponsors desire to have 5 representatives of the smallest expected class, and then perform contingency tests to evaluate relative reproductive contributions. These analyses need not be held to a minimum of 5 for the lowest expected frequency in a contingency table - that rule of thumb is too conservative. Further, there is no reason why exact techniques or computer intensive methods cannot be use in place of the conventional chi-square.
Additional specific technical suggestions are identified in the next section of this memo.
Specific Comments and Editorial Suggestions
In reviewing documents, the ISRP identifies editorial suggestions or very specific technical issues that would benefit the project sponsor but are not critical to the decision-makers. Rather than leave these comments out or confuse the primary text, the ISRP usually puts these at the end of their reports.
Page 10, line 3: if smolt releases have resulted in largest adult return, doesn’t this answer one of the questions?
Table 1.5a, caption: what is meaning of local vs. non-local?
Table 1.5: some expansion of the meaning of extinct and extant in the context of the ISS would be helpful at the beginning of the document.
Page 23, Table 3.1: re the natural origin in Column 3 – is there any possibility these are unmarked feral fish or is the genetic analysis reliable?
The equations in section five need explanation. For example, the equations presented on page 31 are inconsistent. That is, the equation for P cannot be solved to give the equation for L due to missing parentheses. The formulas should be corrected. The explanation of the capture efficiency adjustments would be enhanced by a numerical example. The equation for confidence interval on page 34 appears to have an incorrect constant. If the ISRP correctly understands what is being done here, the value 2.95 should be 2.58. This provides a smaller sample size than is presented.
On page 36 the recommended use of Dunnett’s test is incorrect due to the fact that the proportions will sum to 1, so the data are not independent. The ISRP was not able to duplicate the power estimates presented in Table 5.9.
Page 37. For example, if 20% of the adults sampled in generation one were ISS hatchery origin fish, and reproductive success was equivalent among all origin and cross types, we would expect 20% of the progeny sampled in generation two to be assigned to an ISS adult.
It is a small point, but above sentence is incorrect. It should be “if 20% of the adults sampled in generation one were ISS hatchery origin fish, and reproductive success was equivalent among all origin and cross types, we would expect 20% of the parents identified in the progeny in generation two to be assigned to an ISS adult.” Specifically, if 20% of the parents were ISS hatchery and 80% were natural-natural, then in the next generation 4% would be ISS/ISS, 32% ISS/Natural, and 64% Natural/Natural. So, 36% of the progeny would have at least one ISS parent and 96% would have at least one natural parent.
