At NOAA Fisheries' request, the ISAB reviewed the Comprehensive Passage Model (COMPASS), Version 1.1.
See the full report (above link), and the Executive Summary excerpted from the full report below.
Executive Summary
This report is the fourth in a series of ISAB reports pertaining to the development of the Comprehensive Passage Model (COMPASS) created by NOAA Fisheries along with federal, state, and tribal agencies and the University of Washington. COMPASS is intended to predict the effects of alternative hydropower operations on salmon survival rates and provide ongoing evaluation for the new Federal Columbia River Power System Biological Opinion (BiOp). COMPASS is a welcome addition to the analytical tools available to both scientists and managers. These periodic ISAB critiques have been explicitly intended to provide constructive suggestions to facilitate continuing development of a valuable modeling tool. The specific questions for this round of review, and our responses, are the following:
- Does the model successfully perform the desired capabilities, as listed below?
- Realistically portray the hydro-system and variable river conditions - The fit to available in-river and hydro-system data is quite good. With a few exceptions, the model has captured the impact of the variables considered. The question of how well the model will work for river conditions encountered in future years must await later data.
- Allow for the simulation of the effects of management actions - COMPASS will permit evaluation of a reasonable range of management options, though the passage data are still insufficient to fine-tune the management choices. Full-blown management simulation is (mostly) a future challenge for COMPASS, but the possibilities are promising.
- Characterize uncertainty in prediction - This version provides improved treatment of uncertainty, allowing for the correlation of estimates from sequential projects. The uncertainty is separated into components for stochastic sampling and for differences among time periods. How well that treatment serves the simulation effort must await a fully simulation-capable version.
- Represent hydro-system-related effects that occur outside the hydro-system - The Bonneville Dam (BON) → Ocean → BON survival component of the model is still poorly characterized, in the absence of reliable data from below Bonneville Dam. The ISAB’s sense is that continuing to elaborate latent mortality is somewhat pointless, given the lack of comparable data from the pre-hydro-system period. The ISAB also concluded that it is time to separate the detailed survival experience of fish transported from each collection point (Lower Granite (DAM), Little Goose Dam (LGS), Lower Monumental Dam (LMO), and McNary Dam (MCN) separately, because those seem to be different, and it is not possible to model the transportation alternatives if transported fish from all four projects are treated as a single cohort.
- Is the model too complex or too simple? - The ISAB’s sense is that the model is now of about the complexity that will be useful, and it is manageable.
- Does the model realistically represent the data and its variability? - The model allows for variability in prediction, based on variability in the input parameters. And, at least where the requisite empirical data exist, the model does a credible job of reflecting a dynamic reality. The requisite data are sometimes in short supply, however, and both the COMPASS team and the ISAB recommend that more data of the necessary types be gathered.
- Are the statistical methods sound? - The COMPASS team’s statistical methodology is generally sound, but questions remain about several of the methodology’s finer points (the AIC criteria, log-linear vs. logit-linear regression and prediction, multinomial vs. normal error structures, and the inclusion or exclusion of a grand intercept term in the model). The effort is moving along nicely, but statistical methods are still evolving in this arena, and it is premature to view the methods embedded in COMPASS as firmly set.
- Is the documentation adequate? - The documentation is good, as far as it goes, though we offer some suggestions for additional improvements. The COMPASS team has decided to delay preparation of the User’s Guide for a later effort. The ISAB’s view is that deployment region-wide cannot realistically occur without that Guide. Strategic and management decisions are already being considered, and the BiOp is now reality, all of which argue for early availability.
In response to the ISAB’s third-round critique of the COMPASS document, the COMPASS team provided a variety of responses. The ISAB has used this opportunity to provide additional feedback on those responses, by way of iterating the conversation.
Finally, we provide a detailed critique on each section of the current version of COMPASS 1.1. This critique is offered in the spirit of constructive suggestion to the COMPASS team, and we trust that our critique will be useful in continuing efforts to develop this valuable modeling tool for the region.
See the attached report for full details on the ISAB's findings
