© 2003 Plant Management Network.
Analyzing Positive Finds with Explicit Uncertainty
Lawrence G. Brown, USDA, APHIS, Plant Protection and Quarantine, Center for Plant Health Science and Technology, Raleigh, NC 27606
Brown, L. G. 2003. Analyzing positive finds with explict uncertainty. Online. Plant Health Progress doi:10.1094/PHP-2003-1113-04-RV.
Regulatory realities, culture, and politics impinge on scientifically based policy recommendations. An appreciation of the regulatory framework that creates policy recommendations helps in understanding why it is important to state the confidence intervals, i.e., to state explicitly the uncertainty of each recommendation. This is necessary information for policy makers. Explicit uncertainty around each recommendation is a more approachable way to express what statisticians call the confidence interval. The purpose of this article is to build language bridges between the research community and the plant regulatory agency of the United States, the Animal Plant Health and Inspection Service, Plant Protection and Quarantine (USDA, APHIS, and PPQ). This will be accomplished through a discussion of Potato virus Y (PVY-N), the PPQ New Pest Advisory Group (NPAG), and the Citrus Canker Surveillance Standard proposed by PPQ to the International Plant Protection Convention (IPPC). The IPPC is an international treaty for plant protection to which 117 governments currently adhere. Three examples are given to illustrate real world solutions that used scientific methods to deal with the regulatory constraints and uncertainty that are the reality of policy recommendations. A personalized view is taken when considering the uncertainty that confronts regulators. I believe this regulatory uncertainty can be expressed as a function of two criteria: the probability that an event will occur, and the consideration of all the relevant information known about the pest (10). The scientific community generates knowledge through research and states the confidence interval. PPQ has the research support available to assist the scientific generation of knowledge that can fill data gaps in the regulatory process. Positive pest identifications have uncertainty and specifying the confidence interval around those data gaps defines the level of that uncertainty. The three examples demonstrate how scientific methods and results validation can be a bridge to regulatory solutions within the constraints and uncertainty that are the reality of policy recommendations (in other words, how to interpret a positive pest ID).
How Pests Become a Concern
PPQ considers a pest as positively identified when a PPQ taxonomic specialist in Identification Services or a specialist approved by the PPQ taxonomist declares the ID as official. Pests become a PPQ concern in many ways. The current PPQ Regulated Pest List was compiled using input from scientific societies, universities, and USDA Agricultural Research Service (ARS). Many of these established targets have formal PPQ Emergency Response Guidelines (14). Recurring emergencies are the result of multiple introductions such as with Xanthomonas axonopodis pv. citri (Xac), a bacterial pathogen that causes citrus canker, and medfly, an insect (Ceratitus capitata) pest of fruit. Finally, PPQ must analyze pests new to the US or that are a threat to the US.
There are a number of uncertainties that complicate and constrain formulating an official response to a suspected introduction of a regulated pathogen. An important and useful constraint guiding PPQ actions is the Agriculture Risk Protection Act of 2000 (Public Law 106-224, 106th Congress). Different priorities altered the focus of the Act but the basic mission is to protect the environment and agriculture of the US, while supporting safe trade. The political and economical ramifications that follow an official declaration that the US has a regulated pathogen has far-reaching effects on private and public agencies, such as governments, universities, grower groups, and private interest groups. As a result, these agencies tend to be risk-averse. A risk-averse culture is mollified through education, where the known risks and uncertainties are explicitly defined. (10). Paradoxically, it is often difficult to obtain funding to gain additional information from such a risk-averse culture because of a feeling of throwing money down a rat hole. We can challenge this “rat hole mentality” can be challenged by emphasizing “that the greater the uncertainty, the greater the expected value of the additional information” (10).
Clear identification is the first uncertainty that is addressed. The USDA Emergency Program Manual (14) states in chapter 6 that the presence of a new pest is suspect until it has been submitted to the PPQ National Identification Service for confirmation. The pest must be officially identified quickly and appropriate regulatory response quickly initiated as its presence may have negative consequences to US agriculture through global quarantines or embargoes of US commodities by trading partners. And this is before the suspect becomes a positive and is announced as such by the country that has the problem.
However, even a positive ID by PPQ has varied types of uncertainty associated with it. For example, the taxonomy may be ambiguous, such as the case with strains of Potato virus Y (PVY). According to De Bokx and Huttinga (4), PVY-O, PVY-C, and PVY-N are the three main strains in the PVY subgroup. Those which cause mild symptoms in most potatoes varieties are called PVY-O and PVY-C (4). When symptoms on potatoes are used to describe the PVY-N strains, usually two general pathotypes are observed. The acronym PVY-N is used for convenience and encompasses those pathotypes that cause veinal necrosis on tobacco and those that cause necrotic rings are called potato tuber necrotic disease or PVY-NTN (2,3,4). PVY-N is a regulated pathogen but not a serious threat to potato production. PVY-NTN is a serious pathogen of potato and is a quarantine pathogen.
Precise naming avoids problems or confusion caused by synonyms, transliterated vernacular names, and local laboratory jargon that can be problems confronting regulators. When a name is split into pathotypes, genotypes, or serotypes it can be confusing to a regulator or anybody who requires a precise ID of the pest. Moreover, as the taxonomy of a pest becomes increasingly stratified, obtaining a correct identification may be nearly impossible except in the most advanced diagnostic laboratories. That is, the technology required to identify to pathotype, for example, may not be usable if the technology cannot be transferred to non-research laboratories. Also, the ID coincides with the level of regulatory significance. In other words, the best technology for the field may only detect at the genus or species level (1).
The second uncertainty of a positive identification is that of understanding the epidemiological issues that arise early in the regulatory process and continue until either the pest is eradicated, controlled, or so widespread that it can only be managed. These issues are the spread of plant pests within a population of hosts and the factors that influence pest dispersal. Interpretation of a pest’s ability to become established and spread is based on scientific literature and opinion. Interpretations and recommendations are made with considerable uncertainty due to the knowledge gaps with rarely studied pests in remote geographic areas.
The third uncertainty of a positive ID has political and cultural uncertainties. Specifically, the question is how to determine the response required to a confirmed plant pathogen (positive ID) that has entered the US or has the potential to enter the US. Pests include anything injurious to a plant or plant product, e.g., other plants, plant pathogens, arthropods, nematodes, or mollusks. A potential pest is defined as one that is present in a nearby country or territory, and the proximity of the plant pest threatens the agriculture and ecosystems of the United States.
Making Good Policy
How does the USDA respond when a regulated plant pathogen has entered the US? What is the policy of the USDA for dealing with a pest that is present in a nearby country or territory and poses a threat to the agriculture and ecosystems of the US? Good policy analysis adheres to the “Ten Commandments” for policy analysis as defined by Morgan and Henrion (10) which are: (I) do your homework, (II) let the problem drive the analysis, (III) keep it simple but do not make it too simple, (IV) identify all significant assumptions, (V) be explicit about decision criteria and policy strategies, (VI) be explicit about uncertainty, (VII) perform systematic sensitivity and uncertainty analysis, (VIII) iteratively refine the problem statement and the analysis, (IV) document clearly and completely, and (X) expose the work to peer review.
Commandments one through five and nine through ten are PPQ policy and are followed by most regulatory analysts in the area. Many policy recommendations neither include explicit statements about decision criteria and uncertainty nor do they perform systematic sensitivity analysis and iteratively refine the problem.
How to Explicitly State the Uncertainty
What is a good method to explicitly state the uncertainty of the data? The rest of this paper shall focus on the framework of how analysts should provide policy makers of PPQ and many other organizations with scientific information that clearly states what is certain and what is uncertain. Both the clarity of the facts in the established literature and scientific opinion have uncertainty. What is important is to explicitly state the level of certainty of data and opinion. Doing this will alleviate many problems in science and policy that are complex and involve disagreements over issues of opinion and fact.
Pest risk classification can be uncertain if definitions are not precise. An invasive pest can become established, but not pose a threat to the ecology or economics of our natural resources. Invasive pests that pose little or no threat to the agro-environment are recorded by taxonomists and diagnosticians, and such records are scientifically important but do not warrant PPQ regulation. In contrast, an invasive and threatening pest by definition is one that becomes established and causes economic injury to ecosystems and/or agriculture. PPQ responds to a pest when a pest is invasive and threatening.
The first step in PPQ to explicitly stating the uncertainty of new pest identifications is the New Pest Advisory Group (NPAG). NPAG was set up as a framework for the early analysis of the significance of plant pests believed to be newly detected or posing a threat to US agriculture or ecosystems. NPAG is required to do this with stated certainty and they recommend a response by PPQ designed to protect American resources. NPAG also helps communicate to stakeholders PPQ’s position and intention about the threatening plant pests. The processes of recommending options and developing a response to a new plant pest have five steps (14): Step 1, Prepare a preliminary status evaluation and a data sheet; Step 2, Conduct a meeting of NPAG; Step 3, Prepare an NPAG report; Step 4, Make recommendation and specify action; Step 5, Communicate the final decision; Step 6, Track completion and maintain records. NPAG is the first step in stating the uncertainty in the knowledge about a new pest.
The second step in explicitly stating the uncertainty of the pest identification is developing a regulatory procedure. The exact method of implementation is determining sample size. Regulatory options are available to PPQ to regulate, quarantine, eradicate, or manage invasive pests. This is done either by cooperating with the states, turning it over to state management, or providing education, but always includes timely reporting of the options. For example, pest surveillance is critical to implementing and evaluating the success of a regulatory procedure. Decision guidelines are used to develop surveillance methodologies for specific pests by adapting the methods to the field using the “Ten Commandments” of policy (10). Operating characteristics defined by decision guidelines are defined with direct consultation with the end user. A key decision is determining sample size, which is derived from the expected outcomes. Maintaining simple guides is achieved by a cyclical process of learning, improving, and designing. The same is true for reliability, which is sampled for systematically and maintains precision. Having a stated time-line keeps the project relevant (10).
Decision Guideline Example
Xanthomonas axonopodis pv. citri (citrus canker) is an example of where a decision guideline was set out in the IPPC standard for surveillance for specific pests. This draft standard describes the components of methodologies used for determining the presence or absence of Xanthomonas axonopodis pv. citri (e.g., in the establishment and maintenance of pest free areas) in support of the Guidelines for Surveillance FAO No. 6, 1997. A pest-free area is an area in which a specific pest does not occur as demonstrated by scientific evidence and in which, where appropriate, this condition is being officially maintained (6,7). The scientific underpinning is the work of Hughes et al. (9). This guideline quantifies, simplifies, and has reliability. The purpose is to either declare absence of canker in an area or find a cluster of canker-infested trees.
For the science of the decision guide to be adopted, it has to work within the constraints of resources, personnel, politics, and institutional culture. If the guide cannot work, then the recommendation should be that the action has a level of uncertainty greater than what should be risked. This draft standard, I believe, is science-based, and requires a reasonable amount of resources and personnel, and meets the cultural climate of diverse citrus-producing nations.
How Does the Standard Work?
The key to determining the presence or absence of Xanthomonas axonopodis pv. citri (e.g., in the establishment and maintenance of pest-free areas) is the ability to find a cluster of canker-infested trees or declare an area free of the cluster. The standard proposes to do this with a 95% confidence of finding a 1% level of infestation. Work in the Florida Eradication Program (13) indicates that when the disease is found at this level of infestation, the opportunity for eradication is great. The surveillance program takes a tree census within a regulated area, an area which is subject to phytosanitary measures. Each area is subdivided into sub-areas with each having 2,000 trees (about 20 acres). After all the sub-areas are defined, x number of sub-areas are randomly selected for inspection. Within each selected sub-area, 314 randomly selected trees are visually inspected for citrus canker. All citrus canker suspects are confirmed by standard laboratory procedures (5,8,11,12). The survey is conducted on an annual basis or until citrus canker is identified in a sub-area. Each year (cycle), the area shall be re-evaluated for the addition or deletion of host plants/trees and sub-area are adjusted to maintain a constant number of trees per area. The number of sub-areas surveyed during the next cycle is again randomly selected. Within each sub-area that has been chosen, only 314 randomly selected trees are visually inspected for citrus canker. All citrus canker suspects are confirmed by standard laboratory procedures (5,8,11,12). With the science so validated, the resource, personnel, political, and cultural constraints can be validated. Resource and personnel constraints are the most difficult to validate. The political and cultural constraints are easier to validate when the others are satisfied.
The program is resource- and personnel-efficient because the maximum number of trees to inspect is 98,596. If an area has more than four million trees then the number to inspect still remains at 98,596. When the area has between 344,000 and 3.8 million trees the number of trees to inspect ranges from 49,612 to 90,746. The number of sub-areas ranges from 158 to 289. Although this appears to be a large number of trees to inspect, there are never more than 314 trees to inspect in any sub-area in any given cycle. Most countries can support such a program because two trained people can inspect an average of five to six sub-areas a day since each sub-area contains between 15 to 20 acres of trees. The most expensive part is the initial tree census, but an accounting of trees is already a requirement for most export programs.
This program is an important model of how to explicitly state uncertainty. Because the uncertainty was set at 5%, the analysis was much stronger. This example of how to explicitly state uncertainty is important because this program explicitly states the uncertainty, which is set at 5%. An analysis is stronger when it is explicit about uncertainties. The challenge of analyzing positive finds in the midst of uncertainty creates opportunities for creative thinking and often requires logic to temporarily fill knowledge gaps. In this final way, regulators rely on the research community for timely expert opinions that are grounded in scientific fact and experience. Putting off a request from a regulator because one does not want to deal with any level of risk only forces the regulator to rapidly seek other sources -- which may be more uncertain than the information the researcher can provide! For this reason, when a PPQ person requests information, please be forthcoming about the uncertainties in what one knows as well as clear in one’s opinion and the evidence that supports it. Morgan and Henrion (10) state it best when they say “A piece of analysis will be more useful if it treats the uncertainty explicitly, allowing users to evaluate its conclusions and limitations better in the changing context of the ongoing decision process.”
As Alfred Lord Tennyson once wrote, “To strive, to seek, to find, and not to yield.” That is PPQ’s quest when faced with the challenges of making pest identification, finding out what is known about the pest, and developing an appropriate agency response. The most common approaches to stating uncertainty are either to explain it away or ignore it. Many agencies have survived for years by doing this. In fact, many agencies maximize an outcome by using only the best estimates in the model’s output. Uncertainty should be stated always. Also, the analysis should properly weigh uncertain information from different sources. Government agencies disperse money and resources based on limited information in emergency times, and decision theory using uncertainty can help. Decisions have to be made about whether to spend money to gain additional information. Knowing that there is great uncertainty can help government make resource decisions. This is because, in general, the greater the uncertainty, the greater the expected value of the additional information. If there is little uncertainty then why spend more? If there is great uncertainty, then wise spending has a good chance of reducing uncertainty. Decision theory has helped policy analysis focus on uncertainty, but it does not capture everything that happens in the real world. Cognitive biases in the iterative processes must be recognized when building a model. Real world input is critical. Fortunately, this is balanced because government decisions are not made by one person and are not implemented at one discrete time. Usually the process involves many “actors” making explicit and implicit decisions over an extended time-line.
I wish to recognize Ms. Andrea Lemay and Dr. Eileen Sutker for their constructive comments and suggestions.
1. Brown, L. G. 2001. Emerging potato viruses that confront regulations. Pages 439-450 in: Virus and Virus-like Diseases of Potato. Gad Loebenstein, ed. Kluwer Academic Press, Dordrecht, Netherlands.
2. Chachulska, A. M., Chrzanowska, M., Robagilia, C., and Zagorski, W. 1997. Tobacco veinal necrosis determinants are unlikely to be located within the 5’ and 3’ terminal sequences of the potato virus Y genome. Arch. Virol. 142:765-779.
3. Chrzanowska, M. 1991. New isolates of the necrotic strain of potato virus Y found recently in Poland. Potato Res. 34:179-182.
4. De Bokx, J. A., and Huttinga, H. 1981. Potato virus Y. CMI/ABB Descriptions of Plant Viruses No. 242.
5. Engel, D. S., Graham, J. H., and Stall, R. E. 1991. Genomic relatedness of Xanthomonas axonopodis strains causing diseases of citrus. Appl. Environ. Microbiol. 57:2724-2730.
6. FAO. 1997. Guidelines for surveillance. International Standards for Phytosanitary Measures, No. 6. Secretariat of the International Plant Protection Convention, Food and Agriculture Organization of the United Nations. Rome.
7. FAO. 2001. Glossary of phytosanitary terms. Online. International Standards for Phytosanitary Measures, No. 5. Secretariat of the International Plant Protection Convention, Food and Agriculture Organization of the United Nations. Rome.
8. Gabriel, D. W., Kingsley, M. T., Hunter, J. E., and Gottwald, T. G. 1989. Reinstatement of Xanthomonas citri (ex Hasse) and X. phaseolis (ex Smith) to species and reclassification of X. campestris pv. citri strains. Int. J. Syst. Bacteriol. 39:14-22.
9. Hughes, G., Gottwald, T. R., and Yamamura, K. 2002. Survey methods for assessment of Citrus tristeza virus incidence in urban citrus populations. Plant Dis. 86:367-372.
10. Morgan, M. M., and Henrion, M. 1998. Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis. Cambridge University Press, New York, NY.
11. Rudolph, K., Roy, M. A., Sasser, M., Stead, D. E., Davis, M., Swings, J., and Gossele, F. 1990. Isolation of bacteria. Pages 45-94 in: Methods in Phytobacteriology. Z. Klement, K. Rudolph, and D. C. Sands, eds. American Phytopathological Society, St. Paul, MN.
12. Schaad, N. W. 1988. Initial identification of common genera. Pages 1-15 in: Plant Pathogenic Bacteria. N.W. Schaad, eds. American Phytopathological Society, St. Paul, MN.
13. Schubert, T. S., Rizvi, S., Sun, X., Gottwald, T., Graham, J., and Dixon, W. 2001. Meeting the challenge of eradicating citrus canker in Florida-again. Plant Dis. 85:340-345.
14. USDA-APHIS-PPQ. 2002. Plant Protection and Quarantine Emergency Program Manual. Online. US Department of Agriculture, Animal Plant Health and Inspection Service, Manuals Unit. Frederick, Maryland.