Pipeline Risk Assessment—Controlling the Bias
In other parts of this site, we introduced the concept of pipeline risk assessment Essential Elements. This is a list of ingredients that arguably must be included in any pipeline risk assessment. One of these essential elements—the need for measurements–underlies all QRA. Here, we address another essential element, closely related to the use of measurements—controlling the bias. As used here, this includes dealing with both conservatism and uncertainty.
Essential
This idea of managing uncertainty might at first appear to be a rather obscure, highly technical issue only. However, it is actually an essential element and critical to proper risk assessment. It is essential to an understanding of the risk assessment and the subsequent use of the risk estimates. If not already defined, one of the first questions to ask when viewing a risk assessment is: ‘what is the level of conservatism in this assessment?’.
Note that conservatism plays an important role in both assessing AND managing risks. A critical and challenging aspect of risk management will be “to what level of conservatism should I manage?”. Knowing that risk assessments are generating probability distributions, even when discrete values are reported, how much should a decision-maker focus on the extremes? Avoiding incidents that may happen every few years is certainly important. At the other extreme, how much resources should be expended to avoid the extremely rare, eg once every 100,000 years, incident?
Towards a PRA
The most robust way to deal with uncertainty is to employ a fully probabilistic risk assessment. In the PRA, every input is a distribution with values of central tendency and dispersion. So rather than point estimate of input values, the full distribution of possibilities is used as an input.
The poor man’s PRA: Rather than set up and execute the classic Monte Carlo simulation for dozens of inputs, simply choose two or more points on the hypothetical distribution (eg, P50 and P90+) and perform parallel RA’s to generate values to approximate that distribution.
The risk assessment model advocated here is quantitative risk assessment that also achieves the status of probabilistic risk assessment by using approximate methods. Rather than creating distributions for every input, this QRA performs calculations using several points from the theoretical underlying distributions thereby approximating the use of a full distribution.
For example, suppose depth of cover Is known to range from 10 inches to 40 inch with 32 inches being the median and average of the distribution of possible depth values. This QRA will move towards a PRA by using 32 inches as the most likely value and it also using a value approaching 10 inches as the conservative possible value. These two points on the distribution, one being labeled the more likely or P50 value, in the other labeled P90 plus as the more conservative value represent the distribution in subsequent calculations.All other inputs are similarly Integrated into this QRA by selecting two points from their respective theoretical distributions. Theoretical is emphasized here because in most cases we will not know the actual distribution for any input along the many locations of a typical pipeline analysis. When the QRA is complete . The QRA dust produces two sets of risk estimates, one for the P50 parentheses most likely parentheses scenario an 14A more conservative scenario. The more conservative set of risk assessment results will be the most widely used since there are many benefits for using conservative numbers.
Conservatism
Conservatism is generally taken to mean an intentional bias towards over-estimation of the true risk. Risk assessment incorporating a high level of conservatism will tend to overstate the risks, perhaps by several orders of magnitude. This occurs through the use of input values and calculations that are based on worst-case or at least ‘higher risk’ assumptions. Risk assessment conducted with no conservatism always assumes the most likely values and the results are most often true to the most common actual conditions. So, uncommon conditions are missed unless conservatism is used.
Conservatism is a useful characteristic in many applications of risk management. However, conservatism may also be excessive, leading to inefficient and costly choices when not properly acknowledged in decision-making.
PXX Details
PXX
It is important that a risk assessment identify the role of uncertainty in its calculations. Each assessment should be performed with a pre-determined target level of conservatism (which includes the handling of ‘uncertainty’, for our purposes here). Depending on the intended use of the risk assessment results, various levels of conservatism might be appropriate. As an aid to communication of conservatism level, a PXX designation can be used to show a level of confidence that actual experience will be no worse than estimated. For instance, P90 is the point where 90% of future performance is expected to be ‘better’ than this value—one would be negatively surprised 10% of the time or once out of every ten episodes. P99.9 is very conservative—a negative surprise occurs only once out of every 1,000 episodes.
The risk modeler should determine the level of conservatism appropriate to his audience (normally the users are decision-makers) needs. The PXX designation communicates this to the user of the risk assessment. PXX can refer to various aspects such as the conservatism in each input value or the conservatism in the final estimate.
P90+
A P90+ assessment (P99, P99.9, etc) intentionally contains layers of conservatism. A P90+ risk model assumes that things are ‘bad’ until proven otherwise. This is often done to encourage future data collection as a means of risk reduction and, more importantly, to ensure that risks are not underestimated An underlying theme in a P90+ assessment is that ‘uncertainty shows as increased risk’. This is a conservative approach requiring that, in the absence of meaningful data or the opportunity to assimilate all available data, risk should be over-estimated rather than underestimated. Riskier values are generally assigned, reflecting the assumption of poor conditions, in order to accommodate the uncertainty. This results in a more conservative overall risk assessment.
A P90+ level of conservatism encourages and quantifies the value of data collection via inspections and testing. It also avoids a discrediting of the model that would occur if, through the discovery of non-conservative estimates, it becomes apparent that the model is awarding the ‘benefit of the doubt’, thereby concealing possible risks.
So, a P90+ assessment is done to encourage data collection as a means of risk reduction as well as to protect the model’s credibility. Some pipeline-specific examples of high conservatism include:
- Assigning high rates to various potential exposures, eg using very aggressive corrosion rates, even when very rare
- Assuming poor performance of older coatings and coatings of a certain type, even though, in the vast majority of cases, most coatings continue to perform very well.
- Use of worst-case potential consequences, even when potential for larger consequence events is extremely small.
- Assuming weaknesses in pipe strength, even if no direct evidence suggests their presence
- Underestimating the likely benefit of mitigations
Note that when a number of P90+ inputs are used, they lead to final estimates that are much more conservative—perhaps P99.99 or higher. The P90+ assessment produces a point estimate for an extreme portion of the assumed distribution of actual values. It suggests a very unlikely level of risk, but plausible. Therefore, the P90+ assessment is more appropriate for use in risk management of individual pipeline segments. A P90 level—negative surprises only 10% of the time—or higher is often warranted for risk management of specific pipeline segments. More conservative assessments may also be appropriate when supporting new projects or for presentations in public forums.
P50
A P50 level of assessment represents the best estimates—the most likely values that will occur. A P50 assessment best describes the anticipated behavior of the entire population of pipeline segments. Such estimates are often used to calibrate the risk model. However, P50 values will misrepresent the true risks for individual segments. This is because the P50, as a point estimate for the mode or mean of the assumed distribution for the population, ignores the extreme values in that distribution.
In addition to its use in calibration, a P50 to P70 level of analysis might be appropriate for budget setting or long range planning. The future behavior of whole pipeline systems is better understood via P50 assessments. However, P50 estimates must be used very cautiously since they are designed to better measure the performance of populations rather than individual segments. They are often inappropriate for use in risk management of specific portions of a pipeline.
Bias
Bias
The list of essential elements for good risk assessment notes ‘bias control’. The meaning behind the phrase—controlling the bias—can be less succinctly described as: identify, understand, and manage the uncertainty, conservatism, and subjectivity of the assessment. Much has been written about uncertainty, quantifications of uncertainty using statistical theory, and philosophical implications of knowledge types and lack of knowledge. We will focus here on the more practical aspects of handling uncertainty in our pipeline risk assessments.
‘Controlling’ is a deliberate word choice. We recognize that some bias is intentional and useful so we are not trying to avoid all bias. The right amount of bias applied at the right points in the process, is key. Therefore, the intent is bias control.
Uncertainty is always present in our risk assessments because we have incomplete knowledge of true values—we don’t know the exact material properties at every point along every pipeline; we don’t know how many times an excavator will actually be digging nearby; we don’t know where coating deterioration may have occurred, etc.
Even where we have measured values, we know that no measurement is perfect. All measurements are actually estimates. Measurements sometimes do not even involve the use of a measuring tool. For example, an estimate of 4 excavations per km-year is a measurement of the future activity level near the pipeline.
We estimate or measure things in full recognition that there is an inaccuracy associated with either. We often measure samples to infer values of all members of the population. We understand that the real world involves distributions of possible values, not point values. The nominal wall thickness we record is 0.250” but we know that, at various points along the pipeline, the wall thickness may actually range from 0.231” to 0.267”. We estimate the average risk (expected loss) for a segment of pipeline to be $220/year but we understand that there can be a multi-million dollar incident here next year, and again the year after!
We cannot eliminate uncertainty, but we can manage it. This exactly mirrors risk—we cannot eliminate that either, but can manage it.
Avoiding Surprises
Location-specific estimates of risk should be conservative–tending to overstate actual risks. As a general rule, new information should reduce previous risk estimates. This is because a conservative risk assessment always assumes that things are getting worse unless information indicates otherwise–“guilty until proven innocent”. If new inspection findings result in surprising underestimation of risk, then the risk assessment failed to use an appropriate level of conservatism. The inspection showed actual conditions to be worse than assumed–a very undesirable situation.
Exceptions to this general rule would be discoveries of new failure mechanisms or new factors, previously unknown or underestimated by the state-of-the-art science and engineering, that impact known risk aspects in new ways. These scenarios are expected to be very rare.