A Novel Approach for Government Acquisition and Procurement: Agile Risk Tolerance

In 2015, the commissioner of US Customs and Border Protection (CBP) engaged the Defense Acquisition University (DAU) (Fort Belvoir, Virginia, USA) to review CBP’s acquisition and procurement performance. As a result of this review, DAU offered 66 specific recommendations to CBP leadership, including two that were related to risk and were accepted and earmarked for implementation:

1. Set the tone for more risk tolerance.
2. Reward innovation and risk; do not punish failure.¹

These two recommendations have the potential to significantly improve the US government’s acquisition process.

In 2016, CBP chartered the multiyear Acquisition Management Performance Improvement (AMPI) initiative and organized 13 teams under six executives to address 38 of the DAU’s recommendations. One of these teams, led by CBP’s chief engineer, was tasked with addressing the two recommendations related to risk management. Based on these two DAU recommendations and CBP’s directive to implement them, an agile risk tolerance (ART) process was formulated, including a set of metrics to gauge its effectiveness.

What Is Risk Tolerance?

The term “tolerance” can be defined as “the act of enduring, or the capacity for endurance.”² Tolerance is not a new concept in risk management. The International Organization for Standardization (ISO) defines risk tolerance as an “organization’s or stakeholder’s readiness to bear the risk after risk treatment in order to achieve its objectives.”³ The Enterprise Risk Management Framework of the Committee of Sponsoring Organizations of the Treadway Commission (COSO) defines risk tolerance as “the acceptable variation relative to the achievement of an objective.”⁴ All three definitions have in common the idea of enduring an event or condition. Thus, if an entity is risk tolerant, it is willing to endure the likelihood of an uncertain event or condition and its possible impacts. Conversely, if an entity is risk intolerant, it is unwilling to endure a future event or condition and its potential impacts.

Increasing Risk Tolerance

CBP’s risk tolerance team quickly realized that implementing DAU’s recommendations included two significant hurdles:

1. Progress measurement—Tolerance is a qualitative concept. When one entity—an individual or a government agency—is described as more risk tolerant than another, it is largely a qualitative comparison. It is very difficult to measure change in qualitative terms that are universally understood and accepted and not subject to bias, interpretation or hidden agendas.
2. Cultural change—Risk intolerance is often deeply ingrained in government agencies; however, there are a few exceptions. (The US Defense Advanced Research Projects Agency [DARPA] and the US National Aeronautics and Space Agency [NASA] are perhaps the bestknown agencies, but they do not control a large percentage of the US federal budget. NASA’s fiscal year 2020 budget of US$22.5 billion was only 0.4 percent of the total US$4.8 trillion budget.⁵ DARPA’s budget was even smaller, at US$3.556 billion⁶) Encouraging the rank and file to accept more risk would not be effective in the absence of support throughout the enterprise—from top to bottom. Acceptance and support of risk tolerance practices by leadership are critical.

To address these hurdles, the team developed a strategy with eight goals (figure 1).

This is an examination of each goal as they apply to the two major hurdles and a summary of successes to date.

Measuring Risk Tolerance

To address the first two goals—establishing best practices and measuring performance—there had to be a way to quantify the change in risk tolerance. A search for best practices was not very successful. However, CBP already had traditional risk management techniques in place to measure and record risk impact that could provide useful data to relate risk assessment to risk tolerance.

CBP’s Office of Acquisition (OA) encourages all risk practitioners to prepare two impact assessments when deciding whether to address an uncertainty and how much effort to expend on that response. The first assessment measures the impact the uncertainty could have on organizational goals and objectives if nothing is done, which is called the inherent risk assessment. The second assessment measures the potential impact of the uncertainty after treatment is complete. This is called the target risk assessment. By definition, there is always a chance that uncertainty may occur, despite best efforts. The potential impact remaining after treatment is called residual risk.

The inherent risk assessment is used to decide whether to treat the uncertainty at all, that is, whether to make an effort to change the uncertainty’s potential likelihood or impact. Frequently, enterprises establish thresholds of inherent risk that must be crossed before it is considered worth it to organize and execute treatment. Such thresholds define the enterprise’s risk appetite.

As the treatment plan unfolds, impact assessments are conducted to measure progress. These are called current risk assessments. The target risk assessment is used to decide when the treatment can stop. In other words, when the current risk assessment matches the target risk assessment, the enterprise is willing to accept the residual risk and, if necessary, execute any fallback (contingency) actions if the uncertainty does in fact occur.

For example, the OA assesses risk impact qualitatively, using a 1 to 5 scale for both likelihood and impact. A unique number from 1 to 25 is assigned to each cell of the resulting five-by-five probability/impact diagram (PID). In a standard PID (figure 2), threat impacts are designated on the right, and opportunity impacts are designated on the left. For threats, consequences increase moving up and to the right, and they decrease moving down and to the left; inherent risk assessments are closer to the upper right corner than target assessments. For opportunities, benefits increase moving up and to the left; inherent risk assessments are closer to the lower right corner than target risk assessments.

(This numbering scheme is driven by the common belief that all risk is a threat and, therefore, is more conveniently conveyed by positive numbers; opportunities are represented by negative numbers to balance overall risk [threat and opportunity] exposure.)

The difference between the inherent risk assessment and the target risk assessment can be used to indicate the enterprise’s tolerance for a given uncertainty and its potential impacts, and the aggregation of these differences for all uncertainties can be used as an indicator of overall risk tolerance. The greater the difference between inherent and target threat assessments, the less risk tolerant the enterprise. The greater the difference between inherent and target opportunity assessments, the more risk tolerant the enterprise. Furthermore, the differences between inherent and target risk assessments reflect where the enterprise wants to allocate its risk management resources. An enterprise that allocates the majority of its risk management resources to “burning threats down to zero” is the definition of a risk-intolerant enterprise, as this indicates that it cannot tolerate much residual risk.

Using the difference between inherent and target risk assessments to gauge risk tolerance is a concept, not a measurement. Therefore, in the case of CBP, it was necessary to turn the concept into numbers that would be meaningful and easily understood. The first step was to define the differences as a risk rating delta (RRD): RRD = Inherent Risk Rating – Target Risk Rating, where the inherent risk rating and target risk rating are the numbers 1 through 25, taken from the threat side of the PID for threat risk assessments.

Next, an opportunity rating delta (ORD) was defined as ORD = Inherent Opportunity Rating – Target Opportunity Rating, where the inherent opportunity rating and target opportunity rating are the numbers -1 through -25, taken from the opportunity side of the PID for opportunity risk assessments.

Conveniently, both RRDs and ORDs turned out to be positive numbers. Not so conveniently, they did not trend from better to worse in the same direction. To solve that problem, and to give meaning to risk tolerance levels, five levels of both threats (risk) and opportunities (reward) were defined, as shown in figure 3.

Next, RRD and ORD value ranges were assigned to each tolerance level, as shown in figure 4.

Two graphics were designed to enable broader acceptance and implementation. The first, called the risk-reward ratio (R3) (figure 5), simply plots the enterprise’s current threat and opportunity tolerance as a point on a Cartesian grid. The current R3 can be determined from existing risk data using the technique described previously. The ratio reflects the values from the risk and reward tolerance axes. The graph includes a second point representing the enterprise’s desired R3, which is simply a goal for threat and opportunity tolerance. An arrow connects the points. Over time, one would expect the current R3 to move closer to the desired R3 as the enterprise’s risk tolerance changes.

A second diagram called the R3 summation (figure 6) depicts the current R3s for all programs in a portfolio (yellow points) in relation to the enterprise’s overall desired R3 (green point). Again, over time, one would expect the current R3s to move closer to the desired R3 if the enterprise’s overall risk tolerance is changing.

Custom reports were developed using CBP’s enterprise risk management tool to manage and present relevant calculations and translations. The same results can be achieved using an Excel workbook or a SharePoint list and view.

Changing Culture by Changing Minds

Cultural change is challenging; no one likes to fail. Yet innovative enterprises know that failure is a prerequisite to invention.⁷ Changing a culture means changing minds, starting from the top and extending all the way to the bottom.

In parallel with measures and representations, the team addressed the six remaining “soft” goals of the strategy: process, training, awards and rewards, outreach, communication, and policy.

Process
Risk management is a journey, not a destination.⁸ As such, it requires the same continuous attention as any other element of management. However, risk management has the ability to consume resources long past the point of diminishing returns. Risk intolerance drives enterprises to attempt the impossible: reducing the potential impact of uncertain future events and conditions to near zero. Unrealistic optimism, poor planning and many other factors drive enterprises to pursue opportunities after the need is gone or after the likelihood of success has been all but erased. The risk management highway needs off-ramps. As CBP’s Component Acquisition Executive succinctly stated, any risk management process that attempts to shift resources from threat mitigation to opportunity promotion needs to follow the guideline “fail fast and cheap.” The ART process the team developed (loosely modeled after Agile software development processes) provides these checkpoints, or offramps; it also includes on-ramps within the process to enable rapid and continual improvement. This process, shown in figure 7, is the key to implementing CBP’s agile risk tolerance method.

Training
For several years before DAU’s study and the initiatives it spawned, one team member had regularly presented “lunch and learn” seminars and workshops on a variety of risk management subjects. This forum provided a natural opportunity to address the strategy’s training goals. Beginning in January 2018 and continuing to the present day, the team has conducted 24 seminars with the CBP operations, procurement and acquisition communities to introduce and reinforce ART principles and practices.

These monthly workshops are useful, but they are not a targeted delivery system. Beginning in 2020, the team began scheduling introductory presentations with specific groups, with the goal of reaching all CBP acquisition organizations before the end of the year.

Awards and Rewards
To encourage acquisition professionals to adopt risk tolerance principles and practices, the team developed incentive programs that reward risk tolerance in CBP programs. This addresses the second DAU recommendation with regard to rewarding innovation. Each program is encouraged to build such rewards into its traditional reward and recognition practices and traditions. People can receive both on-the-spot awards from their supervisors and committee-evaluated awards (e.g., given by the Joint Awards Committee). Programs can also receive “savings carry-over” awards based on quantifiable savings realized through risk tolerance practices.

Outreach
After the ART directive was officially promulgated throughout CBP, the team reached out to program offices with targeted training, helping them prepare for portfolio reviews and collecting metrics to gauge their success in meeting or exceeding schedules and reducing costs. To date, the team has provided training in the ART method to more than 350 agency personnel.

Communication
Continuous communication with both leadership and the workforce is another important component of the strategy to change culture. This follows the wisdom of an old proverb: The drop does not carve the stone with force but with the steady dripping. In this spirit, team members have published ART articles in various newsletters and office publications and intend to continue to do so, as well as reporting on ART metrics and successes.

Policy
Formally setting expectations may not be the last step, but it is an important step. The team authored a CBP-level directive in late 2018, signed by CBP’s chief acquisition executive (CAE), chief information officer (CIO) and the head contracting authority (HCA), which established that “…all offices and organizations shall integrate an agile risk tolerance process…into their acquisition management practices and procedures.”⁹ Since then, agency acquisition portfolio reviews—which occur roughly every six months and include approximately 10 percent of active acquisition programs and projects—have required presenters to report on current and desired risk and reward tolerance levels. This directive also established the position of chief risk officer (CRO), charged with the overall responsibility of leading the ART endeavor within the agency.

CONTINUOUS COMMUNICATION WITH BOTH LEADERSHIP AND THE WORKFORCE IS ANOTHER IMPORTANT COMPONENT OF THE STRATEGY TO CHANGE CULTURE.

Conclusion
CBP has completed almost two years of ART formation, introduction and use within the agency. During that time, CBP has published an implementing directive, developed and continued ART training, established relevant metrics, and started tracking metrics on a quarterly basis. There is now tangible evidence that CBP’s ART principles and practices are beginning to take hold in risk management plans and portfolio reviews. The team’s declared intent, as supported by agency leadership, is to increase innovation and acquisition efficiency and effectiveness. Cultural change takes time; however, CBP’s ART team is confident that with the foundations established to date, ART principles and practices will lead to the achievement of the agency’s acquisition and procurement goals.

Endnotes

¹ Defense Acquisition University (DAU), Outbrief— Review of Customs and Border Protection (CBP) Acquisition Program, Fort Belvoir, Virginia, USA, 2015
² Standard College Dictionary, Funk and Wagnalls, USA, 1963
³ International Organization for Standardization (ISO), ISO Guide 73:2009 Risk Management— Vocabulary, Switzerland, 2009
⁴ Committee of the Sponsoring Organizations of the Treadway Commission (COSO), Enterprise Risk Management—Integrated Framework: Executive Summary, USA, 2004
⁵ Amadeo, K.; “NASA Budget, Current Funding, History, and Economic Impact,” The Balance, 30 August 2020, http://www.thebalance.com/nasa-budget-current-funding-and-history-3306321
⁶ Defense Advanced Research Projects Agency (DARPA), “Budget,” March 2019, http://www.darpa.mil/about-us/budget
⁷ Farson, R.; R. Keyes; “The Failure-Tolerant Leader,” Harvard Business Review, August 2002, http://hbr.org/2002/08/the-failure-tolerant-leader
⁸ Knight, K. W.; “Risk Management—A Journey, Not a Destination,” RusRisk/Marsh Seminar, Moscow, 15 December 2010, http://www.scribd.com/document/283750131/A-Journey-Not-a-Destination-pdf
⁹ Office of Acquisition, Directive 5220-045, “Agile Risk Tolerance,” US Department of Homeland Security/Customs and Border Protection, October 2018

Steven Moyer, Ph.D

Is acting chief engineer and chief risk officer for US Customs and Border Protection (CBP). He previously worked at the US Army Night Vision and Electronic Sensors Directorate in the Modeling and Simulation Division.

Gunter Brunhart, Ph.D.

Is a nuclear physicist. He joined US CBP in 2007, serving as technical director and branch chief for engineering management (including risk management) in the Systems Engineering Directorate of the Office of Acquisition. Before joining CBP, he spent many years conducting basic nuclear research and biomedical radiation research and developing US Navy survivable communications in nuclear war scenarios.

Richard Dubs, Ph.D.

Is a subject-matter expert in the Workforce and Knowledge Management Division of the US CBP Office of Acquisition, where he develops tools to predict and analyze workforce needs and career path models for various CBP acquisition disciplines.

Thomas Erickson, PgMP

Is a support contractor to the chief engineer and chief risk officer for US CBP and to all CBP offices for matters related to risk management. He is a retired US Air Force acquisition officer and a registered professional electrical engineer.

Robert Skalamera

Is a support contractor to the chief engineer and chief risk officer for US CBP. He previously served as director for systems engineering policy in the Office of the US Secretary of Defense. He is a member of the Institute of Electrical and Electronic Engineers and the International Council on Systems Engineering.

Rob Kepner, PMP

Is a registered professional engineer and acquisition system engineering professional at the MITRE Corporation, supporting the US CBP’s chief engineer and chief risk officer. He has worked with many federal clients to establish acquisition policies and practices and program management and systems engineering offices.

Marty Meyer

Is an engineering subject matter expert supporting the CBP Office of Acquisition Systems Engineering Director. Before joining CBP in 2010, he worked as a nuclear detection systems engineer for the US Domestic Nuclear Detection Office and as an engineer, project manager and test director for numerous US Department of Defense advanced technology and robotics programs.