Risk Management

Maximizing the Value of Schedule Risk Assessments (SRAs)

, , , , , , , ,

Quick Summary

  • SRAs are only as valuable as the intent and quality behind them. Treating them as a check-the-box compliance requirement produces meaningless results that get filed away and never used.
  • Real-world examples show how cost pressure and last-minute guidance changes at kickoff can derail an SRA, leaving project teams with outputs no one trusts or acts on.
  • Focusing three-point estimates on tasks that drive critical and near-critical paths delivers real insight without the prohibitive cost of analyzing every task in a large IMS.
  • Documented assumptions for selected tasks produce derived duration factors that are far more accurate than gut-feel percentages, and the difference shows up directly in the SRA results.

A previous blog, Schedule Risk Assessment Fundamentals, highlighted that when properly used, schedule risk assessments (SRAs) are a powerful management tool that can contribute to project success. It does, however, require a high-quality integrated master schedule (IMS) that is integrated with a disciplined risk management process. When conducted with the intention to gain a realistic view of schedule outcomes and risk drivers, it provides management with additional insight and an early warning indicator of potential threats to meeting schedule objectives.

The Challenge with Conducting SRAs

Schedule risk assessments are frequently a topic of discussion in project management circles and sometimes for the wrong reason. A well-run SRA can provide the confidence levels for achieving different schedule end dates as well as valuable insight into the IMS. However, the SRA is particularly susceptible to the “garbage in – garbage out” (GIGO) principle.

H&A consultants have recently observed discussions that indicate SRAs are not being properly performed or used to help manage projects. Some of these observations reinforce the GIGO principle.

One H&A senior consultant sat in on a meeting with the government program manager for a large project where the consultant, having worked on the just-completed SRA, asked the government program manager what they would do with the results. The amazingly honest answer was, “Sadly, it will just be filed. It is seen as a check-the-box thing we have to do.”

Using the results of the SRA to stuff the drawer might not be as wild a response as it seems. Too many times, we see the SRA being done in a perfunctory manner using inputs that will not yield useful management information and insight. One way people are cutting corners on the SRA is by applying global factors to the existing duration estimates and running the simulation with those durations. Applying a formula across the board is not the same as analysis. Yes, the SRA can be done so poorly it is meaningless. This is especially true if the applied factors are not realistic.

An Example of What to Avoid

One of our consultants observed that in one case the best of intentions were thwarted in the SRA by cost pressures and lack of management commitment. The manager of the SRA provided a form to each CAM for them to provide the best case, worst case, and most likely case duration estimates and provide written explanations for all three cases. The instructions were for the CAMs to use the form for every incomplete task and future task.

There were about 10,000 such tasks distributed among 70 or so CAMs. On average, that would be 140+ tasks for each CAM. Doing three-point estimates for 140 tasks would be a large expense and consume a lot of valuable time, without even mentioning the cost.

The government program manager, who pays the bills, was present at the kickoff meeting for the SRA and intervened immediately when he heard the directions being provided. He stated that he would not pay for all that effort; it would be too costly. Unfortunately, there was no probing to find out what would be reasonable to this manager in terms of details for the SRA.

Instructions for this SRA were revised on-the-fly and the SRA was done. It was done poorly; in some parts due to the disruption at the kickoff meeting and the poor guidance. The intervention of the government program manager had left the impression that the SRA “was not worth it.” That impression was wrong.

The idea of documenting the three-point estimates is a good one, but too time-consuming to be applied to every task. There are valid options that can still benefit from this detailed look at the durations, yet avoid the significant expense of analyzing all the tasks.

A Better Approach

One approach is to do a detailed analysis of the three-point estimates for specific selected tasks that reduces the number of tasks that require detailed manual estimation. The focus should be on the tasks that provide insight into the part of the IMS most likely to cause the end date or a major event date to change. Examples include those tasks that are:

  1. On one or more of the top number of critical paths.
  2. On the path to the next major event (these tasks can be found using the driving path approach).
  3. Known to be or assumed to be prone to duration risk.
  4. Known to require scarce resources that may have limited availability.
  5. Believed to be drivers of duration risk for other reasons.

A simple example will help to understand why documenting the three-point estimates for some tasks is basic to achieving a useful SRA. If you, acting as a CAM, were asked to estimate the time required to drive 10 miles from your home to work by car in a hypothetical town, you would want to understand the scope of the effort. A drive of 10 miles through the countryside, or on a freeway, or on city streets can be very different.

stylized road map of confusing roads

Some help with the assumptions could improve your estimate. If you were told that the first 5 miles of the drive was in town, where the speed limit is 25 mph with the potential for red lights along the way, and the last 5 miles is on the freeway, where the speed limit is 60 mph, this would help you produce a better estimate.

In fact, you will get a better estimate by understanding the scope, the assumptions, the risks, and so on. When doing the three-point estimate, you would employ a process that includes these steps that consider:

  1. The nature of the task. What exactly is to be done in the task?
  2. Past experience. Have we done this or similar work before?
  3. Capabilities. For example, can you drive, does your car work well, can you go up to 60 mph in your car, do you have gas, and are you equipped for potential weather?
  4. Assumptions. How many traffic lights are there along the way? How long would you wait at a red light? What time of year is it? What day of the week? What time of day?
  5. The risks. Are there possible road issues, such as construction? How about traffic issues? Accidents?

Now, suppose you were required to document your estimated durations. Using the assumption details from above, you might end up with this:

CaseDurationAssumptions
Best17 minutesA dry day, early in the morning before traffic, you have all green lights, and you obey the speed limit.
Most Likely23 minutesThe road is fine, first 5 miles is at 25 mph. Only 2 red lights with a wait time of 2 minutes each, and 7 minutes for freeway travel at 45 mph.
Worst Case32 minutesThe road is slippery and you can only travel at 20 mph. You must stop at 3 red lights and sit for 2 minutes each. You have one unexpected stop for 4 minutes because of other drivers. Also, the freeway speed is only 45 mph.

You now have a set of durations you can use in the SRA. You also have the details needed for explaining the duration estimate. Additionally, there is enough information to be able to change the estimates if presented with new facts or revised assumptions. For example, perhaps the project’s period of performance moves to the right and the work will now be performed in the winter. You can adjust your estimates for winter weather impacts.

You also have enough information to be able to derive factors to be used in formulae to generate three-point estimates for other tasks. Be careful to make sure you only use the information to generate a factor on similar work. In this case, the factors would be -26% and +39%. Those are derived factors. For comparison, a common “gut feel” reaction to the question of what factors should be used is usually more like minus 5% and plus 10%. Using derived factors versus a “gut feel” will yield significant differences in the SRA and the value of the results.

Recommendations to Increase the Value of an SRA

Begin with the intention to treat the SRA as the important and valuable tool it is. Choose to change the approach from a “check-the-box” or compliance mentality into a straightforward process that helps to produce a more realistic and executable IMS. No one likes schedule or resulting cost surprises as discussed in another blog, Maintaining a Credible Estimate at Completion (EAC), that also addresses why a credible forecast completion date is equally important. Here are a few suggestions to improve your approach to conducting SRAs:

  1. Provide clear, specific directions to project personnel on what is expected. Highlight why the SRA is an important step.
  2. Verify a quality IMS has been established.
  3. Validate the risk information.
  4. Do focused analysis of discrete tasks on a given number of critical and near-critical paths and document the rationale for the best/worst/most-likely case durations.
  5. Do focused analysis on driving path tasks if not on the critical paths.
  6. Do focused analysis on known risks.
  7. Use realistic factors derived from reality when applying factors to the larger body of IMS tasks.

Taking Action

Producing a quality IMS takes skilled master schedulers that understand the management and predictive value of a well-constructed schedule. The next step up to improve the realism of the IMS is to conduct an SRA when it makes sense. Examples include conducting an SRA as part of the process to establish the baseline schedule, when there is major change, or before a major event such as a Critical Design Review (CDR).

It is not an easy task to distill the steps to conduct a value-added SRA into a well-defined and useful process. H&A master schedulers and risk subject matter experts often work with clients to establish a pragmatic SRA process. They also train and mentor project teams on how to use the SRA outputs to produce realistic schedules with a higher probability of success. Call us today to get started.

Maximizing the Value of Schedule Risk Assessments (SRAs) Read Post »

Maximizing the Value from Integrated Baseline Review (IBR) Investments 

, , , , , , , ,

A previous blog, How Integrated Baseline Reviews (IBRs) Contribute to Project Success, provided an overview of the purpose and scope of IBRs as well as the benefits of conducting an IBR. This blog adds to the discussion on the benefits of conducting an IBR. It reflects observations gathered from our earned value consultants while assisting clients to prepare for IBR events

As a reminder, IBRs provide the opportunity to verify the:

  • Contractor and the customer have a common understanding of the scope of work, technical requirements, and accomplishment criteria. 
  • Contractor has established an executable performance measurement baseline (PMB) for the entire contractual scope of work that accurately reflects how they plan to accomplish the work within the contractual period of performance, negotiated contract cost, and funding profile. 
  • Required resources have been identified and assigned to the project to accomplish the project’s objectives. For example, the staffing plan accurately reflects the sequence of work as well as resource availability and demand.  
  • Technical, schedule, and cost risks/opportunities have been identified, assessed, and captured in a risk/opportunity register. Risk mitigation actions have been incorporated into the PMB to reduce known threats to an acceptable level. This is often the most valuable component of the IBR to ensure all parties have an understanding of the risks/opportunities, assumptions, and risk mitigation or opportunity capture plans. 

Factors that Contribute to a Successful IBR

Treating an IBR as just a contractual requirement limits its value to all parties. IBRs are essential to the successful execution of any project. IBRs require a focused mindset to clearly define as well as assess the measurable benefits gained for the time and effort invested in the IBR. From our observations, contractors that defined what they expected to gain from an IBR, whether the IBR was contractually required or not, made a measurable difference in the outcomes from the IBR. The effectiveness of an IBR is contingent upon management’s commitment to excellence in implementing their EVMS and their desire to ensure they have reliable and useful data for management visibility and control. And that begins with establishing an executable PMB. 

The following list of factors often influence the perceived value of an IBR and hence the approach a contractor takes to planning and conducting their IBRs. 

  • Recognizing the relative importance of the review.
  • Defining the value or measurable benefits they expect to gain from conducting the review.
  • Well defined risk/opportunity management process. 
  • Timely and sufficient review planning and preparation.
  • Joint or collaborative planning and preparation.
  • Well defined objectives as well as entrance and exit criteria. 
  • Tailoring the IBR approach to best accomplish the review objectives.
  • Communication and expectation management.

These factors were ultimately indicative of whether the IBRs were considered value-added (retrospective assessment by the participants) based on the level of understanding, investment in or attention to, or the degree of success in implementing these factors. Based on H&A earned value consultant’s observations, the single factor that tends to drive the IBR approach is clearly defining the value the contractor expects to gain beyond what is mandatory or contractually required. 

IBR Investment Value

The term “IBR investment value” is purposefully used here. The intent is to invite you to re-assess how IBRs are viewed apart from simply meeting government agency IBR requirements. “IBR investment value” is used to mean a qualitative assessment that encapsulates the value-add or measurable benefits teams often have difficulty defining as well as to help provide the impetus and guiding direction for conducting an IBR. It has both intrinsic and extrinsic properties. 

The intrinsic value of the IBR investment resides in those specific elements of information (as identified by the customer in the form of questions or concerns) that are either exchanged, clarified, or refined through the course of discussions between the customer and performing contractor teams. This intrinsic value can be measured by how well the exchanged information supports:

  • A complete, clear and mutual understanding of the work to be accomplished.
  • The resources needed to get the work done.
  • The detailed plan to perform the work.
  • What resources are available to support the plan.
  • What’s missing or unknown that is needed to complete the work correctly and on time.
  • What risks, issues, concerns, or opportunities are associated with contractor’s concept that need to be fully considered to make the plan work. 

The extrinsic value of the IBR Investment rests wholly in the quality of the exchanges (discussions), and the resulting actions generated from the discussions. This extrinsic IBR value addresses how appropriate, rich and comprehensive the information exchanges were, and answers to questions, such as:

  • Were the discussions responsive to a list of customer information requirements and concerns? 
  • Were the right discussions held? At the right level of detail?
  • Were the right people involved in each discussion? 
  • Did the discussions provide sufficient context? Were they comprehensive? Complete?
  • Did the discussions address associated risks, issues, opportunities or other concerns? Relationships to other discussions/elements?
  • Were all the customer’s questions or concerns answered to their satisfaction?
  • Were the discussions documented to support decisions? Alternatives? Changes? Studies?

The exchanges of essential information (intrinsic value) and the quality of those exchanges (extrinsic value) when combined directly translate to the investment value achieved from the IBR. It characterizes how well the information exchanged provides both teams with the necessary details to successfully define, schedule, budget, and manage the contracted effort relative to the investment into the IBR process. A realistic, risk adjusted PMB helps to prevent schedule delays and cost overruns during project execution that often impact a contractor’s profit margins and tarnishes their credibility with their customers. 

What are the characteristics of a value added IBR approach?  

A successful approach H&A earned value consultants have observed contractors implement is a structured process corporate management actively participates in to ensure they gain the most value from all IBR events. 

This is often an outgrowth from corporate initiatives to retain top project management talent and establishing an EVMS self-governance process. It is part of a corporate culture that is committed to excellence in project management and sustaining a best in class EVMS – becoming efficiently expert at EVM

What are some common characteristics of their IBR approach?

  • A chartered authority or corporate team responsible for assisting project personnel with IBR events in addition to EVMS implementation, self governance, and customer surveillance events. A good practice we have seen implemented is to establish rotating members on the IBR teams from different projects as a means to pollinate best practices across projects. It also provides an opportunity to mentor top talent on track to move up to higher management positions.  
  • A standard repeatable process with defined measurable outcomes that can be tailored to the unique project requirements or objectives. This includes maintaining a set of materials for the internal IBR team to effectively plan and execute an IBR as well as to close out any action items. Examples include training materials to prepare project personnel, process description with team member roles and responsibility assignments, data call list, role based interview question forms with assessment criteria, data quality assessment materials and tools, list of data traces to be performed, schedule risk assessment tools, risk/opportunity evaluation criteria, defined assessment criteria (technical, schedule, cost, resources), in-briefing and out-briefing templates, and template to capture action items to track to closure. The corporate team is often responsible for actively maintaining the content for the IBR teams and conducting training. 
  • They place an emphasis on two components that directly impact the quality of the schedule and cost data.  This includes:
    • Well-documented data driven basis of estimates (BOEs) that can be substantiated using historical or bench-marked data with the goal of reducing expert judgement cost estimates to the lowest level possible as a risk reduction strategy.  
    • The quality of the risk/opportunity management plan and the content in the risk/opportunity register. This content directly affects the ability of all parties to gain a better understanding of the risks/opportunities and best options to mitigate a risk or capture an opportunity. A well constructed schedule is required to be able to perform schedule risk assessments (SRAs). SRAs help to identify where duration risk exists in the schedule and to determine a level of confidence in meeting major project milestones as well as the project completion date.  
  • They perform internal IBRs as a standard practice on all projects regardless of contractual requirements. This is particularly important when subcontractors are performing a substantial percentage of the work effort. The corporate team often assists Project Managers with conducting a joint IBR with major subcontractors.  

Need help establishing a corporate IBR process?

H&A earned value consultants often help clients to establish a corporate EVM council or center of excellence with defined responsibilities to ensure project personnel effectively implement their EVMS, integrate risk/opportunity management into the EVMS, as well as define and implement a standard repeatable process for IBRs and self-governance. Clients often need assistance establishing a repeatable process for conducting schedule risk assessments, an essential component of the IBR process. A defined process that clearly articulates the expected measurable outcomes from conducting IBRs is one way to ensure all parties gain the most value from the event with the end objective of ensuring a realistic and executable PMB has been established.  

Call us today to get started.  

Maximizing the Value from Integrated Baseline Review (IBR) Investments  Read Post »

EVM and Unified Risk Management

, , , ,

Working with numerous clients, H&A earned value consultants have observed many instances where project management teams consider the risk and opportunity (R&O) management process to be something technical in nature, run by engineers and focused on the technical aspects of the project’s product. Meanwhile, there is often a separate risk process going on much less formally to consider risks in terms of the project’s schedule and cost goals. This bifurcated approach is a source of risk itself.

Procuring agencies such as the DoD, NASA, DOE, and others have published their own risk management guides. The Government Accountability Office (GAO) has various reports on this topic including examples of their findings. DCMA mentions risk in their Business Practice 4  Guideline Evaluation Template (GET) Process/Implementation Verification Points often used by contractors to check whether their earned value management system (EVMS) meets the intent of the EIA-748 Standard for EVMS guidelines. The exact questions asked by DCMA are important but the overall idea that risk and EVMS are co-dependent is the critical aspect. This is also true for the DOE. They identify risk management as one of the 10 subprocesses necessary for an EVMS.

Setting the Stage

Risk is defined as a factor, element, constraint, or course of action that introduces an uncertainty of outcome that should it occur, could negatively impact the ability to meet the project’s planned technical, schedule, or cost objectives. Negative impacts are sometimes called a threat where the objective is to mitigate the risk. A realized risk becomes an issue that must be resolved to minimize the impact. An opportunity is defined as a positive risk where the objective is to capture the beneficial impacts. Opportunities are not as common as threats.

R&O management is defined as the process of identifying, assessing, and responding to risks and opportunities throughout the project’s life cycle. The goal of R&O management is to identify potential risks and opportunities, determine the likelihood or probability the risk or opportunity will occur, and determine the impact should a risk be realized, or an opportunity is captured. Risks and opportunities are prioritized so that those with greater impact and a higher probability of occurring receive a greater share of resources and attention.

In this blog, we are using the term risk with a focus on the negative impacts or threats to a project.

Example of Common Project Risks and Risk Assessment Approach

H&A’s senior management routinely reviews literature, considers our work with clients, and discusses with our earned value consultants the main contributors to project failure. These findings are updated regularly and presented in H&A training materials as an Ishikawa Fishbone Cause and Effect diagram. Figure 1 is an example of this type of diagram. 

Figure 1: Example of an Ishikawa Fishbone Case and Effect Diagram

Figure 1: Example of an Ishikawa Fishbone Case and Effect Diagram

When this approach is used for risk assessments, each contributing risk is assessed, and the response documented. An example of a risk/response table is shown below for the first three identified risks.

Risk ItemGood Example of a Real Project Response to an Identified Risk
Poor communicationsGoals are known and documented. Communications plan is in place. Have an established cadence for weekly internal and customer meetings to quickly resolve issues. An internal project performance management dashboard is updated daily with current data. Updated IMS and risk register are broadcast weekly to the team. A strong business rhythm has been established.
Scope creepWork scope (requirements and SOW) are well defined and a change control process is in place. Performers are trained in spotting scope creep and how to handle potential changes in scope.
Inaccurate cost estimateImplemented a process enabling cost estimators to search historical actual cost data, identify analogous tasks, substantiate, and document the basis of estimate. For high risk areas, techniques such as the Delphi method, SMEs, and non-advocate reviews are used. Performance is constantly monitored to spot work elements where the actual costs do not align with the budgeted costs or the estimate at completion (EAC) is triggering internal variance at completion (VAC) thresholds. 

This same type of approach can be used by the project control team to create risk Ishikawa diagrams to identify technical risks that could impact the ability to achieve schedule and cost goals. Likewise, risk Ishikawa diagrams can be used to identify risks in the integrated master schedule (IMS) and time phased budget or estimate to complete (ETC) and EAC.

A Unified Approach to Risk

A unified approach includes technical, schedule, cost, and other risk identification and assessment that is an integral part of a contractor’s EVMS. R&O management should be integrated into the EVMS subsystems including work organization, planning and scheduling, work authorization and budgeting, management analysis and reporting, and change management. 

Identified risks are analyzed and quantified to develop a risk handling strategy. Where applicable, risk mitigation tasks have been entered into the IMS. Ideally a schedule risk assessment (SRA) has been completed to gain an understanding of duration risks that can help to improve the accuracy of the schedule. Assuming the IMS is resource loaded and leveled, the result is a more accurate time phased budget plan as it incorporates the risk handling strategies when the performance measurement baseline (PMB) is established. The R&O process also provides the necessary rationale for determining the budget amount set aside for management reserve (MR).

The R&O assessments should be a normal part of generating the Variance Analysis Reports (VARs) and updating the ETC and EAC. These assessments can also drive the need for processing baseline change requests (BCRs) as well as determining the best approach for corrective actions. 

Using Directed Searches of Identified Risks

To facilitate a unified approach, we recommend establishing a cadence of standing risk review sessions that are conducted in a methodical way to ensure the project manager, integrated product team (IPT) leads, control account managers (CAMs), schedulers, and financial analysts routinely walk through the identified risks that have the potential to impact the project’s IMS or time phased cost.

The intent is to establish a framework such as Ishikawa diagram to guide the risk review session, a directed search of the identified risks should anything further need to be addressed. It is important that a “does anyone have a risk to suggest” approach is not used. Every topic should be covered in every session by walking the Ishikawa risk items. Most of the time it will be a quick “no change” response. Separate Ishikawa diagrams could be used to guide the discussions for the contributing technical, schedule, and cost risks. The meeting room should have the ability to view the live IMS, cost data, and performance analysis data. Team members should be prepared to take notes during the meeting to compile action items.

Figure 2 is an example of a basic Ishikawa diagram of IMS risks the project control team could focus on for the risk review session. This would reflect the project control team’s identified risks to the IMS they routinely monitor.

Figure 2: Example of an IMS Ishikawa Fishbone Case and Effect Diagram

Figure 2: Example of an IMS Ishikawa Fishbone Case and Effect Diagram

For example, updating the current schedule every reporting period has the potential to compromise the integrity of the IMS to provide accurate forecast information about the project’s remaining work. Perhaps the project control team has identified a list of contributing schedule status risks, risk response, and example directed questions for each review meeting. These questions could be focused at the CAM level. The following table is a simple example. 

Risk ItemRisk ResponseExample Directed Questions
IMS critical or driving pathsVerify logic. Verify traceability exists and has not been damaged by updates. Review constraints, deadlines, and milestones. Perform data quality check, correct errors.Did milestones move? Did the end date move? What were the baseline dates for starts or finishes that fall into the period?What were the forecasted dates for starts and finishes that fall into the period?What did not happen? Why?
RealismCalculate and assess the Baseline Execution Index (BEI) and Current Execution Index (CEI). Compare the ratio of actual performance to the ratio of future performance.Is the BEI/CEI result within goals? Are there performance discrepancies? Does the forecast need to be updated to align with reality? Is the forecast showing the performance the team can achieve based on what has been achieved?
Quality of ETC/EACVerify updates are occurring. Compare current ETC/EAC to previous ETC/EAC.Has the ETC been updated? What changed and why? For example, for activities with material requirements, price or usage variances may impact the ETC/EAC. For activities with labor requirements, availability or personnel changes may impact future work effort ETC/EAC.

The same approach would be used for guided budget and cost risk discussions. Tailored cause and effect diagrams should be created for a company business environment and each project’s unique characteristics.

Interested in learning more?

H&A’s training courses purposely include content on R&O management and integrating it into the EVMS. H&A’s Project Scheduling as well as Advanced Earned Value Management Techniques (AEVMT) workshops in particular include more discussion on R&O topics.

A company’s EVMS should be designed to aid the identification and management of risks and opportunities. For example, during the process of developing the schedule and budget baseline, activity durations, resource requirements, and budget distribution can be refined to reflect identified and assessed risks. Proactively identifying and managing risks improves project performance. The expectation of specific risks occurring leads to contingency plans that lower the likelihood and impact of risks as well as the establishment of schedule margin and MR to address identified and assessed risks.

Call us today at (714) 685-1730 to get started.

EVM and Unified Risk Management Read Post »

Scroll to Top