Incorporating IMS Information Directly into Independent Estimate at Completion (IEAC) Formulas

By / / , , ,
, , ,
Incorporating IMS Information Directly into Independent Estimate at Completion (IEAC) Formulas

“When you need to discuss the schedule, look at the schedule.”

– A Scheduler’s Lament

There are many existing formulas for calculating an Independent Estimate at Complete (IEAC) from earned value data. A recent study of a sample of projects found that the calculated IEACs analyzed at the 25%, 50%, and 75% complete points were not accurate when compared to the final actual cost of work performed (ACWP). The following table lists the thresholds used to assess the accuracy of the IEACs at the different complete points for the sample projects.

Percent CompleteAccuracy Threshold
25%Within +/- 10% of final ACWP
50%Within +/- 7% of final ACWP
75%Within +/- 5% of final ACWP

While working on that study of the accuracy of commonly applied IEAC formulas as well as on a small project as an analyst for a customer, the idea for using data directly from the integrated master schedule (IMS) in conjunction with the cost performance data to create a new IEAC formula emerged.

Using Data Directly from the IMS to Calculate an IEAC

It should be noted that none of the generally used IEAC formulas use data directly from the IMS. The IEAC formulas use data found in the cost performance portion of the earned value monthly reports to customers.

IMS data is only used indirectly in the IEAC formulas. When a task is started and progress updated, the earned value (the budgeted cost for work performed or BCWP) is developed from the progress reported. This is measured against the cost baseline (the budgeted cost for work scheduled or BCWS).

At the same time, in the IMS environment, the schedule analysts are calculating the Baseline Execution Index (BEI) for task completions/finishes. BEI (for finishes) measures how many of the tasks baselined to be completed by the cut-off date were completed. If all the tasks were done (BEI = 1), their value would have been earned. Of course, other tasks could have started, progressed, and maybe even finished. For this example, the Schedule Performance Index (SPI) calculated at that point (BCWP/BCWS) should be at least 1 and potentially higher. The SPI reflects the baseline value of completed tasks plus the in-process claimed baseline value. The in-process claimed value can be subjective in some cases.

The argument, if there were one, might be there is no need to try and include BEI or similar schedule measures in the IEAC formulas since they already include SPI.

However, there is a whole different and unique set of information coming from the IMS that is not currently used in the IEAC formulas. That information is what we chose to call “Duration Performance” and “Realism Ratio.” These are measures of the actual duration for completed tasks and the forecast duration for future tasks.

Calculating Duration Performance

The IMS data includes the baseline number of days assigned to each task as well as the actual number of days to complete each task. If a task is baselined to take 10 days (Baseline Duration = 10) and the task took 15 days to complete (Actual Duration = 15) then it is taking 150% of baseline to do the work.

This is similar to the Cost Performance Index (CPI) that uses the BCWP and the ACWP to determine how efficient the work performance has been. The formula BCWP/ACWP shows how the work accomplished compares to the cost of that work performed.

If we assume, for labor at least, that taking longer to complete a task often leads to costing more than baselined, we can use the Duration Performance to develop an IEAC.

To develop the Duration Performance, we would use the IMS from the month being analyzed to perform the following actions:

  1. Filter out all summary tasks and look only at real work tasks.
  2. Decide what to do with level of effort (LOE) – keep it or ignore it.
  3. Filter for all tasks that are completed (100% complete).
  4. Add up the baseline duration in days for all these completed tasks.
  5. Add up the actual duration days for these same completed tasks.
  6. Compare the actual duration days used to the baseline duration days.

An example would be:

  • 100 completed tasks
  • Total baseline days duration = 1,000
  • Total actual days duration = 1,500
  • Duration Performance = 1,000 / 1,500 = .67

One of the common IEAC formulas is the “SPI times CPI” that is calculated like this: ACWP + Budgeted Cost of Work Remaining (BCWR) / (CPI x SPI) where BCWR = Budget at Completion (BAC) – cumulative to date BCWP.

Now that we have a duration performance factor, we can develop a new IEAC. The Duration Performance IEAC would be done using the CPI from the same month as the IMS where ACWP + BCWR / (CPI x Duration Performance Index).

Using some actual data from a project for a single month we see:

  • Duration Performance Index = .82
  • BEI = .72
  • CPI = .92
  • SPI = .94 (significantly higher than the BEI)
  • ACWP = $9.2M
  • BCWR = $18.3M
  • IEAC using standard formula with CPI x SPI = $9.2 + $18.3 / (.92 x .94) = $30.3M
  • IEAC (Duration Performance) = $9.2 +$18.3 / (.92 x .82) = $33.5M

Assessing the Realism Ratio

When we look at the remaining tasks to be completed, we can use the Realism Ratio to assess how the future forecast durations compare to the performance so far.

The data needed are the baseline duration and the forecasted duration for all tasks that have not been started. This concept excludes in-process tasks. In our example from before, the data we created looked like this:

  • 100 completed tasks
  • Total baseline days duration = 1,000
  • Total actual days duration = 1,500
  • Duration Performance = 1,000 / 1,500 = .67

We would use the same IMS to do this:

  1. Filter out all summary tasks and look only at real work tasks.
  2. Decide what to do with LOE – keep it or ignore it.
  3. Filter for all tasks that are not started.
  4. Add up the baseline duration in days for all these tasks not started.
  5. Add up the forecasted duration days for these same tasks not started.
  6. Compare the forecasted duration days to the baseline duration days.

Let’s say there were 100 tasks not started. If the forecasted days were 1,000 and the baseline days were 1,000 that would yield 100%. When we did the example, the Duration Performance was .67. This means that performance to date was .67 but the future will be 100% or 1. You can see the disconnect. That disconnect we call the Realism Ratio (in this example, .67/1).

Data from the actual project for the same month as discussed earlier shows:

  • Duration Performance = 122% of baseline
  • Future Performance = .86 or 86% of baseline.

This means that the future durations are cut significantly.

We would use this data to develop a factor called a Realism Ratio (86/122 = .70) and that would be used to develop an IEAC using this formula: IEAC (Realism Ratio) = ACWP + BCWR / (CPI x Realism Ratio).

Using the same sample project data from above and adding in an assessment of the forecasted durations for the remaining work, we see:

  • Duration Performance = .82
  • BEI = .72
  • CPI = .92
  • SPI = .94 (significantly higher than the BEI)
  • ACWP = $9.2M
  • BCWR = $18.3M
  • Realism Ratio = .70
  • IEAC using standard formula with CPI x SPI = $9.2 + $18.3 / (.92 x .94) = $30.3M
  • IEAC (Duration Performance) = $9.2 +$18.3 / (.92 x .82) = $33.5M
  • IEAC (Realism Ratio) = $9.2 +$18.3 / (.92 x .70) = $37.6M

The project is not complete, so the final ACWP position is not known. There is a dramatic difference between the three IEACs. The difference between BEI and SPI indicates that in-process tasks and other factors such as LOE are potentially affecting SPI.

What can we learn from this sample project?

In this example, additional investigation is warranted. There are potential issues with the realism of the baseline and current schedule that are signaling a cost growth issue is likely to occur. Relying on just the time-phased cost data for IEAC calculations may not be sufficient to assess whether a contractor’s range of EACs included in their monthly cost performance reports are realistic. For more discussion, see the blog on Maintaining a Credible Estimate to Completion (EAC) and the blog on Using EVM Performance Metrics for Evaluating EACs.

Are there lurking cost growth surprises in your projects? You may want to consider revisiting your estimate to complete (ETC) and EAC process to verify there is an integrated assessment of the schedule and cost data to identify potential disconnects. H&A earned value consultants can provide an independent assessment of the quality of the data as well processes and procedures to help you verify your EACs are realistic. Call us today at (714) 685-1730.

Incorporating IMS Information Directly into Independent Estimate at Completion (IEAC) Formulas Read Post »

Establishing a Robust EVMS Self-Governance Process

By / / , , ,
, , ,
Establishing a Robust EVMS Self-Governance Process

A previous blog, Benefits of an EVMS Self-Governance Process, discussed why establishing a self-governance or self-surveillance process is important and how an effective process builds confidence with the customer. With a structured and repeatable process in place, effective self-governance demonstrates management’s commitment to maintaining the EVMS and open communications with the government customer. Self-disclosure and quickly addressing EVMS compliance issues are essential.

H&A earned value consultants often assist contractors to implement a robust self-governance process as their level of EVMS maturity increases over time. This blog highlights how H&A provides support and technical expertise to help a DOE contractor to do just that.

Developing the Self-Governance Process and Tools

H&A is a strong teammate in the development and implementation of a robust EVMS self-governance process for TRIAD at the Los Alamos National Laboratory (LANL). TRIAD is the prime contractor that provides laboratory management and operations for LANL. H&A is involved in developing tools, refining processes, establishing business rhythms, and summarizing data necessary to support the implementation of a leading-edge EVMS self-surveillance capability. The H&A team is also instrumental in developing the tools necessary to analyze, review, and act on the monthly data set TRIAD provides to the DOE EVMS compliance team.

These tools generate the DOE EVMS compliance metrics (automated and manual) in accordance with the DOE EVMS Metric Specification to ensure TRIAD is able to view the data the same way as their DOE counterparts. Once the tools generate the DOE EVMS compliance metrics, the results are passed to the TRIAD System Surveillance Officers (SSOs) to review and confirm flagged items are either actual fails or exempted/justified based on the rationale captured in the tool. For failed metrics, the SSOs and the project teams use the source data from the tool to identify the root cause and proactively correct EVMS compliance issues. Each month the EVM compliance data is collected across projects, summarized, and graded at the TRIAD level, and then gathered into an EVMS compliance dashboard for TRIAD leadership review and action.

Monthly Self-Surveillance Process

The monthly self-surveillance process includes the following activities.

  1. For each project, the tool generates the automated metrics from the DOE compliance flat files and then collects the results of manual testing into a single file with all 183 DOE metrics. This tool enables an SSO to review the flags, access the source cost and schedule data, apply exemptions/waivers, and then share the data with the project team to resolve issues. By trending this data across the project’s life cycle and capturing SSO exemptions and monthly actions, the team can analyze the data, determine root causes, address issues, and capture historical EVMS compliance actions in one place.
  2. An EVMS summarization tool then collects the results from each project and rolls the lower-level results into a summary TRIAD level. Each metric grade (Pass/Fail/Caution) considers weighted EVMS performance across multiple projects to ensure grading is aligned with the exit criteria for the DOE corrective action plans. In addition to TRIAD level grading for each metric, the summary tool also rolls up the metrics to the 10 EVMS Maturity Subprocess areas and 56 Attributes of an EVMS which is documented in the Compliance Assessment Governance (CAG) Appendix to the DOE EVMS Compliance Review Standard Operating Procedure (ECRSOP). This summarization tool provides the subprocess area and attribute grading at both the project and TRIAD levels. By viewing the data across projects and time, the EVMS core team can quickly identify systemic or project level issues.
  3. A set of tailored EVMS compliance summarization metrics is presented in a “dashboard” configuration for the EVMS core group and senior leadership to review. Leadership uses this summary data to determine where they need to dive deeper into the data and whether TRIAD is meeting their EVMS compliance targets.

Figure 1 illustrates this management level dashboard view. 

Figure 1: Example of the Summary Level Compliance Metrics Across Projects

Figure 1: Example of the Summary Level Compliance Metrics Across Projects

  1. The team also developed and uses a flat file analysis tool that is aligned to the DOE data integrity and quality checks (DIQs). This tool is used for projects transitioning into DOE Critical Decision (CD) Milestones 2 or 3 execution phases that require submittals to the DOE Project Assessment and Reporting System (PARS). This tool ensures the project flat files meet the DOE data quality standards. Like the 183 metrics tool, the flat file tool enables analysts to isolate data quality issues, review the source data, and then determine and track how the team will resolve or justify each issue. In addition to preparing for PARS submittals, these DIQ assessment metrics are also generated monthly to help assess on-going system integration integrity.

Establishing a Best-in-Class Self-Governance Process

In addition to supporting the monthly self-surveillance process, these tools and processes are instrumental in supporting the active surveillance portion of TRIAD’s self-governance efforts. The active surveillance team uses the same tools to summarize and review the “data call” sets in preparation for their reviews. Just like their DOE counterparts, the TRIAD active surveillance team analyzes the 183 DOE compliance metrics to focus their inquiries and document review findings. The H&A team was instrumental in planning, executing, and closing out the recent TRIAD active surveillance that was observed and lauded by the DOE EVMS compliance team.

By supporting the design, development, planning, and execution of all facets of a leading-edge DOE self-governance process, the H&A team helped to ensure our LANL customer has the robust EVMS compliance capability necessary to meet the rigorous DOE EVMS compliance requirements.

As this case study demonstrates, with H&A’s help, TRIAD successfully implemented a structured and repeatable self-governance process with analysis tools that capture objective measures and metrics to actively demonstrate compliance and issue resolution to their customer.

H&A earned value consultants can do the same for you. Call us today at (714) 685-1730 to get started. 

Establishing a Robust EVMS Self-Governance Process Read Post »

Formal Reprogramming: OTB or OTS Best Practice Tips

By / / , , , ,
, , , ,
Formal Reprogramming: OTB or OTS Best Practice Tips

As a result of an Earned Value Management System (EVMS) compliance or surveillance review, the Defense Contract Management Agency (DCMA) or DOE Office of Project Management (PM-30) may issue a corrective action request (CAR) to a contractor. H&A earned value consultants frequently assist clients with developing and implementing corrective action plans (CAPs) to quickly resolve EVMS issues with a government customer.

A recent trend our earned value management consultants have observed is an uptick in the number of CARs being issued related to over target baselines (OTB) and/or over target schedules (OTS). On further analysis, a common root cause for the CAR was the contractors lacked approval from the contracting officer to implement the OTB and/or OTS even though they had approval from the government program manager (PM).

So why was a CAR issued?  It boils down to knowing the government agency’s contractual requirements and EVMS compliance requirements.

What is an OTB/OTS and when it is used?

During the life of a contract, significant performance or technical problems may develop that impact schedule and cost performance. The schedule to complete the remaining work may become unachievable. The available budget for the remaining work may become decidedly inadequate for effective control and insufficient to ensure valid performance measurement. When performance measurement against the baseline schedule and/or budgets becomes unrealistic, reprogramming for effective control may require a planned completion date beyond the contract completion date, an OTS condition, and/or a performance measurement baseline (PMB) that exceeds the recognized contract budget base (CBB), an OTB condition.

An OTB or OTS is a formal reprogramming process that requires customer notification and approval. The primary purpose of formal reprogramming is to establish an executable schedule and budget plan for the remaining work. It is limited to situations where it is needed to improve the quality of future schedule and cost performance measurement. Formal reprogramming may be isolated to a small set of WBS elements, or it may be required for a broad scope of work that impacts the majority of WBS elements.

Formal reprogramming should be a rare occurrence on a project and should be the last recourse – all other management corrective actions have already been taken. Typically, an OTB/OTS is only considered when:

  • The contract is at least 35% complete with percent complete defined as the budgeted cost for work performed (BCWP) divided by the budget at completion (BAC);
  • Has more than six months of substantial work to go;
  • Is less than 85 percent complete; and
  • The remaining management reserve (MR) is near or equal to zero.

A significant determining factor before considering to proceed with a formal reprogramming process is the result from conducting a comprehensive estimate at completion (CEAC) where there is an anticipated overrun of at least 15 percent for the remaining work.

When an OTB is approved, the total allocated budget (TAB) exceeds the CBB, this value referred to as the over target budget. Figure 1 illustrates this.

Before Over Target Baseline
Figure 1 – Over Target Baseline Illustration

When an OTS is approved, the same rationale and requirements for an OTB apply. The planned completion date for all remaining contract work is a date beyond the contract completion date. The purpose of the OTS is to continue to measure the schedule and cost performance against a realistic baseline. The process must include a PMB associated with the revised baseline schedule. Once implemented, the OTS facilitates continued performance measurement against a realistic timeline.

Contractual Obligations

An OTB does not change any contractual parameters or supersede contract values and schedules. An OTS does not relieve either party of any contractual obligations concerning schedule deliveries and attendant incentive loss or penalties. An OTB and/or OTS are implemented solely for planning, controlling, and measuring performance on already authorized work.

Should you encounter a situation where it appears your best option is to request an OTB and/or OTS, the DoD and DOE EVMS policy and compliance documents provide the necessary guidance for contractors. It is imperative that you follow agency specific guidance to prevent being issued a CAR or your OTB/OTS request being rejected.

DoD and DOE both clearly state prior customer notification and contracting officer approval is required to implement an OTB and/or OTS. These requirements are summarized the following table.

ReferenceDoD/DCMA1DOE
RegulatoryDFARS 252.234-7002 Earned Value Management System
“(h) When indicated by contract performance, the Contractor shall submit a request for approval to initiate an over-target baseline or over-target schedule to the Contracting Officer.”
Guide 413.3-10B Integrated Project Management Using the EV Management System
6.1.2 Contractual Requirements.
“…if the contractor concludes the PB TPC and CD-4 date no longer represents a realistic plan, and an over-target baseline (OTB) and/or over-target schedule (OTS) action is necessary. Contracting officer approval is required before implementing such restructuring actions…”
Attachment 1, Contractor Requirements Document
“Submit a request for an Over-Target Baseline (OTB) or Over-Target Schedule (OTS) to the Contracting Officer, when indicated by performance.”
EVMS Compliance2Earned Value Management System Interpretation Guide (EVMSIG)3
Guideline 31, Prevent Unauthorized Revisions, Intent of Guideline
“A thorough analysis of program status is necessary before the consideration of the implementation of an OTB or OTS. Requests for establishing an OTB or an OTS must be initiated by the contractor and approved by the customer contracting authority.		EVMS Compliance Review Standard Operating Procedure (ECRSOP), Appendix A, Compliance Assessment Governance (CAG)
Subprocess G. Change Control
G.6 Over Target Baseline/Over Target Schedule Authorization
“An OTB/OTS is performed with prior customer notification and approval.”
See Section G.6 for a complete discussion on the process.
Contractor EVM SD4DCMA Business Process 2  Attachment, EVMS Cross Reference Checklist (CRC), Guideline 31.
“b. Are procedures established for authorization of budget in excess of the Contract Budget Base (CBB) controlled with requests for establishing an OTB or an OTS initiated by the contractor, and approved by the customer contracting authority?”		DOE ESCRSOP Compliance Review Crosswalk (CRC), Subprocess Area and Attribute G.6
“Requests for establishing an OTB or an OTS are initiated by the contractor and approved by the customer contracting authority.”

Notes:

  1. When DoD is the Cognizant Federal Agency (CFA), DCMA is responsible for determining EVMS compliance and performing surveillance. DCMA also performs this function when requested for NASA.
  2. Along with the related Cross Reference Checklist or Compliance Review Crosswalk, these are the governing documents the government agency will use to conduct compliance and surveillance reviews.
  3. For additional guidance, also see the DoD EVM Implementation Guide (EVMIG) , Section 2.5 Other Post-Award Activities, 2.5.2.4 Over Target Baseline (OTB) and Over Target Schedule (OTS). The EVMIG provides more discussion on the process followed including the contractor, government PM, and the contracting authority responsibilities.
  4. Your EVM System Description (SD) should include a discussion on the process used to request an OTB/OTS. The EVM SD content should be mapped to the detailed DCMA EVMS guideline checklist or the DOE Compliance Review Crosswalk (subprocess areas and attributes) line items.

Best Practice Tips

The best way to avoid getting a CAR from a government agency related to any OTB or OTS action is to ensure you have done your homework.

  • Verify your EVM SD, related procedures, and training clearly defines how to handle this situation. These artifacts should align with your government customer’s EVMS policy and regulations as well as compliance review guides, procedures, and checklists. Be sure your EVM SD or procedures include the requirement to notify and gain approval from the government PM and contracting officer, as well as what to do when the customer does not approve the OTB or OTS. Also discuss how to handle approving and managing subcontractor OTB/OTS situations; the prime contractor is responsible for these actions. Your EVMS training should also cover how to handle OTB/OTS situations. Project personnel should be aware of contractual requirements as well as your EVMS requirements and be able to demonstrate they are following them.
  • Maintain open communication with the customer. This includes the government PM as well as the contracting officer and any other parties involved such as subcontractors. Requesting an OTB or OTS should not be a surprise to them. Verify a common agreement has been reached with the government PM and contracting officer that implementing an OTB or OTS is the best option to provide visibility and control for the remaining work effort.
  • Verify you have written authorization from the government PM and the contracting officer before you proceed with implementing an OTB or OTS. You will need this documentation for any government customer EVMS compliance or surveillance review. Your baseline change requests (BCRs) and work authorization documents should provide full traceability for all schedule and budget changes required for the formal reprogramming action.

Does your EVM SD or training materials need a refresh to include sufficient direction for project personnel to determine whether requesting an OTB or OTS makes sense or how to handle OTB/OTS situations? H&A earned value consultants frequently help clients with EVM SD content enhancements as well as creating specific procedures or work instructions to handle unique EVMS situations. We also offer a workshop on how to implement an OTB or OTS .  Call us today at (714) 685-1730 to get started.

Formal Reprogramming: OTB or OTS Best Practice Tips Read Post »

Introduction to the IPMDAR Data Deliverable – Tips for Producing the Outputs

By / /

Contractors new to earned value management (EVM) often give us a call to help them respond to a government customer request for proposal (RFP) that includes the FAR or DFARS Notice of Earned Value Management System (EVMS) contract clause. Depending on the contract value threshold, the contractor will need to implement an EVMS that can at least produce the contract performance data submittals ($20M or greater) or complete a formal EVMS compliance review ($100M or greater) by a cognizant federal agency (CFA) such as the Defense Contract Management Agency (DCMA). Required data deliverables for a DoD or NASA customer reference the Integrated Program Management and Data Analysis Report (IPMDAR) Data Item Description (DID) as the means to submit monthly performance data. 

What is the objective for placing the IPMDAR on contract?

The government customer wants the monthly source schedule and cost data from the contractor for their own contract performance analysis in a standard format. They need to have reliable schedule and cost data for visibility into current contract performance as well as credible schedule and cost projections for the remaining work. Will the contractor complete the remaining work effort within the contractual schedule and cost targets? The government program manager needs this information for their own planning and budgeting as well as forecasting their funding requirements.

A standard format for collecting the data is important for the government customer so they can perform program portfolio analysis. The DoD established their EVM Central Repository (EVM-CR) to routinely collect contractor data submittals for program portfolio analysis. The data is organized using the MIL-STD-881 (the DoD Standard for Work Breakdown Structures or WBS), as a common basis to organize program data. The IMPDAR DID data submission requirements are defined in the File Format Specifications (FFS) and Data Exchange Instructions (DEI). The FFS and DEI specify the required set of data tables using JSON encoding for the IPMDAR cost and schedule data submissions while narrative text is submitted using Microsoft Word or PDF files so the customer can perform text searches. 

The government program manager can tailor the IPMDAR requirements as defined in the DoD IMPDAR Implementation and Tailoring Guide that complements the DID. For example, they can specify the level of data detail (control account or work package level), whether data can be delivered incrementally, variance analysis options, and requirements for the Performance Narrative Report content.

What is included in the IPMDAR deliverable?

There are three components as outlined below

IPMDAR Components

Notes:

  1. At a minimum, the IPMDAR requires data at the control account level with summary element of cost detail. The contract may specify work package level data.
  2. Inputs from a recent schedule risk assessment (SRA) should be included in the native schedule file submission when available. Depending on the schedule tool, the SRA data may need to be a separate file submission (Word or PDF). Results from the SRA along with other schedule analysis discussions (critical path, driving path, and schedule margin) are required to be included in the Detailed Analysis Report narrative. 
  3. The customer may request the results from a schedule data quality assessment and health metrics be included in the Detail Analysis Report narrative. 

What is required to produce the IPMDAR deliverables? 

For contractors new to EVM, one of their first objectives is to figure out what schedule and cost tools they need to be able to provide the required IPMDAR data and narrative analysis to their customer. H&A earned value consultants are sometimes asked to provide recommendations on commercial off the shelf (COTS) tools for this purpose. Much depends on what the contractor already has in place. 

Common schedule COTS tools such as MSP or Oracle Primavera P6 that have already been implemented will require an add-on to produce the SPD. Keep in mind that the IPMDAR does require SRA data and may require results from performing routine schedule data quality assessments. Some COTS add-ons to MSP or P6 are able to produce the typical schedule data quality metrics as well as produce the SPD. Other COTS scheduling tools such as Deltek Open Plan incorporate the SRA functionality, data quality metrics, and the ability to produce the SPD as part of the core product capabilities.

In most instances, contractors new to EVM do have an accounting/financial system in place to at least capture some level of contract or program/project budget and actual cost data. There may also be some capability to produce ETC data required for EAC financial analysis. The issue is organizing the complete set of time phased cost data (budget, earned value, actual cost, ETC) at the control account and work package level by summary element of cost that aligns with the schedule activity data. A contractor may be able to get by with Excel for a small project, however, it is time and cost prohibitive to create Excel macros to produce the CPD. Most COTS EVM cost tools are able to produce the CPD and have successfully completed the DoD EVM-CR data submission validation checks. This is a better alternative to building an in-house tool.

A data analysis tool such as Encore Analytics Empower is also a good option. Empower can import the time phased cost data from Excel or other COTS EVM tools and produce the CPD output. Empower can also import data from common COTS schedule tools. The benefit to using Empower is the ability to analyze the schedule and cost data in one place to verify alignment, produce interactive dashboards and a variety of analysis data views, and produce the IPMDAR Performance Narrative Executive Summary and Detailed Analysis Report.

Top Three Tips for Implementing Tools to Produce the IPMDAR Outputs

Here are a few tips from H&A’s earned value consultants on implementing tools to support the IPMDAR data submittals. Focus on getting the basics right.

  • Continuously verify the quality of the schedule and cost data. Routinely perform schedule data quality assessment and health checks to proactively resolve schedule construction, status, or data issues. Perform routine cost data validation checks such as earned value and no actual costs for a work package or the cumulative to date earned value exceeds the budget at completion (BAC). Correct all data anomalies before producing the deliverables.
  • Continuously verify the schedule and cost data are in alignment. Consistent schedule and cost data coding is critical to ensure integration and traceability. 
  • Anticipate the scope and level of data detail required. This can impact tool configuration, data structures, and data pulled from other business systems such accounting. For example, be prepared to provide the work package level of detail; actual costs will need to be available at this level. Another example is providing schedule risk assessment inputs; this is usually required at intervals specified in the CDRL.

H&A earned value consultants routinely help clients with constructing the schedule to support the IPMDAR data requirements, setting up the process to do routine data quality checks, integrating the schedule and cost data, and verifying the data before producing the performance reporting data submittals. Another common focus is producing clear and concise variance analysis narrative content. 

We can do the same for you. Call us today at (714) 685-1730 to get started.

Introduction to the IPMDAR Data Deliverable – Tips for Producing the Outputs Read Post »

Comparing the Efficacy of Independent Estimate at Completion (IEAC) Methods Using Real Project Data

By / /
Comparing the Efficacy of Independent Estimate at Completion (IEAC) Methods Using Real Project Data

“Data! Data! Data!” he cried impatiently. “I can’t make bricks without clay.”

-Sherlock Holmes, The Adventure of the Copper Beeches

There are many discussions about EACs and evaluating EACs including using Independent EAC (IEAC) formulae to compare with the contractor EACs. With good reason, we should wonder how accurate are those IEACs that we use so often and sometimes make decisions based on them. Are we misjudging contractor’s EACs based on formulae that are weak or inappropriate?

Humphreys & Associates has initiated a study to determine how accurate IEACs are, and we would like your help. The study will compare different IEAC formulae against the Program Manager (PM) most likely EAC at the 25, 50, and 75 percent complete point for completed projects. The objective is to assess how closely the IEACs and PM most likely EAC were able to predict the final cost outcome for the project.

How Accurate are IEAC Formulae?

Many formulae exist for using recorded data from an earned value management system (EVMS) to make independent estimates of the final cost at completion (EAC) for the element in question. The element might be a control account, a Work Breakdown Structure (WBS) element, or even an entire project.

What is not known is how accurate these methods are at forecasting the final actual cost for the project. This study hopes to determine that answer.

Real World IEAC Data

This study was initiated by collecting earned value data from 12 completed projects. We need projects that are completed because, on a completed project, the final actual outcome is known. We collected project data at the 25, 50, and 75 percent complete points. At each of these points, the IEAC formulae were applied to determine how closely they were able to predict the final actual cost outcome for the project. The quest is to learn how the various IEACs performed. Is any one of them more accurate than the others?

From this investigation, any indication of the relative efficacy of the formulae would be used to inform future use of the IEAC methods.

Our Method for Testing IEACs

In general, the IEAC approach is to use existing recognized formulae. We have chosen these IEACs as a starting point:

  • IEAC 1 = BAC/CPIe at the percent point reported. This formula can be stated in words as “the entire project is performed at the same efficiency as experience to date.”
  • IEAC 2 = ACWP + [BCWR/CPI (.5) + SPI (.5)]. This formula uses weighted SPI and CPI which theoretically allows for sensitivity to both cost and schedule historical performance. The weights used in this application are even at .50 and .50.
  • IEAC 3 = ACWP + [BCWR/CPI x SPI]. This formula uses the SPI and CPI multiplied together which theoretically allows for sensitivity to both cost and schedule performance to date.
  • IEAC 4 = ACWP + BCWR. This formula assumes the remaining work will be done as budgeted with no factoring.

One additional non-traditional IEAC will be used.

IEAC 5 = Use of IEAC 2 weighted SPI and CPI but decreasing the proportion applied to the SPI as the percent of project completion increases. In other words, the impact of schedule performance diminishes as the project becomes closer to completion.

We will also take the average of all the formulae to see how that works.

Initial Data Set

One aerospace contractor and one US Government agency have provided the required data for 12 completed projects with an interest in the outcome of the study. The source of the data and the specific projects will not be disclosed in the study.

These real-world projects did not have an exact 25%, 50%, or 75% dataset. The closest dataset to each of those completion percentages was used. One example dataset looks like this (color coding should be ignored):

Example Product Data

How can you help?

We need more project data to gather enough varying project outcomes to make the test realistic. We do not plan to keep the types of projects or products separate but will take all the data we can get and look at them all.

Please consider providing data for the study. We have created an Excel spreadsheet template to help gather project data in a common format for analysis. You can download this template here. Add as many tabs as needed for each project. Send your completed spreadsheet to humphreys@humphreys-assoc.com.

In a separate blog we will outline other help we need to complete the study and to analyze the results.

IEAC_Study_Project_Data_Template-Rev-06052023Download

Comparing the Efficacy of Independent Estimate at Completion (IEAC) Methods Using Real Project Data Read Post »

Tips for Producing an Earned Value Management System Description

By / / , ,
, ,
Tips for Producing an EVM System Description Blog Post Banner Image.

There are a variety of ways a contractor can produce the documentation that describes their Earned Value Management System (EVMS), how it is implemented, used, and maintained. This is typically referred to as the EVM System Description (SD), though other names may be used. For example, it could be titled the Integrated Project Management System Description or Project Controls System Description to align with corporate or contractual naming conventions. EVM practices may be just one part of their overall project management system.

Regardless of the title, a contractor with an EVMS contractual requirement must be able to provide the necessary documentation to their customer to demonstrate their system complies with the EIA-748 Standard for EVMS guidelines. This includes any additional implementation guidance associated with the government agency responsible for conducting EVMS certification or validation reviews.

What is the purpose of an EVM System Description?

There are two purposes for an EVM SD; one is internal, the other is for the external customer.

The primary user of the EVM SD is project personnel such as project managers, project controls, schedulers, and control account managers (CAMs) responsible for implementing the EVMS on their project. It is essential they have a clear understanding of the required corporate EVM process and procedures as well as how to meet the intent of the EIA-748 guidelines using the applicable schedule, cost, analysis, risk, and other project control toolsets.

The government customer uses the EVM SD to gain an understanding of the contractor’s EVMS. The EVM SD must adequately document how the contractor’s system meets the intent of the EIA-748 guidelines. It should include a map of the EVM SD content to the EIA-748 guidelines or similar government customer checklist. The purpose of a government agency’s compliance review is to conduct a comprehensive assessment of a contractor’s EVMS. This includes the EVM SD as well as related processes and procedures. They also assess how the contractor has implemented the EVMS to verify the EVMS is providing timely, reliable, and auditable information.

Once the Cognizant Federal Agency (CFA) has determined the contractor’s EVMS complies with the EIA-748 guidelines, they conduct routine surveillance to ensure the contractor’s system as documented in the EVM SD and implemented on projects continues to comply with the EIA-748 guidelines. The contractor’s EVM SD is also a typical artifact required for an Integrated Baseline Review (IBR).

What needs to be included in the EVM SD?

An EVM SD should explain the methodology the contractor uses to comply with the EIA-748 guidelines as well as any government agency specific requirements. For example, DCMA uses the DoD EVMS Interpretation Guide (EVMSIG) and their Earned Value Management System Center Business Practice (BP) documents such as the BP 2 EVM System Description Review attachment EVMS Cross Reference Checklist (CRC). The DOE Office of Project Management (PM) uses their EVMS Compliance Review Standard Operating Procedure (ECRSOP) and related appendices including their Compliance Assessment Governance (CAG) document and EVMS Compliance Reference Crosswalk (CRC).

The EVM SD should provide an executive overview, reference the corporate EVMS policy, and address all of the subsystems that make up the EVMS. H&A earned value consultants have reviewed countless system descriptions over the years; contractors take a variety of approaches to discuss their subsystems.

Some contractors align with the EIA-748 five process areas (Organizing; Planning, Scheduling, and Budgeting; Accounting; Analysis and Management Reports; and Revisions and Data Maintenance). Others include additional sections covering indirect costs, material management, and subcontractor management. The planning, scheduling, and budgeting process area is often broken down into two sections – one for planning and scheduling and the other to cover work authorization and budgeting (a total of nine process groups). For more discussion, see the blog on the Benefits of Using Nine Process Groups.

A DOE contractor may align with the DOE CAG 10 subprocesses; this is similar to the nine progress group approach plus a section on risk and opportunity management. Other contractors use the project life cycle approach (initiation, planning, execution, monitor and control, and close out) to organize their content.

We recommend including a section on self-governance or self-surveillance (see Benefits of an EVMS Self-Governance Process). As noted in this blog, contractors with an approved EVMS are expected to establish and execute an annual EVMS self-governance plan.

The government customer expects contractors to map their EVM SD content to the applicable government agency EVMS compliance checklist. This is typically an appendix to the EVM SD. In most instances, this is the DCMA BP 2 EVMS Cross Reference Checklist or the DOE ECRSOP EVMS Compliance Reference Crosswalk. These checklists include attributes and/or questions that provide a more comprehensive guide to assess how a contractor meets the intent of the guidelines. The government customer also uses these checklists to identify and record the areas in the system documentation that meet the intent or note deficiencies the contractor needs to address.

The system description should include a concise description of what’s required, diagrams, illustrations, process flowcharts, as well as sample forms and reports with example data. In preparation for a compliance review, H&A earned value consultants often assist clients in creating an EVMS storyboard that visually depicts the entire system using the workflow diagrams along the inputs and outputs, and who is doing what using the applicable tool that demonstrates the system in operation.

Single or multiple documents?

A common question H&A earned value consultants are asked is whether the EVM SD should be a single document that includes the complete set of flowcharts, forms, and reports or a summary document with supporting procedures. Typically, the supporting procedures include the process flowcharts and sometimes the applicable artifacts (forms and reports). There are pros and cons to each approach.

Single Document Approach
Pros Cons
  • Provides an integrated view of the entire EVMS process.
  • Provides a complete discussion of a topic without having to reference multiple documents. Requirement discussion and process flowcharts with responsibility swim lanes along with example inputs/outputs provides a complete picture.
  • Configuration control is easier to manage, only need to make changes in one place.
  • Easier to search for a topic in a single document.
  • Easier to map the contents of a single document to the DCMA or DOE CRC.
  • Easier to create and maintain cross reference links between sections within a single document (See or See Also references).
  • For a company new to EVM, can appear to be an overwhelming task to produce a single, comprehensive document.
  • Can be a lengthy document.
  • More difficult to partition the content to different process owners to complete their section and to combine the content.
Summary Document with Supporting Procedures Approach
Pros Cons
  • Shorter summary document provides a general overview as a quick introduction to the EVMS. Can appear less daunting. Can be used as a strategy to introduce EVM concepts and incorporate into standard business practices.
  • Modular approach to creating and maintaining content. Can be easier to create the initial content.
  • Useful for targeted training using the individual procedures for a process area. For example, training focused on the steps required for developing the WBS and WBS dictionary with expected outputs.
  • Someone must review multiple documents to gain a complete understanding of the EVMS.
  • Fragmented and potentially siloed view of the system.
  • Potentially larger volume of content (over a single document), content is often repeated for context.
  • Configuration control. More difficult to maintain content to ensure consistency in multiple documents as well as maintain cross references between documents.
  • More difficult to map content in multiple documents to the DCMA or DOE CRC.

Regardless of which approach you take, keep in mind that the government customer will be approving the complete set of EVM SD documents (one or multiple documents). Any document referenced in the EVM SD is also subject to review. When you make changes to your EVM SD, the government customer will need to review and approve those changes to ensure continuing compliance with the EIA-748 guidelines. This is a formal process; changes must be processed through the contracting officer. 

Need help with your EVM System Description?

Whether you need to update your existing EVM SD or create one, H&A earned value consultants can help you to organize the content and artifacts in alignment with your business requirements. They also work with you to ensure your EVMS satisfies the EIA-748 guideline requirements as well as government agency specific requirements. H&A often assists with designing an EVMS and ensuring the project control software tools are configured to support the EVMS as well as verifying the quality of the data. For clients new to EVM, H&A offers a template that provides the basis to develop an EVM SD. This template is intended to assist clients with designing and implementing an EVMS that meets EIA-748 guideline requirements as well as the more rigorous requirements of specific government customers.

Call us today at (714) 685-1730 to get started.

Tips for Producing an Earned Value Management System Description Read Post »

Scroll to Top