EVM Terminology

EVM Terms

Level of Effort (LOE) Best Practice Tips

By / / , , ,
, , ,
Level of Effort (LOE) Best Practice Tips

Clients are often seeking advice from our earned value consultants about implementing a practical approach in response to government customer requirements to proactively manage level of effort (LOE) tasks. The DoD EVMS Interpretation Guide (EVMSIG), NASA guidance, and DOE guidance such as the Compliance Assessment Governance (CAG) document clearly state the requirements for contractors related to planning, maintaining, and managing LOE. DOE also specifies a limit to the percentage of LOE allowed within a control account to avoid skewing performance measurement of the discrete work effort. In addition, both the DCMA and DOE EVMS data quality test metric specifications include manual and automated tests with thresholds specific to LOE.

Common accepted best practices for LOE include:

  • Reducing the amount of LOE to the lowest level possible to minimize the number of activities that need to be actively managed. Objective measures of performance are always preferred.
  • When LOE activities are included in the schedule, they should not drive the date calculations of discrete activities in the integrated master schedule (IMS). They should also not appear on the critical path.
  • LOE must be segregated from discrete work effort. In practice, this means a work package can only be assigned a single earned value method. The work package is one of three types. It is either 1) discrete effort with an assigned earned value technique such as the Milestone or Percent Complete technique, 2) apportioned effort, or 3) LOE. 
  • It must be verified it is truly LOE, i.e., it is management or sustainment type of activity that has no identifiable end products or established relationship to other measurable effort. It is clearly not discrete effort or apportioned effort. Remember that with LOE, the passage of time is the only measurement criteria. At the end of the performance month, the budget value for that month is earned. For this reason, LOE is the least desirable earned value method. 
  • The budget or estimate to complete the work effort is time phased and reflects the planned or forecast period of performance. The period of performance and resource requirements must be substantiated. Determining the basis of estimate for the LOE activity can also help to verify the work is truly LOE.

So, what is the problem? 

Common situations H&A earned value consultants run into are contractors where:

  • Managing the LOE is put on “auto pilot.” This might work for project management type of activities that span the duration of the project. It does not work so well when the LOE is associated with the occurrence of discrete work effort that is subject to change – i.e., the discrete work effort duration changes or the start date and/or the complete date changes. The result?
  • LOE tasks may incur actual cost of work performed (ACWP) with no budgeted cost for work performed (BCWP);
  • LOE tasks earn BCWP with no ACWP; or
  • The estimate at completion (EAC) is greater than the ACWP with BCWP equal to the budget at completion (BAC).

    Any one of these conditions would trip the DCMA and DOE test metrics and should be avoided. These types of situations were illustrated in a previous blog, “Level of Effort Decision Tree” that discusses how to properly replan LOE. 
  • Their EVM System Description doesn’t provide sufficient guidance to project personnel on what proactive management of LOE means. What are the rules for planning and maintaining LOE? How is LOE handled differently from discrete work packages?

    Some System Descriptions allow LOE replanning to occur within the “freeze period,” usually defined as the current reporting period and often plus one additional month. This is contrary to other best practice guidance about how to handle changes for open discrete effort work packages. For discrete effort work packages, changes within the freeze period are not allowed and the work package must be closed to replan the remaining work. What’s the process for handling that open LOE work package? What about retroactive changes when the LOE work occurs earlier or later than planned, or the duration is different than planned? Then what?

    When project personnel lack guidance, then arguments often ensue about what is the “correct” interpretation of the wording in governing documents or test metric specifications that are often inconsistent.
  • Validation checks are not routinely performed. This includes validation checks to ensure that control account managers (CAMs) are selecting the appropriate earned value method for a work package following the EVM System Description guidance during the work definition and planning phase. It also includes routine monthly data checks to identify common data anomalies typically associated with LOE such as ACWP and no BCWP or BCWP with no ACWP. The goal is to fix problems in the current reporting month and avoid making any retroactive changes. You should be catching and fixing avoidable DCMA or DOE EVMS test metric “triggers” every reporting period.

Best Practice Tips

Here is a short list of best practice tips that H&A earned value consultants have helped clients to implement over the years to ensure LOE is properly planned and proactively managed. The approach is tailored for each client to reflect the type of work the company typically performs. This is documented in their EVM System Description, related procedures, and recurring training to ensure project personnel have clear, specific guidance they can follow.

  • Consider using the Percent Complete earned value technique instead of LOE. A best practice is to identify quantifiable backup data (QBD) for a work package using the Percent Complete earned value technique. The QBD for the LOE type of work package could be the milestones identified for the discrete effort work package the LOE work package is supporting. This helps to ensure the work packages are reviewed and managed together.
  • Could the Apportioned Effort method be used instead of LOE? Is it possible to establish a direct relationship between the discrete effort and supporting effort? For example, is historical data available to document that the support number of hours is a given percentage of the discrete effort labor hours? If so, then using the Apportioned Effort method is a much better alternative. When the discrete work package is statused, the apportioned effort work package would be automatically statused as well. 
  • Consider shorter durations for the LOE when that LOE is supporting discrete effort. Should the first occurrence of the LOE trigger a data anomaly test metric, it can be proactively handled along with any future replanning. The remaining LOE would already be in one or more separate work packages so there won’t be any criticism for changing open work packages. Any adjustments can be made in the current reporting period avoiding any retroactive changes that would trigger other data metric tests. What is considered to be “short duration” should be defined in the EVM System Description. An example would be LOE work packages of 3 to 4 months in duration. Be sure to provide specific guidance to project personnel on how to process these types of current reporting period LOE replanning adjustments. The LOE work package breakpoints should be technically related. For example: “Phase I Support,” “Drawing Support,” and so forth instead of generic descriptions such as “April Support,” “May Support,” or “June Support.”
  • Use rolling wave planning. This is by far one of the better solutions. This helps to ensure the discrete tasks and any supporting LOE tasks are planned together before the work is authorized to begin. Shorter durations for the LOE tasks are often used to align with the forward planning window.
  • Incorporate LOE earned value method checks into your routine status and data analysis process. Identify any upcoming LOE activities (for example, the next 60 to 90 days), along with the CAM responsible for the work effort to verify they accurately reflect the current plan. It is always better to proactively replan future LOE when needed instead of defaulting to an “auto pilot” mode. The CAM should understand this is part of their responsibilities.

Does your EVM System Description or training materials need a refresh to include specific guidance for project personnel that documents the preferred approach for planning, maintaining, and managing LOE? H&A earned value consultants frequently help clients with EVM System Description content enhancements or creating specific procedures that reflect your unique business environment. Call us today at (714) 685-1730 to get started.

Level of Effort (LOE) Best Practice Tips Read Post »

Timely Subcontractor Data – Mission Impossible?

By / / , , ,
, , ,
Timely IPMDAR Subcontractor Data – Mission Impossible?

With the arrival of the Integrated Program Management Data and Analysis Report (IPMDAR) requirements for electronic cost and schedule dataset submittals, DoD contractors with EVMS or EVM reporting contractual requirements have a tighter time frame for submitting their month end data. A previous blog, Introduction to the IPMDAR Data Deliverable – Tips for Producing the Outputs summarizes these data reporting requirements. This includes the Contract Performance Dataset (CPD) for the time phased cost data and the Schedule Performance Dataset (SPD) along with a native file export out a schedule tool.

For month end data submittals, the IPMDAR Data Item Description (DID), DI-MGMT-81861C (20210830) states:

1.8.1 Monthly Submission Requirement. IPMDAR data shall be required at least monthly. The reporting frequency shall be specified in the Contract Data Requirements List (CDRL). All reports shall reflect data from the same accounting period and shall be provided at any time after the close of the contractor’s accounting period, but no later than sixteen (16) business days after the contractor’s accounting period end date.

On the surface, you might say requiring data delivery 16 business days after the contractor’s accounting period end date doesn’t sound unreasonable or even much different from the previous Integrated Program Management Report (IPMR) DID (DI-MGMT-81861A), and you would be right.

What is the issue?

Some people think that because the IPMDAR submittals are electronic datasets instead of report formats it is easier to generate and report that information. That is not necessarily true, and shortening the data turnaround time exacerbates the problem. The tighter time requirements also apply when there are EVM reporting subcontractors providing performance data to a prime contractor.

The third sentence of 1.8.1 above states: “All reports shall reflect data from the same accounting period…” This requirement is very challenging, especially when a subcontractor operates on a different month end accounting calendar; for example, a “5-4-4” versus the prime’s “4-4-5” calendar. Even when the prime and subcontractor are on the same month end calendar, for the prime to submit IPMDAR data in 16 business days, the subcontractor has less time to provide their data to the prime. It becomes even more challenging on very large programs that have several tiers of subcontractors.

Subcontractors cry “foul” because they don’t have enough time to get all the performance data ready in the reduced time. Prime contractors cry “foul” because they are held accountable for data that may or may not come from one or more tiers of subcontractors in time for them to conduct basic data analysis and deliver month end data for the IPMDAR. The government customer still insists all the data must be for the same accounting month end date, even though that may not be well defined. Customers also do not want the subcontractor data delayed by a month just to get the subcontractor data “caught up” – i.e., “comparing apples to oranges.”

Is incremental delivery of IPMDAR the answer?

The government suggests that incremental delivery could resolve this dilemma. The DoD IPMDAR Implementation and Tailoring Guide (August 24, 2021) expands on the paragraph from the IPMDAR DID:

1.8.1.1 Incremental Delivery. Reports may be provided incrementally, including preliminary data, with the number of days for delivery of each submittal tailored in the CDRL. Data delivered is not considered authoritative until the final submission and signature. The recommended incremental delivery process is the Schedule, followed by the CPD and the Executive Summary, Government review of submittals, Government directed Detailed Analysis, Contractor Detailed Analysis delivery and all final data.

The IPMDAR Implementation and Tailoring Guide also provides a notional example of how an incremental delivery could be handled:

1. SPD – To be delivered with native file five (5) working days after the end of the contractor’s accounting period (may be labeled preliminary)

2. CPD – To be delivered with the Executive Summary ten (10) working days after the end of the contractor’s accounting period (may be labeled preliminary)

3. Contracting Office to select items for detailed analysis (variances) – to contractor thirteen (13) working days after the end of the contractor’s accounting period

4. Performance Narrative Analysis – to be delivered NLT sixteen (16) working days after the end of the contractor’s accounting period along with any other “final” versions of previously submitted files

Note: The notional incremental delivery plan above is not additive.

Doing the above might demonstrate to the government customer that the prime contractor is at least trying their best to make the prime/subcontract situation work – even though they would be using “estimated data” until the final versions come in from the subcontractors. Does this approach really make the timely delivery of the data easier to attain? The bottom line does not change. Per number 4 above, the prime still has to deliver all the data in “final versions” by the 16th business day following the close of their accounting calendar. The note at the bottom specifies the days indicated in each step are not “additive” – i.e., the contractor does not get 5+10+13+16 = 44 business days.

In some circumstances, incremental delivery might allow some subcontractors a bit more time to get the data to the prime contractor, but there would still have to be tighter delivery dates for the incremental deliveries, so the problem does not really go away.

What are your options?

This difficult situation arises because few contractors consider the implications of having to get all data by their accounting month end. Not all the subcontractor work elements are set up the same way. Contractors who have the EVM reporting requirement, who do or will have EVM reporting subcontractors, should address this basic difference as part of the contract negotiation process. One possible part of this negotiation could be to use the IPMDAR DID, paragraph 1.4, to help level the field for reporting purposes. This paragraph states:

1.4 Direct Reporting Contractor Role.

1.4.1 A Direct Reporting Contractor is any contractor required to provide the IPMDAR directly to the Government. This includes prime contractors, subcontractors, intra-government work agreements, and other agreements, based on the contract type, value, duration, nature of the work scope, and the criticality of the information. In this document, instances of “Contractor” are synonymous with “Direct Reporting Contractor.”

There is a footnote to this paragraph that states:

In the event that the Direct Reporting Contractor is a contractor other than the prime, the Direct Reporting Contractor will additionally report to the prime. Subcontractor data shall be provided to the prime in a manner that supports the contractor’s submission to the Government.

One solution is to negotiate to have each EVM reporting subcontractor deemed a “Direct Reporting Contractor” that submits their data directly to the government, including the customer, as well as to the prime contractor. The prime and subcontractors are each submitting their IPMDAR electronic deliverables to the DoD EVM Central Repository (EVM-CR).

Each level of contract, the prime through however many tiers of subcontractors there may be, will have the same 16 business days after their own accounting month end dates to provide all interested parties with the EVM data. The prime contractor still must at least get estimated subcontractor data to do their monthly assessment, making corrections in the next month after they have received the final version of the data from their subcontractors.

Should the government customer want analysis performed on subcontracted effort, the IPMDAR dataset submittals will be in the DoD EVM-CR. They can do that analysis independently of the prime contractor’s analysis that would be provided after the prime’s 16th business day.

This approach would put pressure on the prime because they will not have seen the subcontractor’s data prior to it being delivered to the government, but that could be addressed in the next reporting period’s “errata” variance analysis narrative. The government would also have the detailed data from the subcontractors when the subcontractor is providing a reduced set of data such as only total cost data to the prime. Should the government customer not want to do that level of analysis, the government customer may need a different contracting solution to avoid requiring EVM reporting down through various levels of contracts. This is often determined by the contract value and risk factors associated with the subcontractor. 

This approach also requires the subcontractor to produce two deliverables. One for the prime contractor in an agreed upon format and one for the government customer following the IPMDAR DID electronic submittal requirements for the DoD EVM-CR. These reporting requirements should be negotiated with the subcontractor well in advance; the subcontractor needs to know their data deliverable and reporting requirements when they bid on the work effort for the prime.

This subcontractor data incorporation issue has been around for many years and can be very confusing. H&A earned value consultants can help you work through the various responses to this requirement in the best possible way for your situation. Call us today at (714) 685-1730 to get started.

Timely Subcontractor Data – Mission Impossible? Read Post »

Incorporating IMS Information Directly into Independent Estimate at Completion (IEAC) Formulas

By / / , , , ,
, , , ,
Incorporating IMS Information Directly into Independent Estimate at Completion (IEAC) Formulas

“When you need to discuss the schedule, look at the schedule.”

– A Scheduler’s Lament

There are many existing formulas for calculating an Independent Estimate at Complete (IEAC) from earned value data. A recent study of a sample of projects found that the calculated IEACs analyzed at the 25%, 50%, and 75% complete points were not accurate when compared to the final actual cost of work performed (ACWP). The following table lists the thresholds used to assess the accuracy of the IEACs at the different complete points for the sample projects.

Percent CompleteAccuracy Threshold
25%Within +/- 10% of final ACWP
50%Within +/- 7% of final ACWP
75%Within +/- 5% of final ACWP

While working on that study of the accuracy of commonly applied IEAC formulas as well as on a small project as an analyst for a customer, the idea for using data directly from the integrated master schedule (IMS) in conjunction with the cost performance data to create a new IEAC formula emerged.

Using Data Directly from the IMS to Calculate an IEAC

It should be noted that none of the generally used IEAC formulas use data directly from the IMS. The IEAC formulas use data found in the cost performance portion of the earned value monthly reports to customers.

IMS data is only used indirectly in the IEAC formulas. When a task is started and progress updated, the earned value (the budgeted cost for work performed or BCWP) is developed from the progress reported. This is measured against the cost baseline (the budgeted cost for work scheduled or BCWS).

At the same time, in the IMS environment, the schedule analysts are calculating the Baseline Execution Index (BEI) for task completions/finishes. BEI (for finishes) measures how many of the tasks baselined to be completed by the cut-off date were completed. If all the tasks were done (BEI = 1), their value would have been earned. Of course, other tasks could have started, progressed, and maybe even finished. For this example, the Schedule Performance Index (SPI) calculated at that point (BCWP/BCWS) should be at least 1 and potentially higher. The SPI reflects the baseline value of completed tasks plus the in-process claimed baseline value. The in-process claimed value can be subjective in some cases.

The argument, if there were one, might be there is no need to try and include BEI or similar schedule measures in the IEAC formulas since they already include SPI.

However, there is a whole different and unique set of information coming from the IMS that is not currently used in the IEAC formulas. That information is what we chose to call “Duration Performance” and “Realism Ratio.” These are measures of the actual duration for completed tasks and the forecast duration for future tasks.

Calculating Duration Performance

The IMS data includes the baseline number of days assigned to each task as well as the actual number of days to complete each task. If a task is baselined to take 10 days (Baseline Duration = 10) and the task took 15 days to complete (Actual Duration = 15) then it is taking 150% of baseline to do the work.

This is similar to the Cost Performance Index (CPI) that uses the BCWP and the ACWP to determine how efficient the work performance has been. The formula BCWP/ACWP shows how the work accomplished compares to the cost of that work performed.

If we assume, for labor at least, that taking longer to complete a task often leads to costing more than baselined, we can use the Duration Performance to develop an IEAC.

To develop the Duration Performance, we would use the IMS from the month being analyzed to perform the following actions:

  1. Filter out all summary tasks and look only at real work tasks.
  2. Decide what to do with level of effort (LOE) – keep it or ignore it.
  3. Filter for all tasks that are completed (100% complete).
  4. Add up the baseline duration in days for all these completed tasks.
  5. Add up the actual duration days for these same completed tasks.
  6. Compare the actual duration days used to the baseline duration days.

An example would be:

  • 100 completed tasks
  • Total baseline days duration = 1,000
  • Total actual days duration = 1,500
  • Duration Performance = 1,000 / 1,500 = .67

One of the common IEAC formulas is the “SPI times CPI” that is calculated like this: ACWP + Budgeted Cost of Work Remaining (BCWR) / (CPI x SPI) where BCWR = Budget at Completion (BAC) – cumulative to date BCWP.

Now that we have a duration performance factor, we can develop a new IEAC. The Duration Performance IEAC would be done using the CPI from the same month as the IMS where ACWP + BCWR / (CPI x Duration Performance Index).

Using some actual data from a project for a single month we see:

  • Duration Performance Index = .82
  • BEI = .72
  • CPI = .92
  • SPI = .94 (significantly higher than the BEI)
  • ACWP = $9.2M
  • BCWR = $18.3M
  • IEAC using standard formula with CPI x SPI = $9.2 + $18.3 / (.92 x .94) = $30.3M
  • IEAC (Duration Performance) = $9.2 +$18.3 / (.92 x .82) = $33.5M

Assessing the Realism Ratio

When we look at the remaining tasks to be completed, we can use the Realism Ratio to assess how the future forecast durations compare to the performance so far.

The data needed are the baseline duration and the forecasted duration for all tasks that have not been started. This concept excludes in-process tasks. In our example from before, the data we created looked like this:

  • 100 completed tasks
  • Total baseline days duration = 1,000
  • Total actual days duration = 1,500
  • Duration Performance = 1,000 / 1,500 = .67

We would use the same IMS to do this:

  1. Filter out all summary tasks and look only at real work tasks.
  2. Decide what to do with LOE – keep it or ignore it.
  3. Filter for all tasks that are not started.
  4. Add up the baseline duration in days for all these tasks not started.
  5. Add up the forecasted duration days for these same tasks not started.
  6. Compare the forecasted duration days to the baseline duration days.

Let’s say there were 100 tasks not started. If the forecasted days were 1,000 and the baseline days were 1,000 that would yield 100%. When we did the example, the Duration Performance was .67. This means that performance to date was .67 but the future will be 100% or 1. You can see the disconnect. That disconnect we call the Realism Ratio (in this example, .67/1).

Data from the actual project for the same month as discussed earlier shows:

  • Duration Performance = 122% of baseline
  • Future Performance = .86 or 86% of baseline.

This means that the future durations are cut significantly.

We would use this data to develop a factor called a Realism Ratio (86/122 = .70) and that would be used to develop an IEAC using this formula: IEAC (Realism Ratio) = ACWP + BCWR / (CPI x Realism Ratio).

Using the same sample project data from above and adding in an assessment of the forecasted durations for the remaining work, we see:

  • Duration Performance = .82
  • BEI = .72
  • CPI = .92
  • SPI = .94 (significantly higher than the BEI)
  • ACWP = $9.2M
  • BCWR = $18.3M
  • Realism Ratio = .70
  • IEAC using standard formula with CPI x SPI = $9.2 + $18.3 / (.92 x .94) = $30.3M
  • IEAC (Duration Performance) = $9.2 +$18.3 / (.92 x .82) = $33.5M
  • IEAC (Realism Ratio) = $9.2 +$18.3 / (.92 x .70) = $37.6M

The project is not complete, so the final ACWP position is not known. There is a dramatic difference between the three IEACs. The difference between BEI and SPI indicates that in-process tasks and other factors such as LOE are potentially affecting SPI.

What can we learn from this sample project?

In this example, additional investigation is warranted. There are potential issues with the realism of the baseline and current schedule that are signaling a cost growth issue is likely to occur. Relying on just the time-phased cost data for IEAC calculations may not be sufficient to assess whether a contractor’s range of EACs included in their monthly cost performance reports are realistic. For more discussion, see the blog on Maintaining a Credible Estimate to Completion (EAC) and the blog on Using EVM Performance Metrics for Evaluating EACs.

Are there lurking cost growth surprises in your projects? You may want to consider revisiting your estimate to complete (ETC) and EAC process to verify there is an integrated assessment of the schedule and cost data to identify potential disconnects. H&A earned value consultants can provide an independent assessment of the quality of the data as well processes and procedures to help you verify your EACs are realistic. Call us today at (714) 685-1730.

Incorporating IMS Information Directly into Independent Estimate at Completion (IEAC) Formulas Read Post »

Formal Reprogramming: OTB or OTS Best Practice Tips

By / / , , , ,
, , , ,
Formal Reprogramming: OTB or OTS Best Practice Tips

As a result of an Earned Value Management System (EVMS) compliance or surveillance review, the Defense Contract Management Agency (DCMA) or DOE Office of Project Management (PM-30) may issue a corrective action request (CAR) to a contractor. H&A earned value consultants frequently assist clients with developing and implementing corrective action plans (CAPs) to quickly resolve EVMS issues with a government customer.

A recent trend our earned value management consultants have observed is an uptick in the number of CARs being issued related to over target baselines (OTB) and/or over target schedules (OTS). On further analysis, a common root cause for the CAR was the contractors lacked approval from the contracting officer to implement the OTB and/or OTS even though they had approval from the government program manager (PM).

So why was a CAR issued?  It boils down to knowing the government agency’s contractual requirements and EVMS compliance requirements.

What is an OTB/OTS and when it is used?

During the life of a contract, significant performance or technical problems may develop that impact schedule and cost performance. The schedule to complete the remaining work may become unachievable. The available budget for the remaining work may become decidedly inadequate for effective control and insufficient to ensure valid performance measurement. When performance measurement against the baseline schedule and/or budgets becomes unrealistic, reprogramming for effective control may require a planned completion date beyond the contract completion date, an OTS condition, and/or a performance measurement baseline (PMB) that exceeds the recognized contract budget base (CBB), an OTB condition.

An OTB or OTS is a formal reprogramming process that requires customer notification and approval. The primary purpose of formal reprogramming is to establish an executable schedule and budget plan for the remaining work. It is limited to situations where it is needed to improve the quality of future schedule and cost performance measurement. Formal reprogramming may be isolated to a small set of WBS elements, or it may be required for a broad scope of work that impacts the majority of WBS elements.

Formal reprogramming should be a rare occurrence on a project and should be the last recourse – all other management corrective actions have already been taken. Typically, an OTB/OTS is only considered when:

  • The contract is at least 35% complete with percent complete defined as the budgeted cost for work performed (BCWP) divided by the budget at completion (BAC);
  • Has more than six months of substantial work to go;
  • Is less than 85 percent complete; and
  • The remaining management reserve (MR) is near or equal to zero.

A significant determining factor before considering to proceed with a formal reprogramming process is the result from conducting a comprehensive estimate at completion (CEAC) where there is an anticipated overrun of at least 15 percent for the remaining work.

When an OTB is approved, the total allocated budget (TAB) exceeds the CBB, this value referred to as the over target budget. Figure 1 illustrates this.

Before Over Target Baseline
Figure 1 – Over Target Baseline Illustration

When an OTS is approved, the same rationale and requirements for an OTB apply. The planned completion date for all remaining contract work is a date beyond the contract completion date. The purpose of the OTS is to continue to measure the schedule and cost performance against a realistic baseline. The process must include a PMB associated with the revised baseline schedule. Once implemented, the OTS facilitates continued performance measurement against a realistic timeline.

Contractual Obligations

An OTB does not change any contractual parameters or supersede contract values and schedules. An OTS does not relieve either party of any contractual obligations concerning schedule deliveries and attendant incentive loss or penalties. An OTB and/or OTS are implemented solely for planning, controlling, and measuring performance on already authorized work.

Should you encounter a situation where it appears your best option is to request an OTB and/or OTS, the DoD and DOE EVMS policy and compliance documents provide the necessary guidance for contractors. It is imperative that you follow agency specific guidance to prevent being issued a CAR or your OTB/OTS request being rejected.

DoD and DOE both clearly state prior customer notification and contracting officer approval is required to implement an OTB and/or OTS. These requirements are summarized the following table.

ReferenceDoD/DCMA1DOE
RegulatoryDFARS 252.234-7002 Earned Value Management System
“(h) When indicated by contract performance, the Contractor shall submit a request for approval to initiate an over-target baseline or over-target schedule to the Contracting Officer.”
Guide 413.3-10B Integrated Project Management Using the EV Management System
6.1.2 Contractual Requirements.
“…if the contractor concludes the PB TPC and CD-4 date no longer represents a realistic plan, and an over-target baseline (OTB) and/or over-target schedule (OTS) action is necessary. Contracting officer approval is required before implementing such restructuring actions…”
Attachment 1, Contractor Requirements Document
“Submit a request for an Over-Target Baseline (OTB) or Over-Target Schedule (OTS) to the Contracting Officer, when indicated by performance.”
EVMS Compliance2Earned Value Management System Interpretation Guide (EVMSIG)3
Guideline 31, Prevent Unauthorized Revisions, Intent of Guideline
“A thorough analysis of program status is necessary before the consideration of the implementation of an OTB or OTS. Requests for establishing an OTB or an OTS must be initiated by the contractor and approved by the customer contracting authority.		EVMS Compliance Review Standard Operating Procedure (ECRSOP), Appendix A, Compliance Assessment Governance (CAG)
Subprocess G. Change Control
G.6 Over Target Baseline/Over Target Schedule Authorization
“An OTB/OTS is performed with prior customer notification and approval.”
See Section G.6 for a complete discussion on the process.
Contractor EVM SD4DCMA Business Process 2  Attachment, EVMS Cross Reference Checklist (CRC), Guideline 31.
“b. Are procedures established for authorization of budget in excess of the Contract Budget Base (CBB) controlled with requests for establishing an OTB or an OTS initiated by the contractor, and approved by the customer contracting authority?”		DOE ESCRSOP Compliance Review Crosswalk (CRC), Subprocess Area and Attribute G.6
“Requests for establishing an OTB or an OTS are initiated by the contractor and approved by the customer contracting authority.”

Notes:

  1. When DoD is the Cognizant Federal Agency (CFA), DCMA is responsible for determining EVMS compliance and performing surveillance. DCMA also performs this function when requested for NASA.
  2. Along with the related Cross Reference Checklist or Compliance Review Crosswalk, these are the governing documents the government agency will use to conduct compliance and surveillance reviews.
  3. For additional guidance, also see the DoD EVM Implementation Guide (EVMIG) , Section 2.5 Other Post-Award Activities, 2.5.2.4 Over Target Baseline (OTB) and Over Target Schedule (OTS). The EVMIG provides more discussion on the process followed including the contractor, government PM, and the contracting authority responsibilities.
  4. Your EVM System Description (SD) should include a discussion on the process used to request an OTB/OTS. The EVM SD content should be mapped to the detailed DCMA EVMS guideline checklist or the DOE Compliance Review Crosswalk (subprocess areas and attributes) line items.

Best Practice Tips

The best way to avoid getting a CAR from a government agency related to any OTB or OTS action is to ensure you have done your homework.

  • Verify your EVM SD, related procedures, and training clearly defines how to handle this situation. These artifacts should align with your government customer’s EVMS policy and regulations as well as compliance review guides, procedures, and checklists. Be sure your EVM SD or procedures include the requirement to notify and gain approval from the government PM and contracting officer, as well as what to do when the customer does not approve the OTB or OTS. Also discuss how to handle approving and managing subcontractor OTB/OTS situations; the prime contractor is responsible for these actions. Your EVMS training should also cover how to handle OTB/OTS situations. Project personnel should be aware of contractual requirements as well as your EVMS requirements and be able to demonstrate they are following them.
  • Maintain open communication with the customer. This includes the government PM as well as the contracting officer and any other parties involved such as subcontractors. Requesting an OTB or OTS should not be a surprise to them. Verify a common agreement has been reached with the government PM and contracting officer that implementing an OTB or OTS is the best option to provide visibility and control for the remaining work effort.
  • Verify you have written authorization from the government PM and the contracting officer before you proceed with implementing an OTB or OTS. You will need this documentation for any government customer EVMS compliance or surveillance review. Your baseline change requests (BCRs) and work authorization documents should provide full traceability for all schedule and budget changes required for the formal reprogramming action.

Does your EVM SD or training materials need a refresh to include sufficient direction for project personnel to determine whether requesting an OTB or OTS makes sense or how to handle OTB/OTS situations? H&A earned value consultants frequently help clients with EVM SD content enhancements as well as creating specific procedures or work instructions to handle unique EVMS situations. We also offer a workshop on how to implement an OTB or OTS .  Call us today at (714) 685-1730 to get started.

Formal Reprogramming: OTB or OTS Best Practice Tips Read Post »

Introduction to the IPMDAR Data Deliverable – Tips for Producing the Outputs

By / /

Contractors new to earned value management (EVM) often give us a call to help them respond to a government customer request for proposal (RFP) that includes the FAR or DFARS Notice of Earned Value Management System (EVMS) contract clause. Depending on the contract value threshold, the contractor will need to implement an EVMS that can at least produce the contract performance data submittals ($20M or greater) or complete a formal EVMS compliance review ($100M or greater) by a cognizant federal agency (CFA) such as the Defense Contract Management Agency (DCMA). Required data deliverables for a DoD or NASA customer reference the Integrated Program Management and Data Analysis Report (IPMDAR) Data Item Description (DID) as the means to submit monthly performance data. 

What is the objective for placing the IPMDAR on contract?

The government customer wants the monthly source schedule and cost data from the contractor for their own contract performance analysis in a standard format. They need to have reliable schedule and cost data for visibility into current contract performance as well as credible schedule and cost projections for the remaining work. Will the contractor complete the remaining work effort within the contractual schedule and cost targets? The government program manager needs this information for their own planning and budgeting as well as forecasting their funding requirements.

A standard format for collecting the data is important for the government customer so they can perform program portfolio analysis. The DoD established their EVM Central Repository (EVM-CR) to routinely collect contractor data submittals for program portfolio analysis. The data is organized using the MIL-STD-881 (the DoD Standard for Work Breakdown Structures or WBS), as a common basis to organize program data. The IMPDAR DID data submission requirements are defined in the File Format Specifications (FFS) and Data Exchange Instructions (DEI). The FFS and DEI specify the required set of data tables using JSON encoding for the IPMDAR cost and schedule data submissions while narrative text is submitted using Microsoft Word or PDF files so the customer can perform text searches. 

The government program manager can tailor the IPMDAR requirements as defined in the DoD IMPDAR Implementation and Tailoring Guide that complements the DID. For example, they can specify the level of data detail (control account or work package level), whether data can be delivered incrementally, variance analysis options, and requirements for the Performance Narrative Report content.

What is included in the IPMDAR deliverable?

There are three components as outlined below

IPMDAR Components

Notes:

  1. At a minimum, the IPMDAR requires data at the control account level with summary element of cost detail. The contract may specify work package level data.
  2. Inputs from a recent schedule risk assessment (SRA) should be included in the native schedule file submission when available. Depending on the schedule tool, the SRA data may need to be a separate file submission (Word or PDF). Results from the SRA along with other schedule analysis discussions (critical path, driving path, and schedule margin) are required to be included in the Detailed Analysis Report narrative. 
  3. The customer may request the results from a schedule data quality assessment and health metrics be included in the Detail Analysis Report narrative. 

What is required to produce the IPMDAR deliverables? 

For contractors new to EVM, one of their first objectives is to figure out what schedule and cost tools they need to be able to provide the required IPMDAR data and narrative analysis to their customer. H&A earned value consultants are sometimes asked to provide recommendations on commercial off the shelf (COTS) tools for this purpose. Much depends on what the contractor already has in place. 

Common schedule COTS tools such as MSP or Oracle Primavera P6 that have already been implemented will require an add-on to produce the SPD. Keep in mind that the IPMDAR does require SRA data and may require results from performing routine schedule data quality assessments. Some COTS add-ons to MSP or P6 are able to produce the typical schedule data quality metrics as well as produce the SPD. Other COTS scheduling tools such as Deltek Open Plan incorporate the SRA functionality, data quality metrics, and the ability to produce the SPD as part of the core product capabilities.

In most instances, contractors new to EVM do have an accounting/financial system in place to at least capture some level of contract or program/project budget and actual cost data. There may also be some capability to produce ETC data required for EAC financial analysis. The issue is organizing the complete set of time phased cost data (budget, earned value, actual cost, ETC) at the control account and work package level by summary element of cost that aligns with the schedule activity data. A contractor may be able to get by with Excel for a small project, however, it is time and cost prohibitive to create Excel macros to produce the CPD. Most COTS EVM cost tools are able to produce the CPD and have successfully completed the DoD EVM-CR data submission validation checks. This is a better alternative to building an in-house tool.

A data analysis tool such as Encore Analytics Empower is also a good option. Empower can import the time phased cost data from Excel or other COTS EVM tools and produce the CPD output. Empower can also import data from common COTS schedule tools. The benefit to using Empower is the ability to analyze the schedule and cost data in one place to verify alignment, produce interactive dashboards and a variety of analysis data views, and produce the IPMDAR Performance Narrative Executive Summary and Detailed Analysis Report.

Top Three Tips for Implementing Tools to Produce the IPMDAR Outputs

Here are a few tips from H&A’s earned value consultants on implementing tools to support the IPMDAR data submittals. Focus on getting the basics right.

  • Continuously verify the quality of the schedule and cost data. Routinely perform schedule data quality assessment and health checks to proactively resolve schedule construction, status, or data issues. Perform routine cost data validation checks such as earned value and no actual costs for a work package or the cumulative to date earned value exceeds the budget at completion (BAC). Correct all data anomalies before producing the deliverables.
  • Continuously verify the schedule and cost data are in alignment. Consistent schedule and cost data coding is critical to ensure integration and traceability. 
  • Anticipate the scope and level of data detail required. This can impact tool configuration, data structures, and data pulled from other business systems such accounting. For example, be prepared to provide the work package level of detail; actual costs will need to be available at this level. Another example is providing schedule risk assessment inputs; this is usually required at intervals specified in the CDRL.

H&A earned value consultants routinely help clients with constructing the schedule to support the IPMDAR data requirements, setting up the process to do routine data quality checks, integrating the schedule and cost data, and verifying the data before producing the performance reporting data submittals. Another common focus is producing clear and concise variance analysis narrative content. 

We can do the same for you. Call us today at (714) 685-1730 to get started.

Introduction to the IPMDAR Data Deliverable – Tips for Producing the Outputs Read Post »

Comparing the Efficacy of Independent Estimate at Completion (IEAC) Methods Using Real Project Data

By / / ,
,
Comparing the Efficacy of Independent Estimate at Completion (IEAC) Methods Using Real Project Data

“Data! Data! Data!” he cried impatiently. “I can’t make bricks without clay.”

-Sherlock Holmes, The Adventure of the Copper Beeches

There are many discussions about EACs and evaluating EACs including using Independent EAC (IEAC) formulae to compare with the contractor EACs. With good reason, we should wonder how accurate are those IEACs that we use so often and sometimes make decisions based on them. Are we misjudging contractor’s EACs based on formulae that are weak or inappropriate?

Humphreys & Associates has initiated a study to determine how accurate IEACs are, and we would like your help. The study will compare different IEAC formulae against the Program Manager (PM) most likely EAC at the 25, 50, and 75 percent complete point for completed projects. The objective is to assess how closely the IEACs and PM most likely EAC were able to predict the final cost outcome for the project.

How Accurate are IEAC Formulae?

Many formulae exist for using recorded data from an earned value management system (EVMS) to make independent estimates of the final cost at completion (EAC) for the element in question. The element might be a control account, a Work Breakdown Structure (WBS) element, or even an entire project.

What is not known is how accurate these methods are at forecasting the final actual cost for the project. This study hopes to determine that answer.

Real World IEAC Data

This study was initiated by collecting earned value data from 12 completed projects. We need projects that are completed because, on a completed project, the final actual outcome is known. We collected project data at the 25, 50, and 75 percent complete points. At each of these points, the IEAC formulae were applied to determine how closely they were able to predict the final actual cost outcome for the project. The quest is to learn how the various IEACs performed. Is any one of them more accurate than the others?

From this investigation, any indication of the relative efficacy of the formulae would be used to inform future use of the IEAC methods.

Our Method for Testing IEACs

In general, the IEAC approach is to use existing recognized formulae. We have chosen these IEACs as a starting point:

  • IEAC 1 = BAC/CPIe at the percent point reported. This formula can be stated in words as “the entire project is performed at the same efficiency as experience to date.”
  • IEAC 2 = ACWP + [BCWR/CPI (.5) + SPI (.5)]. This formula uses weighted SPI and CPI which theoretically allows for sensitivity to both cost and schedule historical performance. The weights used in this application are even at .50 and .50.
  • IEAC 3 = ACWP + [BCWR/CPI x SPI]. This formula uses the SPI and CPI multiplied together which theoretically allows for sensitivity to both cost and schedule performance to date.
  • IEAC 4 = ACWP + BCWR. This formula assumes the remaining work will be done as budgeted with no factoring.

One additional non-traditional IEAC will be used.

IEAC 5 = Use of IEAC 2 weighted SPI and CPI but decreasing the proportion applied to the SPI as the percent of project completion increases. In other words, the impact of schedule performance diminishes as the project becomes closer to completion.

We will also take the average of all the formulae to see how that works.

Initial Data Set

One aerospace contractor and one US Government agency have provided the required data for 12 completed projects with an interest in the outcome of the study. The source of the data and the specific projects will not be disclosed in the study.

These real-world projects did not have an exact 25%, 50%, or 75% dataset. The closest dataset to each of those completion percentages was used. One example dataset looks like this (color coding should be ignored):

Example Product Data

How can you help?

We need more project data to gather enough varying project outcomes to make the test realistic. We do not plan to keep the types of projects or products separate but will take all the data we can get and look at them all.

Please consider providing data for the study. We have created an Excel spreadsheet template to help gather project data in a common format for analysis. You can download this template here. Add as many tabs as needed for each project. Send your completed spreadsheet to humphreys@humphreys-assoc.com.

In a separate blog we will outline other help we need to complete the study and to analyze the results.

IEAC_Study_Project_Data_Template-Rev-06052023Download

Comparing the Efficacy of Independent Estimate at Completion (IEAC) Methods Using Real Project Data Read Post »

Scroll to Top