EVM Terminology

EVM Terms

EVM Consulting – Modeling & Simulation

, , , , , , , , ,

Fighter Jet Air Plain Flying in Front of Moon

Forewarned is Forearmed

Forewarned is forearmed. John Farmer, of New Hampshire, said that in a letter in 1685. But that advice is most likely biblical and very much older. No matter the source of the thought, we should take it as divine guidance if we are project managers. Maybe we should have it cut into a stone tablet, so we can share it with our team members.

Most of our work as project managers is spent in the “controlling” phase which is made up of the three steps “measure, analyze, act.” Our EVMS and IMS exist to be able to support this management function. The measuring part is done very well in our EVMS and our IMS; we know where we are and how we got there. The analyzing is equally well handled in the IMS and EVMS. Only the management task of acting is not well supported. Generally, we lack decision making support and tools.

EVM Consulting - Measure, Analyze, Act

Deterministic Path

No matter how well constructed and how healthy our IMS is, it has a deterministic path forward. The logic links between the activities are there because we expect them to be fulfilled. Indeed, if activity “B” is a finish-to-start successor to activity “A” we fully expect that at some point activity “A” will finish and will provide its output to activity “B”. That is a single path forward and it is a deterministic path. It is also a somewhat simplistic model.

EVM Consulting - Deterministic Relationships in EVMS

Multiple Outcomes

Our management system asks us to perform root cause analysis followed by corrective action. But what if there is more than one corrective action to be taken. And worse; what if the corrective actions can have multiple outcomes with each enjoying its own probability. That means multiple choices and multiple outcomes. How would we show that in our plan? How would we analyze the multiple possible futures that such a situation presents?

Happily, there are ways to model a future without a set path. And once we have the future model, there are also ways to simulate the outcomes to give the probabilities we need to decide which actions to take. We are talking about probabilistic branching, and we are saying that we can build a probabilistic map of the future to use in making decisions; especially making decisions on corrective actions.

Take a simple example of running a test on the project. The expectation is that eventually we will pass the test. We will keep trying until we do. In the IMS deterministic model the test portion of the IMS might look like this:

EVM Consulting - Run the Test then Use the Product

Simulation

We can simulate this situation with different expected durations for the test. That is helpful information, but it does not explain or even capture what is going on in those different durations. It looks like we are just taking longer to do the testing but is that really what is happening? What is going on here? We certainly don’t show that.

In the real world, this simple model might have three potential outcomes. There might be three paths we can take to get to the point where we use the product. Each path has a time and money cost. We might run the test and find that we passed. Or we might have to stop the test for issues on the item or the test setup. We might even fail the test and must correct something about the product to improve our chances of passing a rerun. Eventually we will get to a usable product. But what do we put in our estimate and our plan? What do we tell the resources we need? What do we tell the boss? The customer?

EVM Consulting - Real World Testing

Full Future Model

We now have a much better understanding of the future and can explain the situation. We also can simulate the situation to find out the most likely time and cost outcomes, so we can explain the future without any histrionics or arm waving.
If the issue is important enough we can build out the full future model and simulate it.

EVM Consulting - Full Future Model and Simulation

No matter how far we pursue the model of the future, having a valid model and being able to stand on solid ground are very valuable to us as project managers.

This is not to say that we should model out complex situations as a routine in the IMS. That would be impossible, or at least prohibitively costly. We are saying that when situations arise, we need to be able to use the IMS to help us make decisions.

This type of probabilistic modeling of the future is particularly useful in defining major decision points in our plan. When we reach a decision point the IMS may have multiple branches as successors but that implies we take every branch and that is not valid. Modeling each branch and its probabilities is valid. In the example below, where the milestone represents a decision point, we have shown three possible paths to take. If each were modeled out into the future with time and cost data, we should have the information we need to choose the path we wish to pursue. Without processes and tools like this, we would be flying blind.

Future Blog Posts

This discussion will be continued in future blogs to develop a better foundational understanding of the process and power of probabilistic modeling in our EVMS.

EVM Consulting - Decision Point

Good information sets the stage for good decisions. The IMS and the EVMS have sufficient information to help us model the pathways ahead of our critical decisions. We just need to learn to take advantage of what we have available to us.

Find out how an experienced Humphreys & Associates EVM Consultant can help you create a full future model and simulation of your most vital EVMS Systems. Contact Humphreys & Associates at (714) 685-1730 or email us.

EVM Consulting – Modeling & Simulation Read Post »

Project Management: Earned Value Consulting; Could You Use Some?

, , , , ,

Failed Projects

A recent article discussed the results of a survey on the reasons that projects failed. The definition of failure was that the project was abandoned. Abandonment does not occur frequently in the world of government projects; especially defense projects where there should be strong “must have” needs driving the project. These projects tend to persist until completed even though the outcomes are not satisfactory. But there is a lot to learn from the list of reasons for failure.

Of the sixteen reasons listed, the top four had to do with changes to the environment that had given rise to the project. For example, changes in the company priorities was the most often cited reason for abandonment. In that same vein were issues with changing objectives and inaccurate definition of requirements. These types of failures are not topics for this blog since they do not immediately involve execution of the project.

Execution Problems

Lower on the list of those environment related reasons for failure were the ones more related to execution problems. These are of significant interest to a project management using an earned value consulting company such as Humphreys & Associates (H&A). These reasons related more to issues of poor project management that could have been corrected. In this area were reasons like “poor change management,” “inaccurate cost estimates,” “inaccurate time estimates,” and “inexperienced project management.”

The answers given in a survey situation depend very much on the mindset of the person responding. Is the reason really “inaccurate cost estimates” or should it have been “failure to execute to the estimate”? How many times have you seen a problem in execution “swept under the carpet” as being an inaccurate estimate or plan? One of these two answers points to the estimating system and process while the other points to project management. The estimates were generated; and, at some point, they were deemed to be sufficiently detailed to launch the project. If a scrubbed and blessed estimate is “inaccurate” that would still be a failure of project management. If the problem were really a failure to execute, then how easy would it be to blame the problem on poor estimates? This blog will discuss the cited failures as if they were execution failures.

Earned Value Management (EVM) Consultant Specialists

There are situations in life where the need for specialized advice is common and well accepted by us all. When your doctor is unsure of the medical issues, the doctor will send you to a specialist. The reason is obvious. The specialist has learned so much more about a specific problem and has so much experience diagnosing and treating the problem that it would be foolish not to secure the services of that specialist. In fact, it might be malpractice. A project management consultant can be thought of much like a medical specialist.

There are similar situations in business where the need for specialized knowledge is critical. Large companies tend to have in-house legal departments to cover the day-to-day legal issues and tasks that are central to their businesses. However, the need to go to outside counsel for large or unusual issues is accepted. Companies do not hesitate to engage the services of outside law firms to help them through troubled times. Project management consultants are like outside counsel.

What if there were a project management or earned value management situation you have never encountered before? A good example would be the times that H&A has been called in to help clients navigate the unhappy circumstances of needing to go over-target. Going through the over-target-baseline (OTB) or over-target-schedule (OTS) process is not a common experience. It is a tense time when careers can be on the line and the company reputation might also be at risk. It takes specialized knowledge to get it right. In some cases, it even takes the objective view of an outsider to help make the right decisions.

Specialized Knowledge

Another example of specialized knowledge being crucial is when the customer has deemed some issue on the project to be deficient. In some situations, a customer’s Corrective Action Request (CAR) can result in cost penalties and damaged reputations; possibly even worse consequences could result. Engaging the services of an EVM consultant with experience in identifying problems, building Corrective Action Plans (CAP)s, and leading or helping implementing the corrective actions is often a valuable and necessary action. Ask yourself how smart it would be to assume that those who were involved in causing the issue would be capable of creating a satisfactory solution.

These scenarios are aligned with the idea of project management consulting being something you only need in a crisis. There are other non-crisis needs for specialized support. Often H&A is engaged simply to help a client prepare a proposal. A proposal situation puts heavy demands on the company staffing levels and can require areas of specialized knowledge not available in the company. What if the company has never created a fully compliant Integrated Master Schedule (IMS) and they could use help the first time? What if there are not enough trained and experienced schedulers to work on the proposal? What if the company does not have a documented project management system?

Make or Break Opportunities

Projects can be huge and risky. They can be make-or-break opportunities to a company. Where so much can depend on good project management, smart companies recognize the need for an outside opinion and outside talent. Just like the internal legal department, the internal project management group sometimes needs to call on outside subject matter experts. While it might be obvious, let’s look at some reasons why this is true.

There are more ordinary everyday reasons to engage a project management consultant. Perhaps an organization just managed to win a new project bigger than any they have won before. In this case, they may not be ready to handle the project in terms of experience, systems, and even just talented headcount. A project management consulting company such as H&A can bring solutions to your earned value woes. It can also provide temporary training staff to get things going until the client is ready to take over.

Poor Communication

Let’s get back to the survey of reasons that projects failed. Are there issues on the list where project management consulting could have made a difference? Imagine an improved project management process and staff after a period of consulting to support creating or improving systems and training personnel?

The fifth most frequent reason for failure is “poor communication.” A good project management system with trained personnel is all about communication. Communication of plans, communication of progress, communication of issues, and communication of corrective actions are all actions required in a project management system. Quite often the problem of “poor change management,” cited as the sixth most common reason for failure, is reduced or eliminated after using the consulting services of a specialist?

What about the twelfth cited problem of “inadequate resource forecasting”? Would a well built and maintained resource-loaded Integrated Master Schedule (IMS) go a long way in providing forecasts of resource needs and the impacts of not having the resources? In fact, a proper IMS would help with several of the cited reasons for failure, such as inaccurate duration estimates. In fact, the application of a process, such as Schedule Risk Analysis (SRA), with the help of an experienced consultant can identify such issues in advance while there is still time to take action.

Earned Value Training

Disregarding the threat of failure as a motivator, the need for constant improvement should be enough reason to consider a project management consultant. We can all laugh at the time-worn clichés of “not-invented-here” or “we’ve never done it that way;” however, these are clichés for a reason. There is resistance to outside help and there is resistance to change. But outside help can be a great logjam breaker. An experienced and knowledgeable consultant can be your voice when you need someone who has, to use another cliché, “been there and done that.”

In fact, our consultants can laugh when they say they have “been there” and they have more than a T-shirt to prove it.

Project Management: Earned Value Consulting; Could You Use Some? Read Post »

Updates to the Compliance Review Series of Blogs

,

Time Lapse of Freeway representing the progress of time.

2020 Update

Humphreys & Associates has posted a 2020 update to the series of blogs discussing the DCMA Compliance Review (CR) process. “Compliance Review” is the term used for the formal EVM System review DCMA performs to determine a contractor’s compliance with the EIA-748 Standard for EVMS guidelines. This can also include, as applicable, Surveillance Reviews and Reviews for Cause (RFC).

DCMA used to follow a 16 Step compliance review process. This changed to an 8 Step process with the release of the DCMA Instruction 208 (DCMA-INST 208) titled “Earned Value Management System Compliance Reviews Instruction.” This Instruction has been rescinded and replaced with a set of DCMA Business Practices (BP). These Business Practices split out topics that the old DCMA Instruction 208 covered in one document. Whether you are a contractor new to the EVM contracting environment or a seasoned veteran, if the Earned Value Management System (EVMS) compliance and acceptance authority is the Defense Contract Management Agency (DCMA), these new Business Practices apply to you.

The four updated blogs include:

  • EVMS Compliance Review Series #1 – Prep for the DCMA Compliance Review Process. This blog presents the set of DCMA Business Practices (BP) that define the EVMS and Review process and specifically discusses Business Practice 6 “Compliance Review Execution.” It also discusses what you can expect should you need to complete the DCMA Compliance Review process through the 5 phases and 23 Steps outlined in BP6. It is critical you are able to complete each step in the process successfully the first time through to prevent delays. The best way to make sure you are prepared is to conduct one or more internal EVMS Mock Reviews, the topic for the next blog.
  • EVMS Compliance Review Series #2 – Conducting Internal Mock Reviews (Self Assessments). This blog discusses the importance of conducting a thorough internal review of your EVMS. You may or may not have the expertise in-house to conduct this simulation of a Compliance Review. An independent third party can help you prepare for a DCMA compliance review. The objective is to conduct the EVMS Mock Review to simulate everything DCMA will do. DCMA also expects a thorough scrub of the schedule and cost data – data traceability and integrity is essential.
  • EVMS Compliance Review Series #3 – Using Storyboards to Depict the Entire EVMS. Do you need a refresher on the role of storyboards in a compliance review? Storyboards can make a difference in training your personnel and explaining to the DCMA personnel how your EVMS works. Storyboards can take many forms, and if you don’t have one in place, consider starting with the flow diagrams in your EVM System Description.
  • EVMS Compliance Review Series #4 – Training to Prepare for Interviews. This blog highlights the importance of conducting training for your personnel, particularly the control account managers (CAMs), so they are able to complete successful interviews with DCMA personnel. H&A recommends completing a three step training process to proactively address any issues.

Help Preparing for a Compliance Review

Do you need help preparing for a DCMA compliance or surveillance review? Download the set of DCMA Business Practices and read our updated blogs so you have an idea of what is ahead. Humphreys & Associates can help you conduct a Mock EVMS Review, perform a data quality assessment, create a storyboard, or conduct EVMS interview training and mentoring for your personnel. Call us today at (714) 685-1730 or email us.

Updates to the Compliance Review Series of Blogs Read Post »

Formal Reprogramming – What Happened?

, , , , , , , , , , ,

Graph of an Increasing Budget

A long time ago, in a galaxy far, far away….an Over Target Baseline (OTB) – by design – was a rare occurrence (and the OTS concept did not even exist as part of Formal Reprogramming). Formal Reprogramming was a very difficult and cumbersome process that most contractors (and the government) really did not like to consider. The government, in its 1969 Joint Implementation Guide, said:

“Reprogramming should not be done more frequently than annually and preferably no more frequently than once during the life of the contract.”

The Office of the Under Secretary of Defense (OUSD) Acquisition, Analytics and Policy (AAP) – formerly PARCA – , in their latest OTB/OTS guide, states that Formal Reprogramming now has expanded to include an Over Target Schedule (OTS).  However, in that guide, it is stated in Paragraph 1.3.8:

“Ideally, formal reprogramming should be done no more than one time during the life of a contract. However, there may be instances where another formal reprogramming is warranted… When formal reprogramming is accomplished in accordance with the procedures in this guide, with a realistic cost and schedule estimate established for the remaining work, it should not be necessary to undergo formal reprogramming again.”

Today, though, whenever contractors incur a significant cost or schedule variance, instead of resolving the variance cause, the first words seem to be: “Let’s do an OTB or OTS.”  The lure of “getting rid of cost and schedule variances” seems too good to pass up.  Unfortunately, an OTB/OTS implementation has never been an instantaneous process. With AAP’s 12 step OTB/OTS process, it is obvious that the contractor will not be able to start today and incorporate the OTB/OTS in the next Integrated Program Management Data and Analysis Report (IPMDAR) dataset. In fact, AAP’s OTB/ OTS guide states in paragraph 3.8:

“It may be difficult to ascertain the length of time it will take to implement a new baseline based on the scope of the effort. It is not uncommon for the entire process to take up to six months which would be too long of a period without basic cost reporting.”

The last line of the above cited paragraph was referencing the reporting requirements to the customer when an OTB/OTS is being implemented.

The IPMDAR Implementation and Tailoring Guide (5/21/2020) even recognizes the issues with timeliness of implementing an OTB/OTS:

2.3.2.5.5  Formal Reprogramming Timeliness. Formal reprogramming can require more than one month to implement. During formal reprogramming, reporting shall continue, at a minimum, to include ACWP, and the latest reported cumulative BCWS and BCWP will be maintained until the OTB/OTS is implemented. 

So why does it take so long to implement the OTB/OTS?  Can the contractor just adjust the bottom line variances and move on?  Actually no, nothing is really that simple.  This is one of the reasons that implementing an OTB and OTS should not be taken lightly.   The AAP OTB/OTS Guide addresses adjustments this way:

“3.5.6.2 Adjusting Variances: A key consideration in implementing an OTB is to determine what to do with the variances against the pre-OTB baseline. There are essentially five basic options. This is a far more detailed effort than these simple descriptions imply, as these adjustments have to be made at the detail level (control account or work package).”

When considering the number of control accounts and work packages involved in a major contract, a Formal Reprogramming can become a rather daunting task.  The contractor also has to report the effects of the Formal Reprogramming in the IPMDAR Reprogramming Adjustments columns. These adjustment columns appear on both Format 1 and Format 2 of the IPMDAR database, which means the contractor must undertake the assessment for both the contract’s WBS and the OBS – for each WBS element and for each OBS element reported.  This can be further complicated if the OTB/OTS exercise were flowed down to subcontractors for a given program.  The AAP OTB/ OTS Guide, paragraph 3.8 also states:

“The customer should be cognizant of the prime contractor’s coordination complexities and issues with its subcontractors. The time to implementation may be extended due to accounting calendar month overlaps, compressed reiterations of contractor ETC updates, internal reviews, subcontractor MR strategy negotiations, senior management approvals, etc., all while statusing the normal existing performance within a reporting cycle.”

In the early days, when implementing an OTB with variance adjustments, the company and the customer agreed on a month-end date to make the data adjustments.  Then the contractor ran two CPRs or IPMRs (now the IPMDAR): (1) the first report as though no OTB had been implemented [to determine the amount of adjustments to cost variance (CV) and schedule variance (SV) at all the reporting levels] and, (2) the second report [after the OTB implementation had been completed – no matter how long it took] showing the Column 12 adjustments plus whatever BAC changes were being implemented.

Under the current OTB/OTS Guide, it appears as though this process is being done all at once. As stated in the AAP OTB/ OTS Guide paragraph 3.8 above, this implementation could take up to 6 months to complete, so lagging the second report until the OTB/OTS implementation is completed seems logical. The last sentence in paragraph 3.8 also stipulates that regardless of how long implementation takes, the contractor and customer will agree on interim reporting that will be required, further stating that:

“In all cases, at least ACWP should continue to be reported.”

Perhaps this agreement with the customer should also specify the content of the first IPMDAR following OTB/OTS implementation.

All things taken into account, the process of requesting and getting approval for an OTB or OTS can be a long and difficult process, especially if, at the end of it all, the contractor’s request is denied.  Even if it were approved and the contractor implements and works to the newly recognized baseline, immediately doing another one is not a pleasant thought – and remember, it was not intended to be pleasant. Reprogramming was always supposed to be a last resort action, when reporting to the current baseline was totally unrealistic.

Now, what about those cases where a contract has one or two elements reporting against totally unrealistic budget (or schedule) baselines?  The AAP OTB/ OTS Guide does cover a partial OTB, but reiterates that this is still an OTB because the Total Allocated Budget (TAB) will exceed the Contract Budget Base (CBB).  In the early days, however, the government allowed what was called Internal Operating Budgets (IOBs) for lower level elements (control accounts, or specific WBS elements, etc.) that were having problems resulting in an unrealistic baseline for the work remaining. The 1987 Joint Implementation Guide, paragraph 3-3. I (5) described IOBs as follows:

“(5) Internal Operating Budgets. Nothing in the criteria prevents the contractor from establishing an internal operating budget which is less than or more than the total allocated budget. However, there must be controls and procedures to ensure that the performance measurement baseline is not distorted.

(a) Operating budgets are sometimes used to establish internal targets for rework or added in-scope effort which is not significant enough to warrant formal reprogramming. Such budgets do not become a substitute for the [control] account budgets in the performance measurement baseline, but should be visible to all levels of management as appropriate. Control account managers should be able to evaluate performance in terms of both operating budgets and [control] account budgets to meet the requirements of internal management and reporting to the Government.

(b) Establishment and use of operating budgets should be done with caution.  Working against one plan and reporting progress against another is undesirable and the operating budget should not differ significantly from the [control] account budget in the performance measurement baseline. Operating budgets are intended to provide targets for specific elements of work where otherwise the targets would be unrealistic. They are not intended to serve as a completely separate work measurement plan for the contract as a whole.”

Current literature no longer specifically addresses Internal Operating Budgets (IOBs), but with the recent trend of contractors jumping to the OTB/OTS conclusion, it might be a better alternative to have individual instances of unrealistic budgets (or schedules) that do not otherwise push the total program to the need for a complete OTB and/or OTS implementation.

These could be good discussion topics for future AAP and DCMA meetings with industry representatives, to determine if there are ways to streamline the process, or at least reduce the amount of requests to implement Formal Reprogramming.  Variances are, after all, performance measurement indicators that should not just be routinely and artificially eliminated.

Formal Reprogramming – What Happened? Read Post »

Along the IMS Time-Now Line

, , , , , , , , , , , ,

Arrows moving to the right.Recently one of our consultants was instructing a session on the Integrated Master Schedule (IMS) with a group of project personnel from one of our larger clients. The group was a mixture of beginners with no real experience in schedules and some much more experienced practitioners; some with more than 10 years of experience. The mixture made it somewhat difficult, but it also made for some interesting discussions that might have been missed in a more homogenous group. One of those things was the usefulness or importance of the “time-now” line.

When the group was asked about the importance of the time-now line and what information could be easily gained from a look at the line, there was silence. The beginners did not have a clue but also none of the experienced people had any response. What should have been a short discussion with just one “slide” as a visual, turned out to be a longer and more informative session on this topic.

The time-now line has different names in different software tools but it refers to the data date, or status date, of the schedule. That also would be the first day of the remainder of the schedule. When a scheduler sorts tasks by date, the time-now line runs down the screen and forms a highly useful visible reference.

In the small example below [see Figure 1], you can see the time-now line and visually assess the situation. Time-now is shown by a vertical line at the beginning of September, so all remaining effort has been scheduled to after that date. In other words, no work can be forecasted in the past. A walk down the line shows Task 1 has both started and completed. Task 2 started but has not completed. In fact, the remaining work in Task 2 has been pushed out by the time-now line. The start of Tasks 5 and 9 are also being pushed out by the time-now line. In most real project schedules, filters and other techniques may be needed to isolate information like this; but in our small example, we can simply “eyeball” the time-now line and see valuable information. Task 9 starts the critical path shown in red tasks.

 

The project start date was August 1. The status date is September 1. Tasks 2, 5, and 9 show gaps from the predecessor to their starts. in the case of Task 2 the cap is to the start of the remaining work. This gap is caused by the time-now being set to September 1 with all remaining work starting after that. The critical path is being pushed by time now.Figure 1

 

A slightly different setup for that same small example [see Figure 2] shows something interesting. The time-now line is still at the beginning of September. But now there is a gap between time-now and work on the critical path. This is an unusual situation and should be investigated for the root cause. It is possible this is an accurate portrayal of the situation, but regardless of the cause, it must be verified and explained.

 

Time now is still at September 1. There is a gap on the critical path at the start of Task 9 which, in this case, is caused by a Start-No-Earlier-Than constraint.Figure 2

 

In yet one more variation [see Figure 3], we see that a broken link results in Task 8 ending up on the time-now line. A task without a predecessor will be rescheduled to start at the earliest possible time (if the task is set to be “As Soon As Possible”). And the earliest possible time is the time-now line; the beginning of September. Just as broken things fall to the floor in real life, “broken things” fall to the time-now line in a schedule. Un-started work can land there. Un-finished work can land there. And un-linked work can land there.

It is further possible to see that Task 2 has had an increase in the remaining duration that has driven it onto the critical path. Task 2 at this moment is the most important task on the entire project. A slip to Task 2 will drive out the end date for the entire project. One question that needs answering is what is holding up Task 2?

If the display had been sorted by increasing total float/slack and the usual cascade by date, then the critical path would be starting at the upper left-hand corner; like the critical path in this example. The action on the project is almost always on the time-now line and the most important action, when sorted as described, will be at the upper left-hand corner.

 

Task 2 is now driving the critical path. Task 8 has fallen back to the time now line. The constraint on Task 9 has been removed.Figure 3

 

So, a walk down the time-now line can help us see the critical path action, find broken parts of the schedule, and locate unusual circumstances that need our attention. Our recommendation is to look at the time-now line any time there is data being changed in the IMS. This will help you catch issues early and keep the schedule cleaner.

Along the IMS Time-Now Line Read Post »

Agile/Scrum Ceremonies and Metrics Useful in EVMS Variance Analysis and Corrective Action

, , , , , , , , , , , , , , , , , , , , , , , , , ,
Agile Scrum EVMS

P. Bolinger, CSM October 2016
Humphreys & Associates

How can Agile/ Scrum be used to support EVMS Variance Analysis and Forecasting in a way that provides program managers with cost and schedule integrated information for no extra effort?

The discipline of EVMS and the Agile/Scrum practices have several touch-points that are covered in two major documents: NDIA IPMD Agile Guide March 2016, and PARCA Agile and EVM PM Desk Guide. Neither of these documents, as yet, drives to the level of specifics when it comes to best practices for use of Agile to support EVM Variance Analysis and EVM Forecasting.

Looking at the literature for Agile/Scrum, we know that there are recommended ceremonies that are conducted at various levels of the product structure and at different times during the project life cycle. These ceremonies are supported by many discussions of the metrics that can be collected at each ceremony and their potential use in managing the technical work of development within the Agile/Scrum framework. But where do these ceremonies potentially support EVMS Variance Analysis and Forecasting?

Now suppose we are presented with a control account that has exceeded the EVMS thresholds for cumulative cost and schedule variances. Wouldn’t it be great to have at our fingertips the underlying data from the process? In this case we might find, for example, that the Velocity is less than that needed to meet the end goal, the story cycle time is longer than desired, the pass/fail ratio is not favorable, too many team members have been absent in the last Sprint, the number of disruptions has been excessive, and the work to accomplish the stories is higher than estimated. It is not difficult to surmise the outcome would likely be a behind schedule and overrun condition in the EVMS. These data measures would provide the fodder for deep diving to the root cause and impact statements.

Where would we get that information?

Let’s start with the Agile/Scrum ceremonies. The particular Agile/Scrum ceremonies that we find conducted during the project are:

Backlog Refinement. This ceremony can be nearly continuous. It involves redefining the backlog of development work (scope), the prioritization or re-prioritization of that work, and potentially the assignment of responsibilities for the backlog.

Release Planning. This is a recurring ceremony aligned with the release cadence for the project. It involves establishing the capabilities and features of the product and when they will be released.

Sprint. This is a short time-defined effort to accomplish the design, code, and test of some subset of the product. The Sprints are controlled by the self-managing teams.

Daily Scrum or Standup. This is a daily (recommended) team meeting to discuss what has happened, what roadblocks exist, what is planned for the day, and other necessary items.

Sprint Review. The session in which the team products are demonstrated to the owner and self-off is accomplished.

Sprint Retrospective. A meeting of the stakeholders to discuss what went right (or wrong) during the Sprint and to define improvement actions that are needed.

The relationship of these Agile ceremonies with EVMS might look like this:

Agile/Scrum Ceremony

Agile Purpose

Relationship to EVM Variance Analysis, Root Cause Analysis, Corrective Action Planning & Follow-up and Forecasting

 

 

 

Backlog Refinement Manage, estimate, prioritize, and organize the product backlog in an on-going routine. Estimating – impacts the EVMS ETC and EAC as well as durations of efforts.
    Prioritizing – in response to issues is corrective action management.
    Organizing the backlog could be a form of corrective action effort.
     
Release Planning Establishing the contents and timing for releases of product. Updates could be part of corrective action planning in response to issues. Creation of new work packages. Changes to planning packages and SLPPs would also.
     
Sprint Short time-box performance unit. Work is done in Sprints. Below the work package. Short span measurement period is possible.
     
Daily Scrum and Stand-up Make short term plan, adjust to issues, discuss problems, clear roadblocks. Much of the daily action would relate to root cause analysis and corrective action planning although the time-frame is very short and the issues may be too small to individually impact feature work package or the Epic control account.
     
Sprint Review Demonstrate the product, update released work, make changes to product. This would relate to corrective action planning and follow-up. Issues would be found here that would impact risks, ETCs, corrective actions, performance metrics.
     
Sprint Retrospective Reflect on the project, progress, people processes, what was good, what was bad, and take actions to improve. This should be the richest source of supporting information for EVMS root cause & corrective action within the VAR realm. Very timely for variance analysis as it happens potentially many times during work package (feature) duration.
Feature Retrospective (not one of the basic ceremonies) Review situation regarding technical scope deficit. Reflect on the project, progress, people processes, what was good, what was bad, and take actions to improve. Because this only happens at the end of the feature it is limited in value for variance analysis timeliness. Any lessons learned can only be applied to future feature work.

But where is the meat? Where do we get actionable data or at least data we can analyze to decide what management efforts are required?

There are numerous potential metrics that can be collected during these ceremonies. These metrics can form the basic data set that could be analyzed to define the root cause of cost and schedule variances. In addition to isolating the cause of issues, within some of these ceremonies the impact of the issue on the Sprint, or Feature, or team may be made. Certainly, these metrics can be used as the basis for projecting future workload and performance.

The total number of potential metrics is not known. In this paper, we looked at 17 metrics and considered what the data might mean. The results of this review are contained in this matrix:

Metric

Type of Measure

Discussion

Sprint Burn up/Burn Down Backlog Value of backlog remaining for Sprint. Decrease is expected when work is done. Increase means work increased. Burn up can include total completed plus remaining; a great metric.
Feature Burn Up/Burn Down Backlog Value of backlog remaining for Feature. Decrease is expected when work is done. Increase means work increased or shifted.
Customer support requests received. Disruption Number of instances. Unplanned interruptions by customer can lower the output of the team if excessive.
Disruption measures Disruption How many and what type (except customer support requests). Higher disruptions impact team efficiency.
Estimate Accuracy (Sprint or Feature) Estimating Measure of budgeted value (estimated value) for the Stories in the Sprint or Feature versus the actual cost (calculated cost) of the Stories when done. Related to team size.
Discovered work Estimating Emerging work discovered during the Sprint. Will translate to extra effort in future if adopted into backlog.
Exceeds WIP Limits Management If WIP limits are set on team or individuals then exceeding set limits will impact efficiency and output.
Retrospective Action Log Management Count of improvement actions listed in Retrospective. Increasing count means issues are not being resolved.
Attendance Management Comparison of actual hours worked by team compared to baseline expectations in plan.
WIP Productivity Measure of the number of stories or points in WIP at any time. WIP growth can indicate bottlenecks and inefficiencies.
Velocity Productivity Measure of the amount of work (Stories or Points) accomplished during a time period. Higher velocity means greater throughput per person/team.
Stability measures Productivity Comparing Sprint by Sprint basic measures from this list. If there is high variability between Sprints in measures, then future is unpredictable.
% Tests Automated Productivity Higher automated testing should increase efficiency & decrease cycle time.
Defects found by team Quality Number of bugs reported during team effort. Measures quality of work. Higher bug incidence translates to lower output and higher costs.
Defects found by customer Quality Number of bugs reported by user/customer. Measures quality of work delivered. Higher bug incidence translates to lower customer satisfaction and higher rework costs.
Pass/Fail (Re-do) measures Quality How does the rate of success in testing compare to the number of attempts? High success rate should mean greater output and efficiency.
Cycle Time Schedule Time from start of a story to complete. Short cycle time is desired.

Let us continue the theme of the behind schedule and overrun control account and look at what information would be available for support to developing the estimate-to-complete. An updated and refined backlog would have the scope of work remaining for the control account. The updated release plan would have the timing for the deliveries to be made in the control account. The metrics collected about the effort expended per accomplished story or story point would provide a factor for projecting future real-work hours. Planned corrective actions and improvements would tell us how we might expect improvement to the quality and improvement in the speed or cost of work. The insights available from a full set of metrics are impressive.

Does a project have to collect all of these metrics? If not all then which ones would be the right ones? Questions like that would be answered by the project management team analyzing their prior experience and the particular challenges of the project. The team would establish a data collection plan, likely described in their Software Development Plan or Program Management Plan that would explain the metrics, meaning, and frequency along with their purpose. With a clear understanding of the technical data to be collected and analyzed, the Control Account Managers would not have a difficult task to define how they would use that data in developing Variance Analyses and generating well-considered Forecasts. In fact, these tasks should be much simpler with the data in hand.

Agile/Scrum Ceremonies and Metrics Useful in EVMS Variance Analysis and Corrective Action Read Post »

Scroll to Top