Agile/Scrum Ceremonies and Metrics Useful in EVMS Variance Analysis and Corrective Action

by Humphreys & Associates on December 7, 2016 last modified May 6, 2021

Agile Scrum EVMS

P. Bolinger, CSM October 2016
Humphreys & Associates

How can Agile/ Scrum be used to support EVMS Variance Analysis and Forecasting in a way that provides program managers with cost and schedule integrated information for no extra effort?

The discipline of EVMS and the Agile/Scrum practices have several touch-points that are covered in two major documents: NDIA IPMD Agile Guide March 2016, and PARCA Agile and EVM PM Desk Guide. Neither of these documents, as yet, drives to the level of specifics when it comes to best practices for use of Agile to support EVM Variance Analysis and EVM Forecasting.

Looking at the literature for Agile/Scrum, we know that there are recommended ceremonies that are conducted at various levels of the product structure and at different times during the project life cycle. These ceremonies are supported by many discussions of the metrics that can be collected at each ceremony and their potential use in managing the technical work of development within the Agile/Scrum framework. But where do these ceremonies potentially support EVMS Variance Analysis and Forecasting?

Now suppose we are presented with a control account that has exceeded the EVMS thresholds for cumulative cost and schedule variances. Wouldn’t it be great to have at our fingertips the underlying data from the process? In this case we might find, for example, that the Velocity is less than that needed to meet the end goal, the story cycle time is longer than desired, the pass/fail ratio is not favorable, too many team members have been absent in the last Sprint, the number of disruptions has been excessive, and the work to accomplish the stories is higher than estimated. It is not difficult to surmise the outcome would likely be a behind schedule and overrun condition in the EVMS. These data measures would provide the fodder for deep diving to the root cause and impact statements.

Where would we get that information?

Let’s start with the Agile/Scrum ceremonies. The particular Agile/Scrum ceremonies that we find conducted during the project are:

Backlog Refinement. This ceremony can be nearly continuous. It involves redefining the backlog of development work (scope), the prioritization or re-prioritization of that work, and potentially the assignment of responsibilities for the backlog.

Release Planning. This is a recurring ceremony aligned with the release cadence for the project. It involves establishing the capabilities and features of the product and when they will be released.

Sprint. This is a short time-defined effort to accomplish the design, code, and test of some subset of the product. The Sprints are controlled by the self-managing teams.

Daily Scrum or Standup. This is a daily (recommended) team meeting to discuss what has happened, what roadblocks exist, what is planned for the day, and other necessary items.

Sprint Review. The session in which the team products are demonstrated to the owner and self-off is accomplished.

Sprint Retrospective. A meeting of the stakeholders to discuss what went right (or wrong) during the Sprint and to define improvement actions that are needed.

The relationship of these Agile ceremonies with EVMS might look like this:

Agile/Scrum Ceremony

Agile Purpose

Relationship to EVM Variance Analysis, Root Cause Analysis, Corrective Action Planning & Follow-up and Forecasting

 

 

 

Backlog Refinement Manage, estimate, prioritize, and organize the product backlog in an on-going routine. Estimating – impacts the EVMS ETC and EAC as well as durations of efforts.
    Prioritizing – in response to issues is corrective action management.
    Organizing the backlog could be a form of corrective action effort.
     
Release Planning Establishing the contents and timing for releases of product. Updates could be part of corrective action planning in response to issues. Creation of new work packages. Changes to planning packages and SLPPs would also.
     
Sprint Short time-box performance unit. Work is done in Sprints. Below the work package. Short span measurement period is possible.
     
Daily Scrum and Stand-up Make short term plan, adjust to issues, discuss problems, clear roadblocks. Much of the daily action would relate to root cause analysis and corrective action planning although the time-frame is very short and the issues may be too small to individually impact feature work package or the Epic control account.
     
Sprint Review Demonstrate the product, update released work, make changes to product. This would relate to corrective action planning and follow-up. Issues would be found here that would impact risks, ETCs, corrective actions, performance metrics.
     
Sprint Retrospective Reflect on the project, progress, people processes, what was good, what was bad, and take actions to improve. This should be the richest source of supporting information for EVMS root cause & corrective action within the VAR realm. Very timely for variance analysis as it happens potentially many times during work package (feature) duration.
Feature Retrospective (not one of the basic ceremonies) Review situation regarding technical scope deficit. Reflect on the project, progress, people processes, what was good, what was bad, and take actions to improve. Because this only happens at the end of the feature it is limited in value for variance analysis timeliness. Any lessons learned can only be applied to future feature work.

But where is the meat? Where do we get actionable data or at least data we can analyze to decide what management efforts are required?

There are numerous potential metrics that can be collected during these ceremonies. These metrics can form the basic data set that could be analyzed to define the root cause of cost and schedule variances. In addition to isolating the cause of issues, within some of these ceremonies the impact of the issue on the Sprint, or Feature, or team may be made. Certainly, these metrics can be used as the basis for projecting future workload and performance.

The total number of potential metrics is not known. In this paper, we looked at 17 metrics and considered what the data might mean. The results of this review are contained in this matrix:

Metric

Type of Measure

Discussion

Sprint Burn up/Burn Down Backlog Value of backlog remaining for Sprint. Decrease is expected when work is done. Increase means work increased. Burn up can include total completed plus remaining; a great metric.
Feature Burn Up/Burn Down Backlog Value of backlog remaining for Feature. Decrease is expected when work is done. Increase means work increased or shifted.
Customer support requests received. Disruption Number of instances. Unplanned interruptions by customer can lower the output of the team if excessive.
Disruption measures Disruption How many and what type (except customer support requests). Higher disruptions impact team efficiency.
Estimate Accuracy (Sprint or Feature) Estimating Measure of budgeted value (estimated value) for the Stories in the Sprint or Feature versus the actual cost (calculated cost) of the Stories when done. Related to team size.
Discovered work Estimating Emerging work discovered during the Sprint. Will translate to extra effort in future if adopted into backlog.
Exceeds WIP Limits Management If WIP limits are set on team or individuals then exceeding set limits will impact efficiency and output.
Retrospective Action Log Management Count of improvement actions listed in Retrospective. Increasing count means issues are not being resolved.
Attendance Management Comparison of actual hours worked by team compared to baseline expectations in plan.
WIP Productivity Measure of the number of stories or points in WIP at any time. WIP growth can indicate bottlenecks and inefficiencies.
Velocity Productivity Measure of the amount of work (Stories or Points) accomplished during a time period. Higher velocity means greater throughput per person/team.
Stability measures Productivity Comparing Sprint by Sprint basic measures from this list. If there is high variability between Sprints in measures, then future is unpredictable.
% Tests Automated Productivity Higher automated testing should increase efficiency & decrease cycle time.
Defects found by team Quality Number of bugs reported during team effort. Measures quality of work. Higher bug incidence translates to lower output and higher costs.
Defects found by customer Quality Number of bugs reported by user/customer. Measures quality of work delivered. Higher bug incidence translates to lower customer satisfaction and higher rework costs.
Pass/Fail (Re-do) measures Quality How does the rate of success in testing compare to the number of attempts? High success rate should mean greater output and efficiency.
Cycle Time Schedule Time from start of a story to complete. Short cycle time is desired.

Let us continue the theme of the behind schedule and overrun control account and look at what information would be available for support to developing the estimate-to-complete. An updated and refined backlog would have the scope of work remaining for the control account. The updated release plan would have the timing for the deliveries to be made in the control account. The metrics collected about the effort expended per accomplished story or story point would provide a factor for projecting future real-work hours. Planned corrective actions and improvements would tell us how we might expect improvement to the quality and improvement in the speed or cost of work. The insights available from a full set of metrics are impressive.

Does a project have to collect all of these metrics? If not all then which ones would be the right ones? Questions like that would be answered by the project management team analyzing their prior experience and the particular challenges of the project. The team would establish a data collection plan, likely described in their Software Development Plan or Program Management Plan that would explain the metrics, meaning, and frequency along with their purpose. With a clear understanding of the technical data to be collected and analyzed, the Control Account Managers would not have a difficult task to define how they would use that data in developing Variance Analyses and generating well-considered Forecasts. In fact, these tasks should be much simpler with the data in hand.

{ 1 comment… read it below or add one }

James Boehne December 30, 2022 at 5:44 am

I believe that the Scrum Master should also be the CAM within the Agile framework. The Scrum Master has the most information on the status of the work and therefore would be best able to take on this role. Separating the CAM from Scrum Master simply introduces a “middle-man”, and reduces the accuracy of the EVMS analysis and associated inputs. What is the H&S opinion on the matter?

Reply

Leave a Comment

{ 1 trackback }

Previous post:

Next post: