On February 2001, seventeen people convened at the Snowbird Ski Resort in Utah to find common ground and develop an alternative approach to document-driven, heavyweight software development processes. The group included representatives from Extreme Programming, SCRUM, DSDM, Adaptive Software Development, Crystal, Feature-Driven Development and Programmatic Programming. They named themselves the “Agile Alliance” and created and signed the Agile Manifesto which became the cornerstone for the Agile movement.
Manifesto for Agile Software Development
We are uncovering better ways of developing software by doing it and helping others do it.
Through this work we have come to value:
Individuals and Interactions over Processes and Tools Working Software over Comprehensive Documentation Customer Collaboration over Contract Negotiation Responding to Change over Following a Plan
That is, while there is value in the items on the right, we value the items on the left more.
Agile Streamlines Documentation
Historically, enormous amounts of time were invested in documenting the product for development and delivery. The list was extensive and included technical specifications, requirements, prospectus, interface design documents, test plans, documentation plans, and untimely approvals, resulting in bottlenecks and long cycle times. Agile does not eliminate documentation, rather, it streamlines documentation by providing the developer with just the essentials that are needed to do the work.
People Over Process
Valuing people more highly than processes or tools is easy to understand because it is people who respond to business needs and drive the development process. If the process or the tools drive development the team is less responsive to change and less likely to meet customer needs.
Negotiation vs. Collaboration
Negotiation is the period when the customer and the product manager work out the details of a delivery with points along the way where the details may be renegotiated. Collaboration is a different creature entirely, with development models such as waterfall, customers negotiate the requirements for the product often in great detail prior to any work starting.
This meant the customer was involved in the process of development before development began and after it was completed but not during the process. The agile manifesto describes a customer who is engaged and collaborates throughout the development process, making it much easier for developers to meet the needs of the customer.
Welcoming Change
Traditional software development regarded change as an expense and avoided it whenever possible. The intention was to develop detailed elaborate plans with a defined set of features and dependencies so that the team had a clear understanding of what should be developed and when it should be worked.
Agile welcomes changing requirements, even late in the development and uses rolling wave planning to decompose work just-in-time, without going into elaborate decomposition of the high-level plan.
12 Principles
The Agile Alliance also created 12 principles that support and expand on the four agile manifesto values. These principles have resulted in a number of widely accepted best practices that are used in many agile approaches such as extreme programming, scrum, DSDM, Adaptive Software Development, Crystal, and Feature-Driven Development.
The Agile Manifesto has spread like wildfire since 2001, forever altering the way we develop custom solutions and making impacts far beyond its original software origin.
P. Bolinger, CSM October 2016 Humphreys & Associates
How can Agile/ Scrum be used to support EVMS Variance Analysis and Forecasting in a way that provides program managers with cost and schedule integrated information for no extra effort?
The discipline of EVMS and the Agile/Scrum practices have several touch-points that are covered in two major documents: NDIAIPMD Agile GuideMarch 2016, and PARCA Agile and EVM PM Desk Guide. Neither of these documents, as yet, drives to the level of specifics when it comes to best practices for use of Agile to support EVM Variance Analysis and EVM Forecasting.
Looking at the literature for Agile/Scrum, we know that there are recommended ceremonies that are conducted at various levels of the product structure and at different times during the project life cycle. These ceremonies are supported by many discussions of the metrics that can be collected at each ceremony and their potential use in managing the technical work of development within the Agile/Scrum framework. But where do these ceremonies potentially support EVMS Variance Analysis and Forecasting?
Now suppose we are presented with a control account that has exceeded the EVMS thresholds for cumulative cost and schedule variances. Wouldn’t it be great to have at our fingertips the underlying data from the process? In this case we might find, for example, that the Velocity is less than that needed to meet the end goal, the story cycle time is longer than desired, the pass/fail ratio is not favorable, too many team members have been absent in the last Sprint, the number of disruptions has been excessive, and the work to accomplish the stories is higher than estimated. It is not difficult to surmise the outcome would likely be a behind schedule and overrun condition in the EVMS. These data measures would provide the fodder for deep diving to the root cause and impact statements.
Where would we get that information?
Let’s start with the Agile/Scrum ceremonies. The particular Agile/Scrum ceremonies that we find conducted during the project are:
Backlog Refinement. This ceremony can be nearly continuous. It involves redefining the backlog of development work (scope), the prioritization or re-prioritization of that work, and potentially the assignment of responsibilities for the backlog.
Release Planning. This is a recurring ceremony aligned with the release cadence for the project. It involves establishing the capabilities and features of the product and when they will be released.
Sprint. This is a short time-defined effort to accomplish the design, code, and test of some subset of the product. The Sprints are controlled by the self-managing teams.
Daily Scrum or Standup. This is a daily (recommended) team meeting to discuss what has happened, what roadblocks exist, what is planned for the day, and other necessary items.
Sprint Review. The session in which the team products are demonstrated to the owner and self-off is accomplished.
Sprint Retrospective. A meeting of the stakeholders to discuss what went right (or wrong) during the Sprint and to define improvement actions that are needed.
The relationship of these Agile ceremonies with EVMS might look like this:
Relationship to EVM Variance Analysis, Root Cause Analysis, Corrective Action Planning & Follow-up and Forecasting
Backlog Refinement
Manage, estimate, prioritize, and organize the product backlog in an on-going routine.
Estimating – impacts the EVMS ETC and EAC as well as durations of efforts.
Prioritizing – in response to issues is corrective action management.
Organizing the backlog could be a form of corrective action effort.
Release Planning
Establishing the contents and timing for releases of product.
Updates could be part of corrective action planning in response to issues. Creation of new work packages. Changes to planning packages and SLPPs would also.
Sprint
Short time-box performance unit. Work is done in Sprints.
Below the work package. Short span measurement period is possible.
Daily Scrum and Stand-up
Make short term plan, adjust to issues, discuss problems, clear roadblocks.
Much of the daily action would relate to root cause analysis and corrective action planning although the time-frame is very short and the issues may be too small to individually impact feature work package or the Epic control account.
Sprint Review
Demonstrate the product, update released work, make changes to product.
This would relate to corrective action planning and follow-up. Issues would be found here that would impact risks, ETCs, corrective actions, performance metrics.
Sprint Retrospective
Reflect on the project, progress, people processes, what was good, what was bad, and take actions to improve.
This should be the richest source of supporting information for EVMS root cause & corrective action within the VAR realm. Very timely for variance analysis as it happens potentially many times during work package (feature) duration.
Feature Retrospective (not one of the basic ceremonies)
Review situation regarding technical scope deficit. Reflect on the project, progress, people processes, what was good, what was bad, and take actions to improve.
Because this only happens at the end of the feature it is limited in value for variance analysis timeliness. Any lessons learned can only be applied to future feature work.
But where is the meat? Where do we get actionable data or at least data we can analyze to decide what management efforts are required?
There are numerous potential metrics that can be collected during these ceremonies. These metrics can form the basic data set that could be analyzed to define the root cause of cost and schedule variances. In addition to isolating the cause of issues, within some of these ceremonies the impact of the issue on the Sprint, or Feature, or team may be made. Certainly, these metrics can be used as the basis for projecting future workload and performance.
The total number of potential metrics is not known. In this paper, we looked at 17 metrics and considered what the data might mean. The results of this review are contained in this matrix:
Value of backlog remaining for Sprint. Decrease is expected when work is done. Increase means work increased. Burn up can include total completed plus remaining; a great metric.
Feature Burn Up/Burn Down
Backlog
Value of backlog remaining for Feature. Decrease is expected when work is done. Increase means work increased or shifted.
Customer support requests received.
Disruption
Number of instances. Unplanned interruptions by customer can lower the output of the team if excessive.
Disruption measures
Disruption
How many and what type (except customer support requests). Higher disruptions impact team efficiency.
Estimate Accuracy (Sprint or Feature)
Estimating
Measure of budgeted value (estimated value) for the Stories in the Sprint or Feature versus the actual cost (calculated cost) of the Stories when done. Related to team size.
Discovered work
Estimating
Emerging work discovered during the Sprint. Will translate to extra effort in future if adopted into backlog.
Exceeds WIP Limits
Management
If WIP limits are set on team or individuals then exceeding set limits will impact efficiency and output.
Retrospective Action Log
Management
Count of improvement actions listed in Retrospective. Increasing count means issues are not being resolved.
Attendance
Management
Comparison of actual hours worked by team compared to baseline expectations in plan.
WIP
Productivity
Measure of the number of stories or points in WIP at any time. WIP growth can indicate bottlenecks and inefficiencies.
Velocity
Productivity
Measure of the amount of work (Stories or Points) accomplished during a time period. Higher velocity means greater throughput per person/team.
Stability measures
Productivity
Comparing Sprint by Sprint basic measures from this list. If there is high variability between Sprints in measures, then future is unpredictable.
% Tests Automated
Productivity
Higher automated testing should increase efficiency & decrease cycle time.
Defects found by team
Quality
Number of bugs reported during team effort. Measures quality of work. Higher bug incidence translates to lower output and higher costs.
Defects found by customer
Quality
Number of bugs reported by user/customer. Measures quality of work delivered. Higher bug incidence translates to lower customer satisfaction and higher rework costs.
Pass/Fail (Re-do) measures
Quality
How does the rate of success in testing compare to the number of attempts? High success rate should mean greater output and efficiency.
Cycle Time
Schedule
Time from start of a story to complete. Short cycle time is desired.
Let us continue the theme of the behind schedule and overrun control account and look at what information would be available for support to developing the estimate-to-complete. An updated and refined backlog would have the scope of work remaining for the control account. The updated release plan would have the timing for the deliveries to be made in the control account. The metrics collected about the effort expended per accomplished story or story point would provide a factor for projecting future real-work hours. Planned corrective actions and improvements would tell us how we might expect improvement to the quality and improvement in the speed or cost of work. The insights available from a full set of metrics are impressive.
Does a project have to collect all of these metrics? If not all then which ones would be the right ones? Questions like that would be answered by the project management team analyzing their prior experience and the particular challenges of the project. The team would establish a data collection plan, likely described in their Software Development Plan or Program Management Plan that would explain the metrics, meaning, and frequency along with their purpose. With a clear understanding of the technical data to be collected and analyzed, the Control Account Managers would not have a difficult task to define how they would use that data in developing Variance Analyses and generating well-considered Forecasts. In fact, these tasks should be much simpler with the data in hand.
This article provides an introduction to the differences between an Earned Value Management System (EVMS) and Agile approaches on projects, and isolates the challenges of implementing an EVMS on a project where the project management has chosen to follow an Agile approach to the development work. The article explores the principles of EVMS and of Agile, and contrasts them to show where there are inherent conflicts. The article then discusses how the conflicts can be mitigated so that the benefits of both the EVMS and Agile can be obtained from a joint implementation.
Earned Value Management is a 50 +/- year old methodology based on widely accepted principles that applies documented, systematized practices to support the processes of organizing, planning, directing, and controlling large complex projects, of any nature, which contain a high degree of uncertainty.
An EVMS is structured compliant to 32 guidelines that define what a project management information system should be capable of doing to support the program management team. Within the 32 guidelines there is a subset of generally recognized core principles. The core principles are:
Organize the entire scope of the project using a Work Breakdown Structure (WBS).
Organize the project team using an Organization Breakdown Structure (OBS).
Integrate the project work with the project team to create management control points (Control Accounts).
Schedule the project work in the Control Accounts across the entire project duration at the appropriate level of detail.
Establish time-phased budgets for the scheduled work in the Control Accounts.
Establish the scope/schedule/budget baseline as the Performance Measurement Baseline (PMB).
Authorize the scope/schedule/budget and control the start/stop of work.
Periodically measure the schedule and the value of completed work and determine the Earned Value.
Record direct costs (actual costs) and summarize into the Control Accounts.
Compare planned, accomplished, and spent to analyze the performance and associated variances.
Develop realistic time and cost estimates for the remaining effort in the Control Accounts.
Rigorously control changes to the Performance Measurement Baseline.
The EVM concept presented in these guidelines is a sound management approach, that once incorporated on any type of program, whether research and development, construction, production, etc. provides all levels of management with early visibility into cost and schedule problems. Earned Value Management now appears as a contractual requirement on programs world-wide. Primary EVM users include the United States, Europe, England, Canada, Australia, China, and Japan. It is a requirement of many U.S. Government agencies, including the Department of Defense (DoD), the National Aeronautics and Space Administration (NASA), the Department of Energy (DOE), the Intelligence Community, the Department of Homeland Security (DHS), the Federal Aviation Administration (FAA), the Department of Transportation (DOT), Health and Human Services (HHS), and others.
EVMS has been adopted by companies in situations where it is not a contractual requirement so that they can gain the discipline and benefits of the structural management approach discussed in the guidelines.
Flexible Planning with Agile
Agile Background
Agile is a 20 +/- year old approach to applying a mindset that values the use of small, empowered, self-organizing, multi-functional teams, mainly in software development, to establish a test driven product development effort. This uses a series of short, rapid incremental builds within projects with a high degree of uncertainty to achieve shorter development times, lower costs, and products more closely aligned with customer requirements.
Agile does not usually appear as a contractual requirement. The concept is adopted by companies and organizations with the belief that a better product will be produced faster, and with less expense, using this approach than if traditional approaches were followed.
There are a core set of principles for Agile that were initially established in the copyrighted Agile Manifesto. These delineated core principles are:
Early and continuous product delivery.
Deliver working software (product) frequently.
Expect change and respond positively to change.
Developers and project business organization (PMO) work together.
Product-focused build teams are at the core.
Support self-organizing teams – trust among peers.
Encourage face-to-face discussions (involve the user/customer).
Working software is the measure of progress.
Maintain a constant sustainable pace of development.
Simplify the process and the product.
Put the highest value on technical excellence.
Improve team effectivity.
So How Does Agile Work?
The Agile mindset or approach is implemented through a process of defining the product backlog into smaller and smaller subsets of work that are structured in a top down fashion. At the lowest level of product backlog, the work elements or requirements can be prioritized and assigned to teams. The self-organizing teams pull work from the backlog and work the tasks to completion in a series of short, time-fixed Sprints or iterations. Sprints are often from 2 to 4 weeks long.
Because the teams are self-organizing, there is no team manager or team lead. The teams work as a group and only pull from the backlog at the last possible minute, and do minimal planning for each Sprint. If the product backlog is properly deconstructed and defined into user stories, then the planning meeting for an entire 4 week Sprint can be accomplished in a few hours.
The teams design, code, test, integrate, and deliver functionality in every Sprint. Since tested product is output every few weeks, all on the project can see the product being created and can contribute along the timeline, as needed, to provide a complete finished product.
The product owner embodies the customer’s perspective and either accepts or rejects the team’s work. At the latter point of each Sprint, the tested product is demonstrated to the product owner.
High Level Side-by-Side
The two approaches are contrasted in the chart shown below. The EVMS is a methodology that is highly documented and highly systematized, while Agile is just the opposite. It is more of a mindset than a methodology with the preference not to have significant process documentation.
The EVMS is applied to entire projects and contracts, while generally Agile is applied to software portions of projects. However, it could also be used on other development work.
The EVMS is usually a contractual requirement with significant implementation and operation constraints while Agile has none of these. As a contractual requirement, the EVMS carries with it the option for customer reviews and the threat of non-compliance, which entails penalties.
EVM
Agile
Methodology
Mindset
Documented
Self-defined
Systematized
Self-defined
Any type of project
Software development (mainly)
High degree of uncertainty
High degree of uncertainty
Applied to the entire project
Applied to portions of the project
Often a contractual requirement
Adopted not required
Lower Level Side-by-Side
In addition to the high level side-by-side, there are significant differences within the details of the two approaches. These are shown in the side-by-side table below.
Agile
EVMS
Minimal documentation
More documentation
Plan at last moment
Plan ahead to end of project
Scope is flexible
Scope is baselined and controlled
Expect and embrace change
Avoid and/or control change
Schedule (Sprint) is fixed. Timebox ends the Sprint.
End the package when the work is done.
Budget is secondary
Budget is baselined and controlled
Cost collection is not mentioned.
Cost collection at the right level is critical
Accommodate and Capitalize on Differences
It is possible to implement Agile along with an EVMS if the EVMS application is set up to accommodate the differences and capitalize on them.
For example, the main reason that Agile’s embrace of change is a potential problem within an EVMS is because often the EVMS is used to plan too far in advance, and then reacting to change is difficult and expensive. If short term planning in the EVMS can be coordinated with the Agile planning, then the two can coexist.
The Agile free acceptance of scope changes within the backlog runs counter to the EVMS imposition of baseline change control. But if the EVMS baseline can be carefully set at a work level above the busy lowest level ups and downs, the impact to baseline change control is manageable.
A surprising chance to capitalize on Agile, within the EVMS, is found in the Scrum team approach in Agile where the team breaks work down into tasks far below what would normally be done in an EVMS, and then meets daily to update progress and provide corrective action. This low level constant attention means that the EVMS benefits from a better look at real progress as assessed by the real performers.
Not all the compromise needs be on the EVMS side of the equation. The Scrum team operations will often be defined to have the least possible recording of what happens during the Sprint. Since the product is king, then only the product really matters. But that misses the opportunities to capture the actions of the team for analysis, and use in upgrading their skills later. Necessary compromises would include some additional recording of the daily actions of the team and capturing the progress and problems. These would then be used in the EVMS functions of performance measurement, variance analysis, and corrective action planning.
One other compromise in the Agile realm that would be needed is the adoption of some minimum documentation of processes so that team operations can be repeatable and stable. Even a self-organizing team cannot change the way they work every time it wishes. That would raise the risk of a chaotic work environment.
These topics are recapped in the table below.
Agile
EVMS Accommodation
Scope is flexible
Select a higher level package for the baseline
Change is expected and embraced
Have the shortest possible planning horizon
Plan at the last possible minute
Have the shortest possible planning horizon
Daily Scrum Stand-up Meeting
Collect the data and use it for performance measurement
Sprint Review Meeting
Use for periodic measurement and analysis
Sprint Retrospective Meeting
Use in Corrective Action Plans
Lack of documentation
Add minimum documentation to stabilize team operations
Bottom Line
Implementing an EVMS is a challenge itself. Implementing Agile is a challenge too; perhaps a more difficult challenge. Implementing the two approaches side-by-side can seem impossible. But it is possible and even beneficial if done in a way the makes needed accommodations in both arenas for the project’s benefit.
