Using Performance Metrics to Manage

In the final installment on a performance management framework, we'll look at using performance metrics and analysis in order to effectively manage agencies and their programs by using remediation and corrective actions. We've already covered the first and second steps, and will focus on the final two in this posting (all four are listed just below):

1. Report performance metric data on pre-defined schedule.

2. Analyze data for troubling trends or missed targets. Operationally research root cause(s) of problems.

3. Provide corrective action for metrics where target was missed or data is trending in wrong direction.

4. Repeat process for next reporting period.

Assuming that reliable metrics have been gathered and reported, and that data trends have been analyzed, an agency should have a good idea about where it stands operationally. The question then becomes how best to use the new information. For example, if I'm an FEMS agency that knows my emergency response times are trending in the wrong direction, and I know that the problem lies somewhere in my call center, what's the next step? (I'll answer this in a minute.) Given the diversity of agency missions that exist within any government, it would be impossible to give specific guidelines on how to fix troubling trends. For the purpose of our framework, however, the important thing is that the information is used to formulate some plan of action, and that the plan of action is clear, has timelines, and is documented for future consideration. Maintaining documentation of attempted corrective actions can be particularly helpful when there are several options for remediation. Each option can be tried over a given reporting period and performance data can be tracked. If there is some improvement in the numbers, the corrective action was likely effective; if there is little or no improvement according to the data, then another option on the list may be your best bet. The important thing in documenting the remediation is not to spin your wheels by proposing the same corrective action repeatedly and expecting a different outcome with each successive attempt.

Going back to our emergency response example in which we assume that the call center has been identified as the source for deteriorating response times, there may be multiple options to improve performance, including additional training, process re-engineering, etc. There may not be an obvious "best" remedial option, but the important thing is to pick one and continue to track response times. If additional training was implemented but the trend is not reversed in response times, then lack of training can be eliminated as both the cause of the problem as well as a corrective action. Continue the cycle of capturing and reporting the metrics, but with a different corrective action this time. Perhaps the response process is streamlined or adjusted and overall times improve. We then have some indication that our proposed solution had a positive effect on the operations that we are tracking. Through trial and error in the corrective action process, while concurrently continuing to track and report data, any agency can improve effectiveness in its operations.

The important takeaway from this exercise is that in order to demonstrate marked improvement in any public sector operation or program, all steps in the framework that we've outlined here (and in past postings) must be followed. Tracking and reporting metrics without proposing and documenting remediation in trouble spots won't bring about the change in negative outcomes that most agencies are seeking. The feedback loop of track-report-remediate-repeat is the fundamental process behind our performance management framework, and is essential in solving government inefficiencies.

Dispatches from the Personal Democracy Forum

As part of a Google fellowship that I received on behalf of Public Performance Systems, I spent the last two days at the Personal Democracy Forum up in New York. I’ll finish up our discussion on a Performance Management Framework in the next few days but I wanted to first report on a number of fascinating highlights and initiatives from the conference which is meant to be a confluence of government, politics, and technology. Perhaps the most notable observation is that many people who were involved in using technology to bring Obama to the masses and to victory have transitioned into developing tools for governing, now that the election is over. It was a well attended conference and there’s clearly a lot of interest in this space.

Several initiatives were launched or revised at the conference on both the Federal level from the likes of Vivek Kundra as well as from Mayor Bloomberg in NYC. New York will launch their Big Apps competition in the Fall in order to encourage developers to come up with interesting utilities to assist the city in providing its vast data sets to the public. Additionally, Kundra announced revisions to the data.gov website and highlighted updates including usaspending.gov and the new dashboard at it.usaspending.gov. The latter is a slick application and looks nice, although I’ve always contended that there is no shortage of dashboarding software out there and the truly difficult part in presenting information is improving data quality and feedback. Nevertheless, these were all great initiatives.

Another interesting set of initiatives are occurring at blog.ostp.gov and mixedink.com/opengov which are both meant to foster discussion from citizens on policy ideas. While these sites are limited to IT policy discussions, the question was raised whether we’re moving in this direction for more general policy formulation (think policy formed through wiki by citizens). This may be a little farfetched, but it was well received by the technology community who are clearly excited about playing a role in government initiatives that impact the use of data in government transparency.

While it wasn’t discussed as much, government accountability was a major theme and I had discussions with several people about what exactly this means. Surprisingly few of them had much experience with performance management but after making my own case several agreed that some level of gathering metrics to assess performance would be useful. Many were more focused on taking various disparate public data feeds and turning them into something useful. My own interest still lies in assisting governments in building those data sets to begin with.

It was a great experience and I hope to attend again next year. For a replay of some of the activities, check out personaldemocracy.com.

Bright Side of Government

Microsoft is sponsoring a great blog with stories on successes in government. My company was featured yesterday and if you have a story to share feel free to add it.

Performance Management Analysis

Continuing our exercise in developing a performance management framework, a summary of the necessary steps for a successful program follows:

1. Report performance metric data on pre-defined schedule.

2. Analyze data for troubling trends or missed targets. Operationally research root cause of problems.

3. Provide corrective action for metrics where target was missed or data is trending in wrong direction.

4. Repeat process for next reporting period.

In previous posts I highlighted the importance of reporting performance metrics in a consistent and well-defined manner. In this post I'll cover the second part of a successful performance management program, analyzing the data from performance measures. A truly operational program must go beyond simple data reporting. Let's take response time from an Emergency Medical Services agency as an example. Is it enough to simply record and report response times? How do you know that what you're reporting is considered a "good" average response time? What percent should be below a specified time? Hard to tell without actually looking at the numbers. The data must be analyzed in the context of identifying troubling trends and researching the root cause of potential problems. All too often, data is reported to meet some external obligation and nobody even bothers to look at it! While complex data analysis is something of a science, there are many cases where any public sector employee can make good use of performance data with no training whatsoever. But to be most effective, it's necessary to record the findings of any current analysis for use in future situations. Raw data alone is useless to an organization, which makes a narrative of performance measures analysis (and eventually corrective action) a requirement for any successful performance management program.

First, don't focus on targets or absolute numbers when analyzing performance data. In most cases, analyzing trends over time is far more useful, and simple graphing exercises in a spreadsheet can highlight even modest improvements (or declines) in performance. This not only gives the layperson in government a powerful data tool, but also provides a baseline of data from which to operate. By focusing on trends, all agencies will feel as if they're operating from their current baseline, no matter how poor it may be. What's more important: that incident response times in Emergency Services are improving over time, or that they're hitting some arbitrary target? By focusing on graphing trends over time, most agencies will have both a powerful tool to evaluate their activities as well as a starting point for a performance management program without enduring criticism of missed targets. This isn't to say that looking at performance data against specific targets isn't useful, particularly when proposing new initiatives within an agency (particularly if it has a budgetary impact). Hard targets may be necessary to justify the initiative's or project's cost and can be used in declaring the initiative a success.

Second, all performance data should be analyzed within the context of whether significant change is due to real improvement in programs and outcomes, or if there is some other lurking variable. In my own experiences, improvements and challenges are often the result of poor data collection processes or errors, and not because of material changes in performance. Always rule out data anomalies first when a particular data point is out of the ordinary (many times by graphing the data these will be easy to spot). In our Emergency Services example, if we're looking at average response times, a few wild outliers could bring up the overall average significantly. What if the outliers were due to a faulty response report or some change in staffing that results in data not being reported properly? Government services are fairly stable over time and one would expect the data to represent this.

Finally, assuming that there is some difference in trends and that data issues are not the underlying reason, find out operationally exactly what the reason is. This is the most difficult part and often entails getting into the "weeds" of the organization. Operations analysts make a living out of improving processes in an organization, but managers with relevant knowledge within an organization should be able to just as easily get behind the numbers to understand what's happening. In these cases, several different metrics may be used to understand what's affecting trends. Going back to our response time example, let's assume we measure both the total number of EMS responses in a month in as well as average response time. If some disaster happened in a given month to cause responses to rise and we consistently measure this number, it may be a telling data point in explaining why average response times increased due to the increased stress on staff. Using multiple measures can help substantiate ideas for certain trends in the analysis phase.

These are just a few examples of good analysis techniques, but there are many ways to slay this dragon. I reiterate, however, the importance of recording any and all analysis that is done in order to create a running narrative to be used in future analysis of why data is trending the way it is. We'll get into the corrective action phase in the next post, but reporting a relevant set of performance metrics followed by analyzing the data are two huge steps towards an effective performance management system.

Performance Management Reporting

In my previous post I outlined a 4 step process for a successful performance management program. to recap:

1. Report performance metric data on pre-defined schedule.

2. Analyze data for troubling trends or missed targets. Operationally research root cause of problems.

3. Provide corrective action for metrics where target was missed or data is trending in wrong direction.

4. Repeat process for next reporting period.

I'll focus on the first step in this blog post as part of the overall attempt to develop a performance management lifecycle outline. Most of the set up work in a successful performance management program will be in this area. Each metric should have a well defined set of counting rules, methodology for collecting data, and a reporting period. Other attributes such as priority, stakeholders, etc may also be important, but the core is in the definition, methodology, and reporting period. Depending on the type of agency, there are a number of pre-defined definitions and counting rules (for an example, see this previous post) so no need to re-invent the wheel if those measures are agreeable. Data systems will vary by jurisdictions but the methodology will depend on their ability to generate performance data.

Once the difficult part of defining the metric and its data is complete, requiring managers to report their data on a regular time frame is essential to a successful program. The time frame should be regular and the metrics required should not change often. Getting managers to buy into the program will depend on the level of effort and predictability in each reporting period. If they are responsible for a large number of metrics then a less frequent reporting period is useful (annually or semi-annually). The trade off is slow feedback when metrics take a turn for the worse or when any new initiatives are launched. For less measures, more frequent reporting (monthly or quarterly) is helpful in root-cause analysis and less burdensome as well. The important thing to remember is that reporting performance data often takes time and resources and managers will grow resentful of heavy, frequent reporting requirements, particularly if the benefits of which are not apparent.

After data is reported it will often need some "scrubbing" for any errors prior to undergoing step #2 above, trend analysis. We'll look at the specifics of that in a future post, but the important takeaway here is to make performance reporting well-defined and as simple as possible for relevant managers. This will help ensure that the agency has the performance metrics necessary to make data-driven decisions.

A Public Sector Performance Management Methodology

Performance management and measurement have taken on a number of different meanings with regard to application in the public sector. In some cases it's regarded strictly as data reporting and in others it takes on a more qualitative form. It may be useful to start a dialogue on coming up with an actionable, consolidated set of objectives and practices to better define what is meant by government performance management. Future posts will break down how to achieve each point in the outline I present here as well as an attempt at a comprehensive methodology for designing a performance management system. The hope is that the system not only provides data, but also a practical management tool for government leaders. First, a list of the stakeholders and their interests in a performance management system:

1. Government Administrators - Information and tools to help manage the day-to-day operations of their jurisdiction as well as to inform policy formulation. The ability to communicate operational data to the public.

2. Public Interests - Actively engaged citizens interested in keeping track of the services that their government is providing.

3. Academic Interests - Research groups hoping to harness public data for academic studies used in policy formulation.

The question then becomes how to organize a program that will meet the interests of all three stakeholders? Part of the current difficulty in getting governments to report performance data has been that guidelines have largely been written by external groups for the sake of providing data to a third party (i.e. academics, research orgs, etc), with few clear, tangible benefits for the governments providing data. There is always the promise of benchmarked data, the ability to compare metrics across jurisdictions, etc., but the governments providing the data are interested in more immediate benefits. I propose a simple system that is standard practice in the private sector but only seems to have recently crept into the public sector:

1. Report performance metric data on pre-defined schedule.

2. Analyze data for troubling trends or missed targets. Operationally research root cause of problems.

3. Provide corrective action for metrics where target was missed or data is trending in wrong direction.

4. Repeat process for next reporting period.

And in its simplicity the above process will satisfy all three stakeholders. The government has a running narrative of operational data and the policies/actions it is undertaking for improvement. The public also has access to both the data on services that it needs as well as information on government policies. Assuming that the jurisdiction picked a standardized set of metrics, academic groups will have access to data for research purposes. Everybody wins!

The above is a simplification of a system that I will flesh out further in future posts, but the idea is to plant the seed of thought. I've looked at various websites and have yet to find this sort of methodology being advertised on government sites and it would be interesting to see it in practice (though I in no way take credit for this as an original idea. It's basic root-cause analysis. The hope is to find tools relevant to the public sector to implement said analysis). As always, I welcome feedback on this concept, and look forward to providing more detail.

The "Fear" of Performance Management

In a discussion with a colleague recently the topic of why more governments don't have an active performance management program came up. I will admit that the discussion was more speculative than scientific, but we generally agreed that many jurisdictions and agencies likely don't implement performance management programs out of fear of both what they might find as well as how the data that is reported might be used against them (the remainder of non-practitioners likely have no idea what it means!). There are stories of early meetings of CompStat in New York city in which supervisors were skewered based on crime stats in their area (although later accounts suggested a softening in the tone), and perhaps this is what government managers reference when thinking about reasons not to further performance management programs. But anyone focused on that aspect is ignoring the second part of that story which is the potentially positive impact of programs like CompStat. 

New York city has one of the lowest violent crime rates and the lowest property crime rate of large cities in the US. This is in stark contrast to the late 80s when it had one of the highest. The rates dropped precipitously throughout the 1990s, around the same time that CompStat came into being. While this relationship may very well be spurious, I imagine that the new management style in the police department had at least SOME effect on crime outcomes. I use CompStat as an isolated case of one agency's efforts, but it would be an interesting exercise to look at cities and states with advanced performance management programs (Atlanta, Albaquerque, and North Carolina spring to mind) and analyze the short and long term impacts of those programs both socially and politically. The reason this might be helpful is that it would be informative to cities who "fear" such programs in demonstrating the long term value of these programs. Certainly there are painful short term realizations of inefficiencies that would be made from better data and analysis but several case studies on improved outputs might put those fears to rest.