Performance Management Reporting

In my previous post I outlined a 4 step process for a successful performance management program. to recap:

1. Report performance metric data on pre-defined schedule.

2. Analyze data for troubling trends or missed targets. Operationally research root cause of problems.

3. Provide corrective action for metrics where target was missed or data is trending in wrong direction.

4. Repeat process for next reporting period.

I'll focus on the first step in this blog post as part of the overall attempt to develop a performance management lifecycle outline. Most of the set up work in a successful performance management program will be in this area. Each metric should have a well defined set of counting rules, methodology for collecting data, and a reporting period. Other attributes such as priority, stakeholders, etc may also be important, but the core is in the definition, methodology, and reporting period. Depending on the type of agency, there are a number of pre-defined definitions and counting rules (for an example, see this previous post) so no need to re-invent the wheel if those measures are agreeable. Data systems will vary by jurisdictions but the methodology will depend on their ability to generate performance data.

Once the difficult part of defining the metric and its data is complete, requiring managers to report their data on a regular time frame is essential to a successful program. The time frame should be regular and the metrics required should not change often. Getting managers to buy into the program will depend on the level of effort and predictability in each reporting period. If they are responsible for a large number of metrics then a less frequent reporting period is useful (annually or semi-annually). The trade off is slow feedback when metrics take a turn for the worse or when any new initiatives are launched. For less measures, more frequent reporting (monthly or quarterly) is helpful in root-cause analysis and less burdensome as well. The important thing to remember is that reporting performance data often takes time and resources and managers will grow resentful of heavy, frequent reporting requirements, particularly if the benefits of which are not apparent.

After data is reported it will often need some "scrubbing" for any errors prior to undergoing step #2 above, trend analysis. We'll look at the specifics of that in a future post, but the important takeaway here is to make performance reporting well-defined and as simple as possible for relevant managers. This will help ensure that the agency has the performance metrics necessary to make data-driven decisions.

A Public Sector Performance Management Methodology

Performance management and measurement have taken on a number of different meanings with regard to application in the public sector. In some cases it's regarded strictly as data reporting and in others it takes on a more qualitative form. It may be useful to start a dialogue on coming up with an actionable, consolidated set of objectives and practices to better define what is meant by government performance management. Future posts will break down how to achieve each point in the outline I present here as well as an attempt at a comprehensive methodology for designing a performance management system. The hope is that the system not only provides data, but also a practical management tool for government leaders. First, a list of the stakeholders and their interests in a performance management system:

1. Government Administrators - Information and tools to help manage the day-to-day operations of their jurisdiction as well as to inform policy formulation. The ability to communicate operational data to the public.

2. Public Interests - Actively engaged citizens interested in keeping track of the services that their government is providing.

3. Academic Interests - Research groups hoping to harness public data for academic studies used in policy formulation.

The question then becomes how to organize a program that will meet the interests of all three stakeholders? Part of the current difficulty in getting governments to report performance data has been that guidelines have largely been written by external groups for the sake of providing data to a third party (i.e. academics, research orgs, etc), with few clear, tangible benefits for the governments providing data. There is always the promise of benchmarked data, the ability to compare metrics across jurisdictions, etc., but the governments providing the data are interested in more immediate benefits. I propose a simple system that is standard practice in the private sector but only seems to have recently crept into the public sector:

1. Report performance metric data on pre-defined schedule.

2. Analyze data for troubling trends or missed targets. Operationally research root cause of problems.

3. Provide corrective action for metrics where target was missed or data is trending in wrong direction.

4. Repeat process for next reporting period.

And in its simplicity the above process will satisfy all three stakeholders. The government has a running narrative of operational data and the policies/actions it is undertaking for improvement. The public also has access to both the data on services that it needs as well as information on government policies. Assuming that the jurisdiction picked a standardized set of metrics, academic groups will have access to data for research purposes. Everybody wins!

The above is a simplification of a system that I will flesh out further in future posts, but the idea is to plant the seed of thought. I've looked at various websites and have yet to find this sort of methodology being advertised on government sites and it would be interesting to see it in practice (though I in no way take credit for this as an original idea. It's basic root-cause analysis. The hope is to find tools relevant to the public sector to implement said analysis). As always, I welcome feedback on this concept, and look forward to providing more detail.

The "Fear" of Performance Management

In a discussion with a colleague recently the topic of why more governments don't have an active performance management program came up. I will admit that the discussion was more speculative than scientific, but we generally agreed that many jurisdictions and agencies likely don't implement performance management programs out of fear of both what they might find as well as how the data that is reported might be used against them (the remainder of non-practitioners likely have no idea what it means!). There are stories of early meetings of CompStat in New York city in which supervisors were skewered based on crime stats in their area (although later accounts suggested a softening in the tone), and perhaps this is what government managers reference when thinking about reasons not to further performance management programs. But anyone focused on that aspect is ignoring the second part of that story which is the potentially positive impact of programs like CompStat. 

New York city has one of the lowest violent crime rates and the lowest property crime rate of large cities in the US. This is in stark contrast to the late 80s when it had one of the highest. The rates dropped precipitously throughout the 1990s, around the same time that CompStat came into being. While this relationship may very well be spurious, I imagine that the new management style in the police department had at least SOME effect on crime outcomes. I use CompStat as an isolated case of one agency's efforts, but it would be an interesting exercise to look at cities and states with advanced performance management programs (Atlanta, Albaquerque, and North Carolina spring to mind) and analyze the short and long term impacts of those programs both socially and politically. The reason this might be helpful is that it would be informative to cities who "fear" such programs in demonstrating the long term value of these programs. Certainly there are painful short term realizations of inefficiencies that would be made from better data and analysis but several case studies on improved outputs might put those fears to rest.

Citizen Awareness of Government Performance Measures

One aspect of the performance reporting cycle that I don't see discussed much is the role of the citizen in holding governments accountable for tax dollars. There are often waves of citizen anger over an isolated project or issue that is a lightning rod for criticism of government waste, but few groups seem to focus on the need for evaluating the performance of governments as a whole. A quick google search of "citizens for government accountability" did not seem to yield much in the way of citizens interested in government performance metrics (but did reveal a lot of anger about other issues!). There are some organizations that I have mentioned in previous posts with an interest in this area, but they are more academic than citizen-focused (I forgot to mention in previous posts the Performance Institute who offers a tremendous array of workshops and forums on government performance).

Ultimately I believe that the success or failure of government performance management in the public sector will be attributable to citizen engagement in the topic. Many people have grown used to easily finding out about crime statistics in a given neighborhood and would certainly notice if the resource was taken away. It will be interesting to see if that will translate to ensuring that their trash was collected, that equipment at the local park is full operational, or that the number of potholes in the streets are going down and not up. A notable paper by the Urban Institute addressed this point back in 2000 but it would be nice to see an update in this area. Most experts will admit that performance measurement has improved a great deal in the last 10 years, but the real question might be whether or not the average citizen cares, and if not, what will the impact on this area of government be in the future?

(as an aside, I'm simultaneously writing this post and watching a special on Federal attempts to clean up the Chesapeake and other waterways. It led me to the EPAStat quarterly report which unfortunately doesn't tell me if the fish or crabs are coming back to the estuary, but does tell me how many Chesapeake Bay Significant Discharge Permits were issued. With uninformative data like this there's no wonder it's hard to engage people in performance management.)

Will new Federal CIO, CTO Change Performance

There are several new faces in DC and it will be interesting to see what their impact on Federal performance management will be. The previous announcement of the Vivek Kundra as the nation's CIO was followed up with this week's announcement that Aneesh Chopra will be the nation's CTO and Jeffrey Zients will be the nation's Chief Performance Officer. This is a high-powered team of professionals that has worked in the city, state, and private sector and have been tasked to work together on bringing accountability back to the Federal government. My hope is that they will take a performance metric approach to accountability and ensure that agencies in the Federal government have a comprehensive performance plan which will be used to analyze effectiveness. I will post articles and comments on their progress moving forward.

Benchmarking Within a State

I mentioned in a previous post the attempts by ASCA to come up with benchmarking measures within the Corrections industry. There are several other groups attempting to benchmark data points within an industry (see HSRI Core Indicators for one example), but a couple of states have focused on a more intra-state approach to benchmarking across all government areas, but within the state. North Carolina and Florida have both embarked on projects to define metrics across jurisdictions in their states for the purpose of benchmarking against each other over time. It will be interesting to see if there is any consolidation among the varying groups attempting to come up with common performance metrics, be they state, industry, or professional organizations/commissions (such as the Performance Management Commission). While it may make sense to take from each group relevant measures, it may not be feasible to incorporate all of them for fear of drowning agencies in performance measurement reporting.

Comparison of Federal Performance Plans: HHS vs. DOJ

Browsing through several different Federal agency performance reports online there are some clear leaders in terms of quality of measures and others that have some challenges. In municipal performance reporting, police departments have often led the way in statistical performance data, largely because crime information is well-defined and one of the most monitored by citizens. Social service agencies typically lag behind their public safety counterparts with respect to municipal performance metrics. At the Federal level, it appears that the opposite is true.

The HHS Administration for Children and Families website has the agency performance reports going back to 2000. A look at the HHS ACF 2008 performance report shows some decent measures (many are outcome-oriented, a break from the output-focused measures of many agencies) that have been created by ACF as well as what looks like honest reporting of those measures. I assume honest reporting based on the fact that the agency has met less of its targets over the last several years. The measures appear well thought out and given the honest reporting of missed targets, the data appears to be reliable. Additionally, the reports get shorter each year since 2000. That's probably a good thing with respect to the public actually reading the document, and the agency focusing more intensely an a narrow set of measures and goals. If you can't explain a measure on a cocktail napkin, it's less likely to be reported and recorded accurately over time.

On the other side of the public spectrum is the DOJ Performance and Accountability Report for 2008. The report is a combination of both performance and financial data and comes just under the whopping-300 page mark. The parts that focus on performance metrics are lacking in a number of ways. Most are output-oriented and don't inform management or the public as to the effectiveness of the agency (One of the measures, "Terrorist Acts Committed by Foreign Nationals Against U.S. Interests", is zero in most years with 2001 the notable exception and would almost certainly be known without needing to be included in the performance report. The measure is neither informative nor a helpful management tool). While the municipal law enforcement agencies have made great strides in performance reporting, the Federal level agencies seem to just be getting their feet under them. One reason may be because of the difficulty in attributing crime and arrest rates to a Federal agency whose jurisdiction is the entire country. Most responsibilities of the agency are shared with state and local jurisdictions, yet the DOJ has little or no control over those agencies. Regardless, there are almost certainly some performance measures that the DOJ could come up with that are more worthwhile. If not, how can their effectiveness truly be measured? A good start would be to narrow the report into something more digestible that might actually be used in agency management.