Rating the agencies
Every other year the Office of Personnel Management conducts a mammoth survey of the federal workforce. From the responses of more than 200,000 federal employees, the OPM calculates agency-by-agency scores on four dimensions: leadership (which indicates how favorably employees of an agency regard their leaders), performance (which indicates the extent to which employees believe their agency promotes improvements in processes, products, and services), talent (which indicates the degree to which employees think their agency has the talent needed to achieve its goals), and job satisfaction (which indicates how satisfied employees are with their jobs). In the 2008 survey, the results of which were released a couple of months ago, 37 agencies were rated.
The official release of the results listed only the agencies that placed in the top ten of each of the four categories, apparently to avoid embarrassing the bottom-dwelling agencies. Joe Davidson of the Washington Post subsequently obtained the complete lists, which included all 37 agencies, and identified the bottom ten on each dimension as well as the top ten in a piece in the Post. He also put the full listings — ratings as well as rankings — for all 37 agencies available online.
Working from the online data, I determined that the four dimensions are easily reducible to one overall dimension; the correlations between agencies’scores on any pair of dimensions are all .88 or above. In plain English, the four sets of scores, despite some idiosyncrasies, all boil down to the same thing, which we can think of as an agency’s overall performance in management and administration.
Combining the four sets of scores into a single standardized score produces the following:
The best-run federal agencies, according to this measure, are the Nuclear Regulatory Commission, the National Science Foundation, the Office of Management and Budget, and the National Aeronautics and Space Administration. Three cabinet departments — HUD, Homeland Security, and Transportation — are bottom-of-the-listers. The worst-run agency by far, though, appears to be the Broadcasting Board of Governors, which oversees nonmilitary international broadcasting by the government. It used to be part of USIA, but it became independent in 1999, and, to judge by the assessments of those who work there, seems to be something of a disaster. What is it about the Broadcasting Board of Governors that’s soo bad? Basically everything, according to the OMB survey: It ranks dead-last on three of the four dimensions iand 36th of 37 on the other dimension. (I don’t know much about the Broadcasting Board of Governors and haven’t been able to dig out much, except that the former chair, Kenneth Tomlinson, was accused of, but not prosecuted for, improprieties of various sorts both there and in his previous stint as chair of the Corporation for Public Broadcasting.)
Comments
So…if one institution is at +1 and another at 0 and a third at -1, what does it really say? Is this about the distance or the ranking or something else?
Posted by: Doug Hess | March 31, 2009 03:52 PM
Doug: It’s about the distance, expressed in standard deviations. If one agency is at 0, that means that across the four dimensions that have been melded into one dimension, it’s score falls exactly at the mean. An agency with a score of +1.0 would score, on the cumulative scale, one standard deviation higher; one with a -1, one sandard deviation lower.
Posted by: Lee Sigelman | March 31, 2009 04:11 PM
I know that, but what does it mean to be that far apart? For instance, if you know your IQ is two st. deviations below 100, you know that it’s not just rare but well…not good for your chances in some arena’s in life. But if your workplace is a deviation or two away on this measure: does it mean that people are just unhappy worker bees, or is anything meaningful actually related to it?
Posted by: Doug | March 31, 2009 09:45 PM
Ah. Of course that’s a substantive question, not a statistical one. The same question can be asked about any survey-based measure. How different is, say, “strongly agree” from “agree,” or “excellent” from “good”?
I don’t know whether scores on these dimensions have been validated against any behavioral measures. In any event, I surely wouldn’t attach much importance to small differences on these scores. But when an agency stands out as far as the one does, it strongly suggests that something unusual is amiss there.
Posted by: Lee Sigelman | March 31, 2009 10:43 PM
If a student says I’m a “good” teacher, they’re smart; if they say I’m an “excellent” teacher, they’re a genius. It’s all very simple.
Posted by: Doug | April 1, 2009 10:18 AM
I’m following a link back from Matt Yglesias. How can you say a agency is “well run” if the only things you measure are how happy the employees are? There is no scoring based on the agency accomplishing anything
Posted by: Campesino | April 1, 2009 11:30 AM
Campesino:
I don’t think it’s accurate to say that the ratings convey just “how happy” the employees are. Certainly employee morale factors in, but the employees’ perceptions of agency performance on the various dimensions go well beyond their individual level of happiness.
Your point is that whether an agency is effective in terms of accomplishing its established goals isn’t reflected in these ratings. Of course that’s true, but that’s not the point of the ratings. Could, say, the Department of State be “well run” even if it weren’t successful in, say, maintaining amicable relations with other nations? Could, say, the Deaprtment of Education be “well run” even if the American educational system were a mess? These ratings are attempts to get at administrative efficiency and performance, period, via employees’ perceptions of how well their agency is managed. Broader questions about agency effectiveness in terms of goal accomplishment aren’t part of these ratings.
I’d love to see a measure comparing agencies on a scale of task accomplishment, though I can’t even begin to imagine how that would be done.
Posted by: Lee Sigelman | April 1, 2009 11:55 AM
There’s a huge, and often rather dry, body of literature on measuring agency performance. I fact it’s all the rage.
One the web, start with: http://www.whitehouse.gov/omb/expectmore/index.html
For academic readings: Start with the collection of essays in “Quicker, Better, Cheaper” edited by…(I forget who), or “Report Cards” by Gormley and Weimer (which is just about organizationl report cards, not other methods), or “Challenging the Performance Movement” by Beryl Radin for a critcism of it all that is highly readable.
Posted by: Doug Hess | April 1, 2009 07:58 PM