GAO compared SES members’ performance awards against their evaluation ratings in the 24 largest agencies. It found nearly all SES members rated better than average, a statistic improbability anywhere but Lake Wobegone ofA Prairie Home Companion fame. About 85 percent of SES members scored either a 5, the highest rating, or a 4, the second highest, for fiscal 2010-2013.
Supervisors feel pressure to rate executives highly because the ratings become criteria for myriad compensation decisions, GAO said.
But budget constraints and recent guidance from the Office of Management and Budget has kept the total amount of performance awards down and erased distinctions among SES. Executives with ratings of 5 earned $4,991 more than those with ratings of 4 in fiscal 2010. By 2013, the difference had shrunk to just $2,604. Between those in the middle of the pack, the difference is even smaller. Executives with ratings of 4 took home $690 more than those with ratings of 3.
As the difference lessens, the perception that awards are not directly linked to performance grows, GAO noted. Yet an executive at rating level 5 is much more likely to receive a performance award than one rated at 4 or below.
More at Federal News Radio.
From Cam Harvey, with applications to portfolio management:
We provide some new tools to evaluate trading strategies. When it is known that many strategies and combinations of strategies have been tried, we need to adjust our evaluation method for these multiple tests. Sharpe Ratios and other statistics will be overstated. Our methods are simple to implement and allow for the real-time evaluation of candidate trading strategies.
Recommended. This is a huge issue in classifying causal mechanisms in medicine, evidence-based policymaking and management, etc.
On one hand, any given “experiment” (read: regression result) may be faulty (sample-dependent); on the other hand, if we have multiple tests, we need good ways of aggregating those results.
If we have this problem with inference in the field of finance (with the amount of time, attention, and cash expended to figure out the best trading strategies), then what hope is there in evidence-based policymaking (where, in most cases, we have – at best – one experiment)?
From a discussion of the new dietary guidelines:
Instead of accepting that this evidence was inadequate to give sound advice, strong-willed scientists overstated the significance of their studies.
Much of the epidemiological data underpinning the government’s dietary advice comes from studies run by Harvard’s school of public health. In 2011, directors of the National Institute of Statistical Sciences analyzed many of Harvard’s most important findings and found that they could not be reproduced in clinical trials.