Saturday, September 24, 2011

Testing Times At The Office

POOR Hawthorn. The Hawks will not be playing in next weekend’s Grand Final, having failed last night by a miserable three points to get the better of Collingwood. There will be post-mortems, of course, and the most obvious explanation will be that Jeff Kennett’s leg-weary boys ran low on ergs in the fourth quarter. Few who watched the game, a genuine classic, will disagree with that assessment, but readers of this morning’s Silly will know better: If only the MCG had been officially designated a five-star sustainable sports stadium, Hawthorn’s final-term fitness and work rate would have soared, making vistory inevitable.

This is science we are talking about here, folks -- good, hard, solid, settled science, the sort the Silly and Phage never hesitate to report when field surveys and results, even the most dubious ones, provide just the merest hint of a suggestion they might endorse those newspapers’ jolly green preconceptions. In this instance it is the benefits of working in sustainable office environments, with the Silly reporting the latest news from the Clipboard Kingdom in the story’s first paragraph “The new wave of ultra-green office buildings,” it begins, “is bringing unexpected benefits for their occupants beyond the obvious savings in cost and energy.”*

Overlook the sentence’s cack-handed construction and grammatical imprecision, which are no more nor less than one expects these days from quality journalists and those who teach them.* Instead, examine the, ahem, scientific method the story asserts has established the benefits. After quoting two twenty-something lawyers in Sydney, writer Nick Galvin switches his attention to the south, where he reports the following:
The pair's sentiments are backed by an increasing body of evidence, including a detailed study with another law firm, Oakley Thompson, after it recently moved into a newly refurbished green building in the centre of Melbourne [500 Collins Street]. The study, conducted with the University of Melbourne, found sick days had fallen 39 per cent and the lawyers' billings ratio rose 7 per cent despite the overall hours worked falling 12 per cent. The survey even found the firm's secretaries were typing 9 per cent faster in the new building and with greater accuracy.
Detailed study, eh? Well, not quite. As anyone who examines the actual results will ascertain very quickly indeed, the methods are about as reliable as Hawthorn’s set shots.  Let us take those claims one by one:

SICK DAYS: The study contrasts five years of pre-renovation sickie stats with those for just nine months after the office was awarded its five green stars. The five-year figures were “normalised”, just like tree-ring data, and the post-renovation numbers were not, with the number crunchers asserting they saw no need to do so.

Even so, incidences of illness in post-renovation January exceeded those for the same month’s five-year average. And where there were differences, they were generally slight. March numbers were only a whisker below the five-year norm, as were those for the previous November. Such overall gains as the study claimed to observe are attributable almost solely to three months of somewhat wider divergence (see Chart 3).But given that the post-renovation numbers are based on part of just a single year, it is reasonable to assume that some of the previous years’ pre-renovation figures were just as low.

Logic and surmise are our only guides on this point as, for reasons unexplained, the authors neglect to provide each of those years’ individual trend lines. 

BILLINGS: The claim is that lawyers brought in 7% more fees while working 12% fewer hours. Let the report itself, which Galvin evidently neglected to absorb, explain why that summarised result is pure nonsense.

First, there is the differing natures of the compared samples (underscore added at the Billabong): 
QUOTE #1: The data on which these conclusions were made covered only ten months before the office move, and ten months after the move. Before this period, data was collected manually and the records were not readily comparable. The differences in data collection approaches would also make the comparison questionable.
 Then there is the matter of the survey’s participants:
QUOTE #2:the pre-occupancy survey was conducted four months before the move, while the post-occupancy survey was three months after the move … It was a voluntary survey and of the total of 20 employees (including partners and solicitors), 12 (60%) agreed to participate. For the survey to obtain meaningful comparisons, the post-occupancy survey should cover the same participants as in the pre-occupancy survey. However, only eight of the 12 original participants responded to the second survey, so that the comparison covers only 40% of the total staff.

But not to worry. Another firm, stockbrokers this time, also was surveyed, with an unnamed senior executive quoted as saying “productivity has gone through the roof.” Alas, there is no way to verify that enthusiasm. As the survey’s authors concede (they do an awful lot of concedin’):
QUOTE #3: … Because Lonsec [the brokerage] only agreed to join the Study after the move, the survey could only be retrospective, with participants directly comparing their new accommodation with their memory of their past accommodation. This is less objective than asking staff to rate a building before a move and then, some time later asking them to rate the building again.
FLYING FINGERS: And finally there is the astonishing claim, the most improbable of them all, that typists are faster and more accurate if tapping away in a green building. As with those busy, busy lawyers, the before-and-after sample was miniscule. An improvement of 9% is claimed, but go to Chart 5 and you will be struck that only five secretaries were tested. Of these, according to Chart 6, one – Typist 4 – was more accurate in her previous surroundings!

There is plenty more in the findings that should have raised reporter Galvin's eyebrows, and it is to be hoped that one of those savvy statistical sorts who make Catallaxy Files so interesting will bring an expertise greater than that of a specialist in Etruscan semiotics to the subject. But even without a more probing analysis, two things stand out as causes for alarm.

The first is that, once again, the Silly has demonstrated its eagerness to swallow and regurgitate more of the green guff its reporters and editors find so congenial, accuracy and insight be damned.

The second is far more alarming and can be found in the survey’s URL, which announces with its “vic.gov.au” that it is now an official document and thus certain, absolutely certan, to be cited when justification is sought for more green handouts or, inevitably, punitive taxes and levies on those employers whose offices are, as yet, uncarpeted with green cant.

If only the Hawks had known what wonders motivated statisticians can work, they would have sent last night’s final score over to Melbourne University and had themselves declared normalised winners. Making stuff up, that's the new normal.


UPDATE: What a pity the survey did not include Ned from the US version of The Office this activist version of The Office in the sample of happy employees. Ignore the preaching at the end, and thanks to Dan Lewis for pointing out the error.




* Buildings do not come in waves. Benefits, unexpected or otherwise, are brought to, or bestowed upon, recipients. Why are the savings in cost and energy obvious? Where is the comma after occupants?

11 comments:

  1. There's another shortcoming in this "study"—comparing people moving from a green building to a non-green building.

    In college, I was taught about a case where a firm decided to try and improve productivity by making changes in the office environment. I forget most of the exact changes, but the first and last ones I mention below are correct.

    First, they improved the lighting. Productivity went up. Then they installed a coffee machine. Productivity went up. Then they introduced flexitime. Productivity went up. And on it went. Then they restored the lighting back to what it had originally been. And productivity went..... UP.

    The lesson wasn't that certain improvements bring productivity increases, but that productivity improvements were due to management being seen to be interested in making changes for the benefit of the staff.

    So here we have a study of staff moving to new buildings. Is the fact that they are "green" relevant, or is the fact that they have moved into new surroundings more important?

    ReplyDelete
  2. "But not worry. Another firm, stockbrokers this time, also also..."

    In my experience, the odds of grammatical errors creeping into one's own writing are greatly increased the moment you comment on someone else's.

    ReplyDelete
  3. That video wasn't from The Office. That's why it wasn't funny.

    ReplyDelete
  4. Then they restored the lighting back to what it had originally been. And productivity went..... UP.

    Ironically, the name for this phenomenon is the Hawthorne Effect, after the GE plant where the original study took place.

    ReplyDelete
  5. Productivity was greater after the move than before? You don't say! Haven't these reporters been involving in preparing for an office move: from tired, shabby offices to bright, shiny offices (green or not)? Of course productivity was fractionally better -- office moves are stressful and annoying and detract from productivity. Stephen Milloy would have a field day with this rubbish 'study'.

    ReplyDelete
  6. It looks as though the article was based on a press release put out by property developers.

    ReplyDelete
  7. "Of these, according to Chart 6, one – Typist 4 – was more accurate in her previous surroundings!"

    Speaking as a touch-typist, the more I type, the faster and more accurate I become. It's a no-brainer, and if your typists aren't getting better as the time goes on, then they're not working as well as they should be.

    Since I don't type fulltime, my average isn't anywhere near as high as I'd like, but I also don't make a living from it.

    ReplyDelete
  8. The most productive thing you can achieve in a move is to do the move in such a way that the unproductive no-hopers get left behind. ie, move a sufficient distance to cause the dropkicks to quit.

    ReplyDelete
  9. PhillipGeorge(c)2011September 25, 2011 at 11:30 PM

    Prof, its a funny word 'normal' - progressives want to normalize gay marriage. But gays are down there at about 2-5% which is very abnormal statistically speaking. So what we need is a new improved normal - one in which an appreciation of numbers plays only a minor role. A sartorial more enlightened normal -for an age of reason - just like the French had.

    Anonymous - re: the Hawthorne effect. I think this is born out in working with delinquent children - any and all programs work. All they wanted was some attention. The milk of human kindness. Pasteurized, boxed and government approved or course.

    Gay normality, green offices and pasteurized milk. The new normal.

    RMS Titanic is making good speed - I heard it from the bridge.

    ReplyDelete
  10. At least it began "The new wave of ultra-green office buildings, IS” and not ARE.

    ReplyDelete
  11. Mother Hubbard's DogSeptember 26, 2011 at 9:50 PM

    That would be "minuscule", Prof, not "miniscule".

    ReplyDelete