Michael Shermer has a brilliant Baloney Detection Kit which can “help draw the boundaries between science and pseudoscience”.
But don’t let the science and pseudoscience thing put you off – it has other simpler uses. In a recent lecture I asked my students to apply the detection kit to a particularly bad piece of “research”.
The report was modestly entitled “Sweden’s Largest Facebook Study” it’s available online here. It’s in English with a Swedish summary. There are several easy-to-spot flaws with the report but the first thing that sets the alarm bells off is the fact that the report was not released into academia but was given exclusively to the media. This meant that before anyone could actually read the report the media was gleefully telling us that researchers have proven that “Facebook makes us miserable”, it spreads “unhappiness” and we are even miserable if we don’t logon often enough.
People who disagreed couldn’t do anything since the report was not available. This may be good marketing but its also very bad science. Things did not get better when the embargo was over and the report was released.
The first thing that strikes the reader is that there is a PR company’s logo above the university logo. I’m all for partnerships but PR isn’t research and research isn’t PR. Credibility in the research was not enhanced.
The next thing that hit me was the research location and topic. The report is published at Göteborg Research Institute which is under the School of Business, Economics and Law but the highest ranking researchers where from psychology. Releasing a report to the media instead of peer review is one thing but not publishing within your own discipline is also a bit odd – what are these people attempting to conceal…
Then I looked at the authors’ qualifications. Two PhD students, three undergrads and a PR person. Seriously??? This is not a guarantee for good science! Not that it makes that much of a difference but the undergrads were not even students at the same university. Curious.
The final thing that got me going was the subjects included in the study. Sweden has 9 million people and 52% of them are on Facebook (Olle Findahl Svenskarna och Internet 2011 has an English summary). The “Largest Facebook Study” is based on data from an online survey of 1011 respondents. This hardly qualifies it as largest anything. Getting three undergrads to pester 1000 friends to fill in a couple of bad questions may give you data to play around with but its hardly the basis for making the statements they make in the report. Asking people if they are happy? Seriously?
But we had lots of fun in class – I gave my students the report and the baloney detection kit which is based around 10 questions (more details here):
1. How reliable is the source of the claim?
2. Does the source make similar claims?
3. Have the claims been verified by somebody else?
4. Does this fit with the way the world works?
5. Has anyone tried to disprove the claim?
6. Where does the preponderance of evidence point?
7. Is the claimant playing by the rules of science?
8. Is the claimant providing positive evidence?
9. Does the new theory account for as many phenomena as the old theory?
10. Are personal beliefs driving the claim?
They tore the report to shreds! They had questions, spotted errors and anomalies after the briefest of readings. I am proud of them all.
The whole thing reminded me of the Martin Rimm affair of 1995. Rimm was a student at Carnegie Mellon who had also carried out “research” and arrived at the fascinating conclusion that 83.5% of images traded on Usenet are pornographic. He secured a publication in a law review (prestigious, but not used to dealing with statistics) and then sold an exclusive to Time Magazine who used it as a cover story
Naturally when the article with the glorious title
“Marketing Pornography on the Information Superhighway: A Survey of 917,410 Images, Description, Short Stories and Animations Downloaded 8.5 Million Times by Consumers in Over 2000 Cities in Forty Countries, Provinces and Territories.”
was finally made available it was torn to shreds, the law review was criticized and so was the editor of Time Magazine. It was not a pretty sight.
So what are we left with? Two PhD students gained some buzz in Swedish media but are viewed with suspicion in academia – this is going to follow them no matter how much they will eventually try to downplay it. The research institute and the School of Economics has sullied its name (why on earth did they let this happen??). I also wonder were the supervisors were in all of this? They may be innocent bystanders but still they have to take some of the heat.
So what we are left with is one happy PR company. What a mess baloney makes….