Annals of Dubious Research Interpretation

When you do a one-off survey of young people, it’s impossible to distinguish an age effect from a cohort effect. It seems not everyone gets this.

This entry was posted in Uncategorized. Bookmark the permalink.

15 Responses to Annals of Dubious Research Interpretation

  1. Sacha Blumen says:

    Sorry Andrew, but as a mathematical and not a social scientist, what is a cohort effect? Is it (at a guess) something about surveying a non-representative part of a certain age group and drawing conclusions about people in that age-group, as opposed to surveying various age groups and then saying “oh variations in reponses to question X are somewhat related to age” ?

    I’m talking out of my hat here – sorry!

  2. Andrew Leigh says:

    Sorry Sacha, I shouldn’t use jargon. Cohort effects are effects specific to those who were born in a particular era (eg. I’ll always be a 1972 baby). Age effects, obviously, change over time (eg. I won’t always be 32).

  3. Sacha Blumen says:

    Ahhh – I’ll always enjoy being a 1973 baby!

  4. Mike says:

    Is it just my reading of this article or is it truly an appalling piece of writing. This is a trashy opinion piece. Long on rhetoric and short on fact. The Author has no credible background in sociology, psychiatry, human behaviour, economics…

    Andrew, Cohort groups aside I just don’t believe the conclusions she has reached. The generalisations are so wide and sweeping that they can not have any basis in fact.

    So naturally this was as a result of a far reaching investigation. Covering young people in diverse occupations. From differing cultural socio and economic backgrounds …sure

    “The survey was conducted by the labour market economist Richard Denniss for the Australia Institute. He asked 50 young people, between the ages of 23 and 28, in full-time work, about their attitudes to job security and wage bargaining.”

    http://www.youth.infoxchange.net.au/news/ix/results.chtml?filename_num=01178

  5. Sacha Blumen says:

    Wow! A sample size of 50. Definitive.

  6. Sacha Blumen says:

    Unfortunately there are slightly more than 50 in that population.

  7. mike says:

    I have spent about an hour doing searches to try to track down the original paper but have not had any luck.

    I have to say when the base info is trash so are the conclusions. I doubt that Richard Denniss intended any conclusions to be drawn from that small a sample.

    Sasha, what do you think a few hours down the Marble Bar and we can formulate policy.

  8. Sacha Blumen says:

    *laugh* Sounds good. And perhaps the confluence of much alcohol and bar-talk may inadvertantly produce good policy!

    No, I’m off to shop for my partner’s b-day tomorrow (yes, I’m a last-minute guy I know!)

  9. Andrew Leigh says:

    C’mon guys, size isn’t everything! Because of the admin cost, I tend to think of qualitative surveys as akin to medical trials, which frequently have about 50 participants. So long as you use proper statistical tools, a small sample size isn’t fatal.

    That said, I’m sure beer-induced policymaking is innately superior to the status quo…..

  10. mike says:

    Andrew,

    The small sample in this case is fatal. Way to many variables education levels alone makes it flawed. I really am surprised that I can not find the original paper the information is drawn from.

  11. Sacha Blumen says:

    Can you analyse data using very small sample sizes? (as a pure mathematician, I don’t know a great deal about analysing data)

  12. mike says:

    Sasha,

    I tend to be very cynical of small sampling. I once got 8 heads and two tails while tossing a coin my conclusions were:

    Heads is more probable than tails on the toss of a coin. The probability being 4:1

    The coin I had was flawed and was slightly miss weighted favoring the Head side.

    Even had I tossed the coin 50 times I think that the result would be something like 28:22. So in geneal terms I would have conclude that heads was more probable on the toss of that coin.

    Small samples are dangerous no matter what. I use this example if only to highlight the fact that with only two possible outcomes data can distort.

    You dont know a greal deal about analysing data PMSL.

  13. mike says:

    For the record the reason I have not been able to find the source document on the net is …wait for it… Richard Denniss has not yet written the paper !

    Richard Denniss:
    “the paper is not finished yet but i have been conducting a number of focus groups on which my comments were based.”

    Thanks to Richard for letting me know.

  14. Andrew Leigh says:

    Sascha, in general, I think it’s fine to use small samples. It just makes it harder to find results that are statistically significant at the 95% level (the level that most of us tend to go for). So it makes things harder for the researcher.

    For example, I just ran your coin example through Stata’s PRTESTI function. It told me that if I got 22 heads and 28 tails, the p-value on the difference would be 0.08 – in other words, an 1/12 chance that it happened by luck, and an 11/12 chance the coin is biased. This is a high probability, but it doesn’t attain the 95% confidence level.

  15. Richard Denniss says:

    Sorry to be so late to the conversation but i just came across it. As the Richard Denniss in question I hope i don’t sound defensive.

    The 50 or so ‘respondents’ were participants in 5 focus groups. The purpose of the focus groups was in no way intended be a representative sample of Australians, in fact, they were selected because they were not (ie I sought only young people in full time work).

    The purpose of the study was to delve a bit more deeply into the stereotype of the Gen Y employee. The results were facinating and convinced me of three things 1) the description of all Gen Y employees as go-getters is baseless. 2) Young people in full time work’s perception of work has been extensively shaped by their earlier 10 or so years spent in casual work. 3) most young people are, quite understandably, nervous about asking their boss for a pay rise. The most interesting thing about this, however, was how many of them thought that they were the only one to feel that way.

    Differnt versions of the paper have now been published in the Journal of Australian Political Economy and on the website of Greens Senator Rachel Siewert http://www.rachelsiewert.org.au

    I resigned from The Australlia Institute after I completed the focus groups so the follow up research has not been completed

    richard

Comments are closed.