Wednesday, October 24, 2012

A word on *that* mindshare survey - the importance of framing

The results of the Mindshare survey first published in the Business Times, and subsequently blogged about everywhere should be known to most Singaporeans who consume online news content.

While I believe that the survey results are broadly reflective of citizen sentiment on the ground, and I am most definitely not a pro-government apologist, I'd like to highlight the importance of framing of questions on surveys of this kind, and how they could influence the survey results.

In my mind (sic), I think the questions in the survey done by Mindshare were poorly phrased. In fact, they were far too leading to be considered well-designed survey questions. More likely, the survey was designed to elicit some form of expected reaction (which would have been anticipated by the surveyors), which would then be parleyed into some kind of media circus (which arguably has happened in the anti-establishment blogosphere).

Tellingly, no serious blogger whom I actually consider noteworthy has commented on the results of the survey. Granted, I probably am not a voracious consumer of online content. Still...

I have reproduced all six presumably Yes-No questions in the survey below. Beneath each question is a parallel version that I have written which I submit would return quite significantly different results.

When you have read all the questions, return to the first one and appraise all the Mindshare questions in their entirety. You will see that they are quite far from being neutrally phrased, and you will appreciate the power of framing in influencing survey results.

MindShare: I don't believe that I will be able to retire comfortably in Singapore.
My version: Do you think that you will be able to retire comfortably in Singapore?

MindShare: We cannot afford to get sick these days due to high medical costs.
My version: Do you think medical costs are affordable or too expensive in Singapore?

MindShare: Public housing prices are getting out of hand these days.
My version: Do you think public housing is reasonably priced today?

MindShare: I should not be spending my entire working life paying off my housing loans.
My version: ...

Honestly, this question has zero value. It has the blatant appeal to emotion behind it, which is why I said earlier that the survey doesn't pass the smell test of a bona fide study. 

The real kicker is not that 75% in the Mindshare survey answered "yes" to this question, but why 25% did NOT answer "yes". I can't imagine why anyone who has an operating brain cell would answer "No, I don't agree" to this statement.

MindShare: Political leaders are paid too much these days.
My version: Would you say that political leaders today are paid too little, too much, or at reasonable levels?

MindShare: There are too many foreign workers taking up job opportunities in our society today.

Depending on what you were specifically looking for, this question could have been framed in many different ways, all of them much more neutral than the Mindshare version, and all of them possibly yielding different responses.

My versions: 

Do you feel disadvantaged looking for a job in Singapore today due to the large foreigner population in Singapore today?

Do you think the large number of foreigners in Singapore makes looking for a job harder (for you)?

Given the large foreign population in Singapore, do you think the number of job opportunities is reduced?

Do you think the number of foreigners in Singapore has affected the unemployment situation?

Mindshare: Singaporeans should be granted priority in employment.

Ah, this question is very easily manipulated to give the desired response, once a suitable interpretation of the result is trotted out. Some examples...

My versions:

Do you think employment should be based on the principle of meritocracy?

If a foreigner is more qualified for a job than a Singaporean, do you think the Singaporean should still be given priority in employment?

All other things being equal, Singaporeans should be granted priority in employment.

If Singaporeans cannot be found to fill certain jobs, those jobs should be made available to foreigners.

Moral of the story: be very careful when reading the results of such surveys. Sometimes, the questions are more telling than the answers themselves.

No comments: