On Wed, Apr 4, 2012 at 9:36 AM, Audrey Abeyta <audrey.abeyta <at> gmail.com> wrote:

Hi Laura, 

Thank you for your feedback. You're absolutely correct: I should have specified that currency is in US dollars (I have now specified the currency in the question text). I do, however, have a question that asks about the respondent's country of residence. The questions in this questionnaire were adapted from Hars & Ou (2001), so I tried to deviate from their structure as little as possible. 

I haven't read Hars & Ou.  My research background is I probably best described as education, marketing and sociology based.  (My dissertation topic is actually fundamentally about online research methods.)

When Hars & Ou did their work in 2001, were they conducting research in online communities?  And were they dealing in global populations?

By not asking both language, country and metro area, by not allowing the expression of income in a local sense, you are creating a junk survey that will not be repeatable.  If you look at the cost of living in Texas and compare it to Chicago, Illinois, there is a huge gulf.  The cost of housing, of petrol, the local taxes, the cost of medical care, the local commodities in terms of food and clothing mean that $8,000 will go much, much further in Texas than they will in Chicago.  In turn, the cost of living in Chicago will be cheap when compared to Sydney and Canberra.  These will look a bit more reasonable when you compare the cost of living to say Tokyo or Moscow.  $8,000 USD does not go very far in Chicago, Sydney, Canberra, Tokyo, Moscow when compared to Texas.

I would STRONGLY urge you to either put in a question that asks country and metro area, and then correct for this by adjusting for cost of living when doing your final results.   If you can't do that, I would STRONGLY urge you to remove the question because the data will be completely meaningless.  (Minimum wage in my territory is $17.78 USD.)
 

Your concerns regarding the over/underrepresentation of certain segments of the Wikipedia population are also well-founded. Because respondents are volunteers, I am aware that there may be a large sampling bias, which I will do my best to correct for during statistical analysis. Additionally, I will acknowledge this limitation in the discussion section of my thesis.

How will you do sampling correction?  I don't see a language connection for one.  The survey just says "Wikipedia", not "English Wikipedia" so I assume you're talking about all Wikipedias.  If not, you will want to consider that my own response included experiences with Simple Wikipedia.  You asked time spent editing Wikipedia, but did not ask the type of work done on the site, nor the volume of edits done, nor the status on Wikipedia.  how are you going to correct for an over representation of English Wikipedia contributors, female contributors, the admin core, and power contributors?

This is hugely important.  If you don't have questions for allowing for those connections, if you don't deliberately seek out minority responses but instead advertise to a select selecting population, your results will be fundamentally flawed and not repeatable.  Given your research questions, I suspect if we both advertised this survey, we would get differences in answers that extremely different and STATISTICALLY significant.

The research design here just looks very, very poor and like there is very little done to correct for groups that may have an incentive to contribute versus occasional contributors who have less of an incentive to contribute and complete your survey. 

--
twitter: purplepopple
blog: ozziesport.com

_______________________________________________
Wiki-research-l mailing list
Wiki-research-l <at> lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l