Friday, March 27, 2020
If you write online surveys, please read this.
This post is a bit astray from my usual approach, and my usual topic areas, so bear with me.
For quite some time now I've been using online surveys to pick up a few extra dollars (it's not much, but it does cover a few little "extras" by way of gift cards). In the process, I've seen some really interesting research being done... but I've also seen some poor planning and design.
Admittedly, I think it unlikely that anyone who actually writes those surveys (especially the poorly-done ones) will ever read this, I thought it a good idea to put these tips out there. Maybe someone who writes online surveys, or will in the future, will read this and remember these thoughts. Or some AI that wanders the web for information on human thought and opinion will read it, and that will influence its reports to its owners (I know that such things do exist, and while a single blog post will only have a small effect, even a small effect is an effect).
Before I go into detail, I'd like to point out that, in the following, I'm deliberately avoiding mentioning most survey sites by name. The exceptions are if I can't find any way to contact them directly about the problem, or if I tried to contact them but couldn't change the behavior.
For clarity, I use a survey portal that sends me to various other sites for the actual surveys. I'd identify them (they're really good), but I'm not sure how they'd feel about me sharing that information, so for purposes of this blog post I'll just call them Survey Portal.
At least all but one of those sites are courteous enough to return me to my original tab once the survey is done (either successfully, or through disqualification). However, that once exception (affinnova.com) just keeps things going in the same window, with the tab handles disabled; if I want to transfer back from the window to a tab, I have to copy-and-paste the URL.
Is there a reason that the survey has to be be in a new, separate window? If not, then please let me stay in the same one, ideally in the same tab.
It's natural that a marketing survey would require a non-disclosure agreement, especially when it involves an upcoming advertising campaign, packaging design, television content, or anything similar.
So when I'm asked something along the lines of, "Do you agree to keep the information you're about to see in this survey confidential, and not share or discuss it with anyone?" and answer "Yes," I think I'm quite reasonable to expect to not be told next that I didn't qualify. (A couple of surveys from FocusVision's Decipher survey platform have even followed that up with something like, "Do you understand that, if you share any of this material, you could be subject to legal action?" And yes, when I answered "Yes," I was disqualified.)
Similar to that is questions like, "Do you agree to pay attention to the questions in this survey, and provide meaningful, thought-out answers?" That's another thing that shouldn't be followed by being told that I don't qualify.
The other example is with a couple of open-ended, text-answered screening questions. When I'm asked to write detailed text about a certain thing, and I write out a lengthy paragraph with the requested amount of detail, I definitely don't expect to see a "you're disqualified" page a half-second after pressing the "Continue" button. Someone reading the answer later might decide that they don't like what I wrote and nullify my responses, but that quickly? It doesn't make sense.
Also on the topic of "weird times to disqualify," there's one that's downright aggravating: to disqualify me after I've spent 5-10 minutes answering opinion questions. It seems like just common sense, as well as common courtesy, that anything that would disqualify me from a survey (other than an offense such as missing an "attention check" or "robot check" question) should be asked up front, not after I've gone through several dozen pages of responses.
(I've also had several cases of being disqualified without answering any questions, though I assume that has more to do with the interface between Survey Router and the survey host than with anything the host is responsible for.)
This is just good sense: Before you ask me about my organization, my children, my pets, my car, or anything else, first ask if I even have those things. Even if 99% of people do, there's still that 1%.
A similar thing that happens much less frequently is a question that starts with, "You mentioned earlier that..." followed by something that not only isn't true of me, but also wasn't discussed before -- sometimes because this is the first question on the survey. This may be a coding problem rather than a writing problem, but it still should be addressed.
Which email domain(s) do you use for personal purposes (eg. online shopping, ride-sharing apps, etc.)? *This question is required.
And worst of all is automatically moving forward with no option to go back. Doing that is practically guaranteed to skew the survey's accuracy, especially in the early questions when participants may not be aware of this "feature." It'll only be skewed a little bit, but why not make a survey as accurate as it can be?
Like I said, I'm not sure how others feel about this; I could be the odd person out on this. (That is to say: I could be wrong!)
In the best cases, there's been a question like "Do you have any feedback for us?" or "How could we improve?" followed by a box for typing in an answer. I've used that box to offer suggestions and corrections, and even seen some of these sites make improvements in some of the above areas (especially the "None of the Above" type).
It's kind of like how there's a Comments section here, where people can tell me how I did. I've probably missed a few things, or perhaps been overly harsh in one or two spots (I hope not), so anyone who happens to be reading this can tell me how I did and I can do better in the future.
So... how did I do?
For quite some time now I've been using online surveys to pick up a few extra dollars (it's not much, but it does cover a few little "extras" by way of gift cards). In the process, I've seen some really interesting research being done... but I've also seen some poor planning and design.
Admittedly, I think it unlikely that anyone who actually writes those surveys (especially the poorly-done ones) will ever read this, I thought it a good idea to put these tips out there. Maybe someone who writes online surveys, or will in the future, will read this and remember these thoughts. Or some AI that wanders the web for information on human thought and opinion will read it, and that will influence its reports to its owners (I know that such things do exist, and while a single blog post will only have a small effect, even a small effect is an effect).
Before I go into detail, I'd like to point out that, in the following, I'm deliberately avoiding mentioning most survey sites by name. The exceptions are if I can't find any way to contact them directly about the problem, or if I tried to contact them but couldn't change the behavior.
For clarity, I use a survey portal that sends me to various other sites for the actual surveys. I'd identify them (they're really good), but I'm not sure how they'd feel about me sharing that information, so for purposes of this blog post I'll just call them Survey Portal.
Keep It Together
Two or three survey sites that I go to insist on opening the actual survey in a new window. This is an inconvenience for me, as I prefer to keep just one browser window open, with simultaneously-open pages in different tabs.At least all but one of those sites are courteous enough to return me to my original tab once the survey is done (either successfully, or through disqualification). However, that once exception (affinnova.com) just keeps things going in the same window, with the tab handles disabled; if I want to transfer back from the window to a tab, I have to copy-and-paste the URL.
Is there a reason that the survey has to be be in a new, separate window? If not, then please let me stay in the same one, ideally in the same tab.
Weird Times To Disqualify
I've had a few surveys that will ask questions that seem like they should be easy, but then immediately disqualify me. I'll give a couple of glaring examples.It's natural that a marketing survey would require a non-disclosure agreement, especially when it involves an upcoming advertising campaign, packaging design, television content, or anything similar.
So when I'm asked something along the lines of, "Do you agree to keep the information you're about to see in this survey confidential, and not share or discuss it with anyone?" and answer "Yes," I think I'm quite reasonable to expect to not be told next that I didn't qualify. (A couple of surveys from FocusVision's Decipher survey platform have even followed that up with something like, "Do you understand that, if you share any of this material, you could be subject to legal action?" And yes, when I answered "Yes," I was disqualified.)
Similar to that is questions like, "Do you agree to pay attention to the questions in this survey, and provide meaningful, thought-out answers?" That's another thing that shouldn't be followed by being told that I don't qualify.
The other example is with a couple of open-ended, text-answered screening questions. When I'm asked to write detailed text about a certain thing, and I write out a lengthy paragraph with the requested amount of detail, I definitely don't expect to see a "you're disqualified" page a half-second after pressing the "Continue" button. Someone reading the answer later might decide that they don't like what I wrote and nullify my responses, but that quickly? It doesn't make sense.
Also on the topic of "weird times to disqualify," there's one that's downright aggravating: to disqualify me after I've spent 5-10 minutes answering opinion questions. It seems like just common sense, as well as common courtesy, that anything that would disqualify me from a survey (other than an offense such as missing an "attention check" or "robot check" question) should be asked up front, not after I've gone through several dozen pages of responses.
(I've also had several cases of being disqualified without answering any questions, though I assume that has more to do with the interface between Survey Router and the survey host than with anything the host is responsible for.)
Stop Assuming!
Sometimes I get a question like, "How many people work in your organization?" without even asking first if I have an "organization" for people to work in. (I don't.)This is just good sense: Before you ask me about my organization, my children, my pets, my car, or anything else, first ask if I even have those things. Even if 99% of people do, there's still that 1%.
A similar thing that happens much less frequently is a question that starts with, "You mentioned earlier that..." followed by something that not only isn't true of me, but also wasn't discussed before -- sometimes because this is the first question on the survey. This may be a coding problem rather than a writing problem, but it still should be addressed.
"None of the Above" and "Other" Are Your Friends
Not everything fits into neat categories, even things that intuitively should. Here are a few actual questions I've seen recently -- some mainly for illustration, though others as the worst offenders in this area -- with their answer choices:Which email domain(s) do you use for personal purposes (eg. online shopping, ride-sharing apps, etc.)? *This question is required.
Forward and Back
I don't know how others feel about this, but I find it a little disturbing when a survey automatically moves to the next question as soon as I click on an answer. What if I mis-click, and hit the wrong answer? I like to pause for a second or two before moving forward to make sure my answer is correct and complete. And, if I realize my mistake after I've moved forward -- such as if the wording of a later question clarifies an earlier one for me -- it's very helpful to have a "back" button on the screen (since using the browser's "back" button kind of screws up the survey's functionality) so I can correct it.And worst of all is automatically moving forward with no option to go back. Doing that is practically guaranteed to skew the survey's accuracy, especially in the early questions when participants may not be aware of this "feature." It'll only be skewed a little bit, but why not make a survey as accurate as it can be?
Like I said, I'm not sure how others feel about this; I could be the odd person out on this. (That is to say: I could be wrong!)
How Did We Do?
This is one item that's been a quite positive surprise for me. Several surveyors finish their surveys by asking things like "How was your experience taking this survey?"In the best cases, there's been a question like "Do you have any feedback for us?" or "How could we improve?" followed by a box for typing in an answer. I've used that box to offer suggestions and corrections, and even seen some of these sites make improvements in some of the above areas (especially the "None of the Above" type).
It's kind of like how there's a Comments section here, where people can tell me how I did. I've probably missed a few things, or perhaps been overly harsh in one or two spots (I hope not), so anyone who happens to be reading this can tell me how I did and I can do better in the future.
So... how did I do?
Post a Comment