In proving reliability of their project results, online researchers have tried to find ways to adopt validity measures customarily used offline. Offline surveys have traditionally verified a portion (at least 15%) of reported participants. Validation isn’t mandated among U.S. researchers, though it nonetheless remains a best practice. (ISO standards call for a minimum 10% re-contact validation of interviews/cases and a 5% monitoring validation of interviews/cases with at least 75% of whole interview monitoring/listening.)

In Canada, members of MRIA (Marketing Research and Intelligence Association) are expected to monitor or validate 10% of  every interviewer’s data collection work in malls, door-todoor or telephone. Consumer Contact (Toronto, ON, Canada),
with 35 years in Canadian MR, cautiously entered the online space after 2000. VP-Client Services Gord Ripley shared, “We’re not really marketers or sales people. Our business is driven by client needs, and they asked us to enter the online research domain. When we did, we applied all our offline protocols to our online work.” ResearchByNet relaunched its new and improved online service in January 2005, serving a base featuring Canadian and U.S. MR agencies, plus a handful of end users.

Carrie Moyer, ResearchByNet’s Director, recently told RBR her group imported Canada’s 10% offline standard into its quant online research “to maintain the consistency of our online research with the other traditional data collection methods. Most of our competitors–software and panel companies–have focused on speed, panel size and cost, and less so on quality. Our focus has always primarily been on quality.”

“Quality Triggers™,” designed to monitor and validate responses, have become a staple in ResearchByNet’s online research projects. Interspersed at varied points throughout each survey, they, in effect, monitor every respondent survey. A ResearchByNet project director takes a near-final draft
of each client questionnaire and studies it to identify five or six aspects of a survey that a respondent might not pay close enough attention to, might try to lie about or attempt to skim through just to finish the survey. Affirmations or denials stated in the screening question are kept in mind to guard against inconsistencies that might crop up later. Grid questions that could be straightlined are considered and steps are taken so that scale questions offer all potentially appropriate answers.

ResearchByNet’s online Quality Trigger process actually attempts to “activate” a validation review of between 8% and 10% of all completed surveys for each client project. Ripley says ResearchByNet’s extra manual steps towards this goal are unique enough to warrant a patent request. “From our preliminary search of patents in place, no other company is
taking the extra step of having a person examine each individual case as a whole if several triggers go off,” Moyer remarked. “That human touch must be done.

“We’ve incurred additional costs–in salaries alone–but our clients aren’t seeing an unduly higher price. They save money in the long run because the thousands of cases we deliver are a cleaner set of data than another company’s thousand cases for a slightly lower price.” She added ResearchByNet’s business (accounting for only 5% to 10% of Consumer Contact revenue) has doubled since an advertising campaign touting the extra validation rolled out 18 months ago. “Clients who hear about our initiative get very excited about it and very involved in the process,” she stated.

Ripley noted a client ordering 1,000 cases gets just that–and for the price quoted in the RFP. “All of our panel partners have agreed to replace at no charge anybody and everybody we deem to be a problem. And our panel providers–the big U.S. panel companies–call what we do a very positive step because it helps them clean up their panels.”

Moyer commented, “We’re unique versus other companies, most of which use algorithms keyed on time spent by respondents on survey frames, number of clicks and so forth. Our project directors apply all available knowledge from hundreds of annual projects. They identify specific areas of the survey that could raise problems prior to its fielding,
then the survey is programmed. At the conclusion of each completed questionnaire, the system reports each case that set off a sufficient number of triggers to require a manual review. Our National Quality Control Department, which monitors and validates all of our research, does a ‘walk-through,’ screen by screen, to verify the data.” Moyer revealed that not every review causes removal of the questioned survey.
She is pleased ResearchByNet online triggers are producing the planned 8% to 10% ratio. “There is no online validation standard in Canada,” she revealed. “The only near-standard is the assurance that no respondent can complete a case  more than once. There is much discussion within the industry here that could lead to better policies and standard ethical practices for online,” she noted.

Moyer told RBR the number of triggers varies by survey. “We institute them on a case-by-case basis after examining each questionnaire’s potential pitfalls,” she related. “The types of questions a client wants asked–open ends, scales or closedends– drive implementation of the triggers.” Ripley described three main trigger types: screening triggers (built around termination questions), main body triggers and demographic triggers. New triggers emerge as time goes on. “We’ve created one for conjoints. As surveys get more sophisticated in their online research, we’re confident we’ll discover new triggers,” affirmed Moyer.

The experience of Consumer Contact’s interviewing force–some have been with the business for over 10 years–could be a reason to loosen validation standards, but that is not the case. “They do an excellent job, but it’s still important to validate their work,” Ripley asserted. “With no validity requirement online and considering how little we know about volunteer panelists–other than where they live, their income, etc.–clients have good reason to be unsettled about online surveys. But validating at least 10% of the cases shows our commitment to do everything possible to deliver the highest quality, build confidence in our data and to produce results as comfortable and acceptable as those found in telephone data.”

Reproduced from the July 2006 issue of Research Business Report by RFL Communications, Inc. (Skokie, IL), which also publishes Research Conference Report, Research Department Report and Pharma Market Research Report, three other MR newsletters. For more information about these publications, please send an e-mail request to info@rflonline.com. Or you can visit http://www.rflonline.com or call RFL at (847) 673-6288.

For a PDF version of this article click here