Note from the Client Summit’s producers:
On September 28th and 29th of 2006 at "The Research Industry Summit on Respondent Cooperation," Kim Dedeker, Procter & Gamble's VP-Global Consumer & Market Knowledge, disclosed a number of internal data quality issues P&G has been grappling with. Her presentations over those two days sparked an industry-wide discourse on research data quality and inspired our initiative to bring together representatives from the research client community to address the problem. The following bylined article by Ms. Dedeker, which first appeared in the October issue of Research Business Report, summarizes her remarks and set the tone for the Client Summit on Research Data Quality.
Research Quality: The Next MR Industry Challenge
By Kim Dedeker
As the head of Consumer & Market Knowledge (CMK) at Procter & Gamble, I’ve been talking recently about the importance of high quality research to support business decision-making. It’s vitally important for P&G, and I believe it’s crucial to the success of the entire market research industry.
At P&G, we have a fundamental belief that the Consumer is Boss. This means that the consumer is at the heart of every decision we make, on every brand, in every region of the world. The role of CMK is to provide insights that enable our brands to make the best decisions for consumers, which, in turn, drives business success for P&G.
I’m focused now on research quality because I fundamentally believe that if we do not have valid data from our research, then we’re letting our consumers down. There is no way to give them products that improve their lives if the data we’re using doesn’t accurately represent them and reflect their voices.
There are two areas that define quality research. First, we need end results to be both replicable and predictive. Secondly, we need to ensure that the underlying samples are representative of our consumer base and are engaged and providing honest opinions during surveys.
I’ll be the first to admit that over the years, we as clients, have put many pressures on the research suppliers. We’ve focused on cost and speed and an unintended consequence was erosion in quality. Now, we’re starting to feel the effects. Research quality has eroded as a result of these pressures, as well as increasing use of online research.
The area I feel is in the greatest need of help is in representative samples. I mention online research, because I believe it is a primary driver behind the lack of representation in online testing. Two of the biggest issues are the samples do not accurately represent the market, and professional respondents.
It’s difficult to get responses from people who represent the target market in online research. This is primarily because the percentage of people who have online access and will actually join a panel is an extremely small portion of the overall population. Some data suggests that only 1 in every 5000 people who receive an invitation to join a panel actually do. So, it’s virtually impossible to recruit a representative sample from this design.
Also, those who are available, have less time and energy to connect with us as marketers and researchers. We need to be respectful of their time and relevant to their lives. When we field a survey with 100 questions or more, we’re not respecting our consumers’ time. When we ask them questions about products or categories they don’t care about, we’re out of touch. And with each interaction, not only are we risking the accuracy of the research we’re conducting today, but we’re potentially damaging our relationship with that consumer for the future.
Secondly, there is a great deal of data that indicates there is an issue with professional respondents. ComScore has reported that 0.25% of Internet users are responsible for 30% of online surveys. This group completed, on average, 80 studies per quarter in calendar year 2004. And members of the survey panels of the 8 leading research firms each belong to an average of 7 other panels.
To further prove the point, P&G asked one of our own suppliers to do some methods research and they found that about 20% of their respondents willingly admitted to being on 5 or more panels. For my money, I’d bet it’s higher.
So, we know there are issues with research quality. There are many examples I could share of what can happen when research quality is compromised. Instead, I’d like to tell a story about the real pain for P&G. It’s something that we’ve seen time, and time again across businesses and across geographies. It’s when we field a concept test that identifies a strong concept. Then our CMK manager recommends that the brand put resources behind it. The marketing, R&D and research teams all invest countless hours and make a huge investment in further developing the product and the copy.
Then later, closer to launch, we field concept and use tests and get disappointing results. And rather than finding an issue with the product, we find that the concept was no good. We realize that the data we’d been basing our decisions on, was flawed from the start.
This is the part that is so hard for our brands and costly for our businesses. And over time, mistakes like this pose credibility issues for CMK and market research in general. If the data that we’re getting from our research isn’t replicable and predictive, then we may as well not do it at all.
At the end of the day, poor research quality can lead us to provide wrong counsel or direction to our brands. And we as researchers have a responsibility to deliver valid data to the company. We have to find the data/insights that convey the true voice of our consumer to provide sound consulting to our businesses.
Because of this, research quality is of paramount importance. This is why I think the industry – both research suppliers and clients – needs to focus on how to improve the overall quality of research.
To enable a productive solution to the data quality issue, I believe we have a responsibility to work together to drive efficiencies in our work processes and organizations to ensure that high-quality research can co-exist with value for our consumers and customers. The only way to get the problem solved is through a stronger partnership between clients and suppliers to create standards for quality so the industry will thrive and consumers will win.