Skip to navigation Skip to content

Panel recruitment – best practice

By Mik S. Oddershede, partner

 

The perfect world

In a perfect world, this article would not have been written, since in a perfect world we would have no need for sampling. We would be able to ask everyone in the target group, and have a response rate of 100%. Unfortunately, the perfect world does not exist, and therefore we have to take a sample from the target group and on this basis test hypotheses and draw conclusions that are subject to the statistical uncertainty associated with sampling a target group. The purpose of this article is not to describe sampling, response rates or statistical uncertainty, but how we define and compile the target group, or "population", which is the statistical concept we use.
Reverting to the starting point of the perfect world, we would have access to everyone in the population and there would thus be no discussion of how we recruit and compile this population. Yet since this is not the case, we find it highly relevant to discuss how to best compile a population. At M3 Research, our starting point is that all sub-populations must be represented, i.e. all demographic and sociographic sub-populations. To achieve this, it must be possible for all sub-populations to be drawn, so that all sub-populations must be included in the database from which the overall population is drawn. We call this database an Internet panel.

 

Development

When George Gallup, in 1936, predicted that Roosevelt would beat Landon he used address lists from which he systematically extracted 50,000 addresses/respondents to beat his "analysis opponent", Literary Digest magazine, which had asked 2,000,000 readers and on that basis predicted that Landon would win the election. George Gallup hereby proved that it is not the number of respondents, but the population from which they are drawn, that is important to opinion surveys. As the telephone became more commonly used, and response rates in postal surveys declined, more and more research institutes began to use a combination of postal surveys and telephone interviews, and during the 1980s and 1990s only telephone interviews were used for most opinion surveys. Telephone interviews could be used because over 90% of the population had access to a fixed-line telephone. The introduction of new technology (Computer Aided Telephone Interviewing) made it possible to structure complex progression in a questionnaire, depending on the replies from the respondent. This was a quantum leap from previous times' very limited routing opportunities.
In the mid-1990s, the first Internet-based surveys took place, although naturally limited to people with access to the Internet, and it was thus not possible to perform representative surveys of a population defined, for example, as "People aged 18-60". The Internet opened up many new opportunities in terms of presenting pictures/films, as well as various other benefits with which most readers of this article are probably familiar. Yet the drawback in the mid/late 1990s was that it was not possible to perform representative surveys of broad population groups, since a number of sub-populations did not use the Internet. In the Nordic region, a breakthrough came in 2007, by when approximately 80% of the population had access to the Internet from their homes, apart from Finland, where only 69% had access, and by 2013 this figure had increased to over 90% (Source: Eurostat).

 

These figures clearly show that, to a high degree, the Internet can be used for representative surveys of both broader and narrower population groups. Yet this question still remains: How do we achieve access to the population in the best possible way in order to perform representative surveys?
There are various approaches to this problem and we can see that our colleagues use different methods to get access to the population when they compile their panels of respondents.

 

Telephone-based recruiting

If only telephone-based recruiting is used, there is a great risk that not everyone in the population has the opportunity to take part in the panel. Let us assume that a research institute contacts 600,000 people in the course of a year, in which case only 14% of the total population will be able to take part in the panel. We can also ask whether the persons who would like to take part in a telephone interview are representative of the population? At M3 Research we do not believe this to be the case, but the greatest problem is still that only 14% of the population even have the opportunity to take part in the panel. This is in conflict with how, in statistical terms, the entire population must be available when the sample is drawn, so that telephone-based recruiting does not provide representative surveys at all. If this were to be the case, as a minimum the research institute would have to contact all households, to give everyone an opportunity to participate. No research institutes in the Nordic countries are anywhere near fulfilling this criterion.
Furthermore, there is a risk that a specific type of person can be recruited via telephone, so that the panel would not be representative of the population.

Recruitment via individual websites via pop-up

Some of our colleagues recruit via individual websites, via pop-up. As in the case of telephone-based recruiting, the challenge is the considerable risk of not reaching the entire population, so that not everyone has the opportunity to take part in surveys. There is also a risk that these websites are visited by a specific type of person, so that the panel will not be representative of the population. We would claim that the fewer websites used in recruitment, the greater the risk of a panel not being representative. The example below illustrates that if 50% of the population have been on one or more of the websites used in recruitment, 42% of the population will have the opportunity to take part in the panel, entailing a considerable risk that the panel is not representative.

Recruitment via newsletters, newspaper subscriptions, etc.

The last recruitment method we have observed in the Nordic market is recruitment for a panel, whereby the "research institute" buys into lists sent out in conjunction with e.g. newsletters, or to other types of existing subscribers. This method is quite simply not usable, since no firms in the Nordic region have access to a sufficiently high percentage of the broad population. In addition, these lists often concern specific topics and the panel that is subsequently formed will consist of persons with an interest in one or several specific topics, and can thus in no way be described as representative of the general population. In the following example, the research institute has had access to lists covering 20% of the population.

Recruitment at M3 Research – multi-source

M3 Research was established in 2007 by partners Mik S. Oddershede and Steen Agerskov. Both partners have been active in several research institutes, specialising in online analyses, and have both been engaged in recruitment for online panels. The start-up of M3 Research was driven by the ambition to compile a panel from which a sample could be drawn that was representative of the broad population in the four Nordic countries, both combined and each country separately. We examined the advantages and drawbacks of the various options and found that a combination of online and offline recruitment had to be the optimum method, if we were to reach the entire population. Our constant objective has been that everyone in the population must be able to register for the panel. After this, based on a sample matrix, we will ourselves select the correct sample for a survey – irrespective of whether this is from a narrow or broad population group. All sub-populations must be represented, so that there is no bias in relation to the individual recruitment method. Our constant objective is to have more than 1,000 sources for the collection of respondents in the Nordic countries, ensuring that all subcultures are represented. From the start in 2007, M3 Research has used

In 2007-2009 we used offline media to collect respondents to a greater extent, but as Internet penetration has increased, in recent years we have had greater focus on online recruitment. This is also because today we can target even very small populations that we can identify as missing from the panel. M3 Research performs a weekly test of the panels in the four countries and based on this test, measures are launched in the sub-populations required, in order to ensure that we continue to have the largest and most quality-conscious Internet panel in the Nordic countries.