When designing research projects, it’s essential to use a sample that is representative and large enough to be robust (within budget constraints), writes 3Gem Research.
Research designed to scrutinize any group of people, needs to be based on a sample that is representative of that group, in ‘the real world’. This includes research whose goal is to make inferences about the whole population of a country, and so needs to be recruited to be ‘nationally representative’ (sometimes shortened to ‘Nat Rep’).
This fundamentally means recruiting a sample of respondents that contains the correct proportions of important sub-groups, based on the relative sizes of ‘real world’ groups within whatever country is being researched, so that we can be confident that results, accurately reflect the views of the whole population i.e. if it was possible, to survey an entire population, we can be confident that its results would be statistically similar to the results that we are able to generate from our Nat Rep sample.
Generally, for Nat Rep surveys, this means ensuring different genders, age groups and the geographical regions of where people live, are sampled in proportions close to that country’s actual proportions.
In the UK, these target demographic proportions are generated from annual population estimates, provided by the government’s ONS website. These statistics are based on UK census data, which is a population-wide survey conducted every 10 years since 1801, and obligatory for all residents of England & Wales to complete (there is a similar census for Scotland). For non-UK countries, there are various sources of population information e.g. the CIA World Factbook.
Representative sampling
Based on this ONS data, 3Gem has created a picture of the UK reflecting the proportions of Men and Women, 6 age groups (18-24, 25-34, 35-44, 45-54, 55-64, 65+) and the 12 GORs (government office regions). These proportions are then used as quotas, to recruit a UK sample that allows us to be confident that overall results, will represent the views of the whole of the UK. In fact, for increased accuracy, we use interlocking quotas of age and gender within all UK regions, so you can be sure that our surveys precisely reflect the views of the UK, as opposed to surveys that just offer ‘a good demographic spread’, without controlling these demographic groups effectively – potentially a waste of time and money, as results are likely to be meaningless.
In some circumstances, we may be interested in other important groups and use additional quotas. For example, if the research topic was pensions and retirement, we may include additional quotas for Working Status, to ensure we have the correct proportions of Working, Non-working and Retired people (although, if Gender, Age and Region quotas are closely met, Working Status tends to fall out naturally, so we generally just monitor it using ‘soft quotas’).
One thing to mention is that some panel providers only include 5 age groups for quotas in Nat Rep surveys, with the upper group being ages 55+. This leads to a lack of accuracy and is likely to under-represent those aged 65+, arguably an age group where behaviours and opinions are likely to be most different to all other age groups, being past retirement age.
Sample Size
Something we are often asked is what size of sample is ‘robust’ for a Nat Rep survey. Whilst sample size is important, generating a representative sample using quotas is equally, if not more, important (as discussed above). For example a sample of n=1,000 that is based on representative quotas is preferable to a sample of n=2,000 that is skewed towards a particular gender or age group.
We’ve had feedback (from non-researchers) that there is an unwritten rule that a Nat Rep survey needs to be at least 2,000 respondents to be robust. However, if the sample has been recruited accurately using representative quotas, a total sample size of 1,000 respondents is sufficient. In fact, the increase in accuracy from a total sample size of 1,000 to 2,000 (dropping from +/-3.1% to +/-2.2%), may be difficult to justify based on cost efficiency alone. (although with the additional consideration, that it depends on what sub-group results are required – we would recommend only seriously using results of sub-groups based on a minimum of 50 people).