Finding the Right Sample Size (the Easy Way)
This article is rated as:
How many respondents do we need to take our survey? Whether it be survey respondents, program participants, or any other group of interest, evaluators are often posed with these “how many” questions. That is, what sample size is required to glean meaningful insight from the data being collected?
The answer: it depends.
Sample size determination depends on the questions being asked and the resources (e.g., time and money) allocated to answering these questions. However, there are tools available to calculate an estimated sample size that is large enough to provide statistically viable results, while small enough to be manageable.
This article will briefly define sample sizes, their importance, and how to calculate them (or how to use a tool to calculate them).
These suggestions are valid for simple project designs (e.g., survey, administration data). Other resources should be consulted for more complex, research-based programs.
Population v. Sample
Before getting to the calculation of sample sizes, we need to first be clear on the difference between a population and a sample.
Population
A population includes all observations, or members, within a group of interest. For example, if we are interested in staff engagement within an organization, the population would include all staff within said organization. Further, if we were interested in staff engagement within the marketing department of the same organization, our population now becomes all marketing department employees.
Sample
A sample is one or more observations (i.e., samples) taken from a population. Using the example above, if the organization employs hundreds of employees, it may be difficult to survey all staff. Therefore, we would take a random sample of staff across the organization in hopes that the sample is representative of the population.
In most instances, sampling an entire population is not feasible. That is why smaller samples are taken from populations. These samples should be large enough to detect statistical differences within the data, but small enough as to not drain all program resources. The number of samples taken from a population is effectively the sample size.
The importance of sample sizes
Sample sizes are important for detecting statistically significant outcomes from your data. Generally, small sample sizes are less representative of the population of interest.
Small sample sizes are more variable and increase the likelihood of rejecting a hypothesis (i.e., fail to detect differences). On the other hand, large sample sizes are more likely to produce better statistical results but come at the cost of increased resource use and, potentially, ethical concerns from sampling more people or subjects than necessary.
Small sample sizes
Pros:
Easier to collect
Less resource intensive
Cons:
Increased variability (e.g., outliers may skew results)
Sampling is less reproducible (e.g., resampling less likely to produce similar results)
Increased likelihood of accepting a false hypothesis (i.e., smaller sample sizes may detect significant differences in the data where no significant difference exists within the population)
Large Sample Sizes
Pros:
Less variability (e.g., outliers less likely to skew results)
Sampling is more reproducible (e.g., resampling likely to produce similar results)
Increased likelihood of accepting the correct hypothesis
Cons:
More difficult to collect
More resource intensive
Ethical concerns (i.e., is it ethical to collect data from thousands of people where dozens or hundreds would suffice?)
However, selecting the appropriate sample size is not as simple as choosing a random number. Sample sizes should be large enough to get accurate, statistically significant results, yet small enough to not overburden the project.
Estimating sample sizes (The Easy Way)
There are numerous methods for calculating sample sizes. The easiest way is to simply use a pre-made tool (check out our free sample size calculator HERE).
Like many online calculators (Calculator.net or Survey Monkey), we use Cochran’s sample size formula (Cochran, W. G., 1977) to estimate sample sizes. Read Finding the right sample size (The Hard Way) for the statistics behind the calculation or use a sample size calculation tool for a quick and hassle-free sample size estimate.
Sample size is not all that matters
Estimating a statistically significant sample size will help improve the validity of your analysis and results. However, sometimes these sample sizes will not be met due to logistics, ethics, or some other external constraint. Does this mean that your data are not valuable? Not at all.
There will be times when you are unable to get the desired estimated sample size. While this may limit the statistical power of any statistical tests run on the data, it does not negate the data. The feedback on a survey can provide valuable insights, regardless of statistical significance. And perhaps generalizing to the general population is not necessary. The goal of the survey may be more relevant to a single time point or program, and the results are less about generalizability and more about getting direct feedback from a program.
For example, when evaluating staff satisfaction using a survey, you may have a population of 100 and an estimated population size of 80 (assuming a 95% confidence level, 50% population proportion, and 5% margin of error). However, when you get the surveys back, you only receive 60 responses. This does not invalidate your results. While running statistical analyses may be limited by the sample size, you are still able to draw some insights from the data. You can get an understanding of satisfaction and dissatisfaction among the respondents which may reflect the current satisfaction of most staff. But generalizability may still be an issue. This is where triangulating your data can help. If you have other data (e.g., other outcome data, interviews, focus groups) that corroborate the results of the survey, you can have more confidence in the validity of the survey results. Tying in other quantitative or qualitative results with the survey data strengthens your findings and provides more confidence that the survey results are generalizable.
Therefore, sample sizes should be addressed on a case-by-case basis. Make efforts to understand the needs of your evaluation, survey, or program to make the best decision with the data you were able to capture. What questions are you trying to answer and are you looking to generalize these answers at a population level? If not, there is likely some wiggle room in how many samples you should collect. However, using sample size estimate provides a “nice to have” where possible, as it will strengthen the conclusions drawn from your analysis.