Evaluation Toolkit for Breastfeeding Programs and Projects

June 2012

4.3 Quantitative data collection methods

Page last updated: 04 November 2013

4.3.1 Surveys

Quantitative surveys are aimed at counting, or quantifying, responses to illustrate changes to a specific population group, changes over time or to allow comparisons to be made between groups. Quantitative surveys require the use of predominantly closed-ended questions. Many surveys will also add a small number of open-ended questions (such as asking for additional comments) to a survey to allow people to expand on their answers to other questions. A survey template is provided at Appendix G to help you get started on drafting a survey.

Surveys may be conducted in a paper-based format (e.g. a satisfaction questionnaire handed to every client who attends a clinic), on-line (e.g. a survey accessed through a web address so that anyone who has access to a computer and knows about the survey can participate), or face-to-face (e.g. a short interview about accessibility of a clinic conducted in the car park and nearby bus stop). Each method has its advantages and disadvantages, depending on the audience.
  • Paper-based surveys are useful when a population doesn’t have access to the internet or comfort with computers, or when you don’t know where to find the population other than in a defined place (such as handing the survey out to people who attend antenatal education classes).
  • On-line surveys have the potential to reach a wider audience, but assume that respondents have access to a computer (which may leave some people out).
  • Telephone surveys (sometimes called ‘CATI’ or computer-assisted telephone interviews) are conducted by a researcher who completes the survey on behalf of the person participating; CATI surveys are used for large, population-based surveys where randomly generated telephone numbers are called, and the person answering the phone is asked a series of closed-ended questions. Telephone surveys can also be used for smaller-scale evaluations, both for semi-structured interviews or for survey questionnaires.
  • Face-to-face surveys are good when you want a quick response, or to engage people in an immediate fashion (e.g. what do you think about this car park, this service, this environment), or when you don’t know where to find the population other than in a defined place. This is the most resource-intensive method as it requires a dedicated researcher to interview people, but it will provide immediate responses.
Online survey software tools are available which allows evaluators to develop, collate and analyse simple surveys using multiple choice questions, matrices, rating tools, etc. .2 These allow for the construction of quantitative and qualitative surveys, which can be administered via websites, email or blogs.

You can also use both paper-based and on-line surveys. The service in the boxed quote at the beginning of section 4 started out using Survey Monkey and then also sent out a paper version to make sure they weren’t leaving people out who didn’t have access to computers.

4.3.2 Sampling

A convenience sample is the most common in service evaluation. It simply aims to get service users to take the survey by, for instance, leaving copies in the waiting room, or by mailing a copy of the survey to every registered client. While this is convenient for contacting people who might complete the survey, it is not possible to generalise from a convenience sample to make statements about a wider population.

Another method of surveying is a stratified or purposive sample. It is aimed at a specific population, for instance young mothers aged 16 to 25 years, mothers from particular socio-economic or cultural backgrounds, or mothers who return to work within six months of giving birth.

As with a convenience sample, you would be focussing a one part of the wider community so the findings would not be representative of all people, but a purposive sample does potentially allow you to contact more of the people who may have a perspective on your particular topic, whether or not they use your service.

Sometimes, the numbers of participants in a program or project are too great to survey everyone, so a randomised sample may be used. This is useful for conducting population-based surveys over large geographical areas. For this, a valid sample is required (i.e. one which can be regarded as representative of the population being surveyed, without bias), and your sampling method needs to be outlined clearly. This method can allow results to be generalised (that is, discussed with a degree of confidence in relation to other, similar, research studies). A randomised sample needs to be of sufficient size to be able to account for sampling errors and for there to be confidence in the results.

A key point of sampling is that everyone who might be eligible to be included has an equal opportunity to be included. For convenience samples, you might leave surveys out in the waiting room (so for instance, everyone who attends the service in that week has the same opportunity to participate if they want). A convenience sample won’t include everyone who might possibly have a view, but within the bounds of what is convenient, it tries to include as many who might be available at that time. For stratified samples, it would be important to research the target population and come up with ways to reach as many as possible who might be eligible. You will need to know how many of that target group could be found within the local population (for instance, the percentage of the community which speak Chinese as a first-language, or are women under the age of 25, or women between 25 and 40). Your survey would seek to reach a good proportion of that population, which means you might have to go out and find them. You might, for instance, seek to reach women under the age of 25 through secondary and tertiary educational facilities, community centres or local recreational facilities, general practice surgeries and women’s health centres. Your study will be strengthened by being able to report that a certain percentage of all eligible females under the age of 25 had participated in your survey.

Randomisation requires even more rigorous sampling. This may require assistance from statistics textbooks, colleagues or evaluators who can help determine your target population, your required sample size and the most appropriate methodology for your survey.

4.3.2 Sampling

A convenience sample is the most common in service evaluation. It simply aims to get service users to take the survey by, for instance, leaving copies in the waiting room, or by mailing a copy of the survey to every registered client. While this is convenient for contacting people who might complete the survey, it is not possible to generalise from a convenience sample to make statements about a wider population.

Another method of surveying is a stratified or purposive sample. It is aimed at a specific population, for instance young mothers aged 16 to 25 years, mothers from particular socio-economic or cultural backgrounds, or mothers who return to work within six months of giving birth. As with a convenience sample, you would be focussing a one part of the wider community so the findings would not be representative of all people, but a purposive sample does potentially allow you to contact more of the people who may have a perspective on your particular topic, whether or not they use your service.

Sometimes, the numbers of participants in a program or project are too great to survey everyone, so a randomised sample may be used. This is useful for conducting population-based surveys over large geographical areas. For this, a valid sample is required (i.e. one which can be regarded as representative of the population being surveyed, without bias), and your sampling method needs to be outlined clearly. This method can allow results to be generalised (that is, discussed with a degree of confidence in relation to other, similar, research studies). A randomised sample needs to be of sufficient size to be able to account for sampling errors and for there to be confidence in the results.

A key point of sampling is that everyone who might be eligible to be included has an equal opportunity to be included. For convenience samples, you might leave surveys out in the waiting room (so for instance, everyone who attends the service in that week has the same opportunity to participate if they want). A convenience sample won’t include everyone who might possibly have a view, but within the bounds of what is convenient, it tries to include as many who might be available at that time. For stratified samples, it would be important to research the target population and come up with ways to reach as many as possible who might be eligible. You will need to know how many of that target group could be found within the local population (for instance, the percentage of the community which speak Chinese as a first-language, or are women under the age of 25, or women between 25 and 40). Your survey would seek to reach a good proportion of that population, which means you might have to go out and find them. You might, for instance, seek to reach women under the age of 25 through secondary and tertiary educational facilities, community centres or local recreational facilities, general practice surgeries and women’s health centres. Your study will be strengthened by being able to report that a certain percentage of all eligible females under the age of 25 had participated in your survey.

Randomisation requires even more rigorous sampling. This may require assistance from statistics textbooks, colleagues or evaluators who can help determine your target population, your required sample size and the most appropriate methodology for your survey.

2 Two examples are available at: Survey Monkey website and Survey Gizmo website.