Information on study participants (i.e., the people who filled out a survey, were interviewed or whose learning outcomes were measured) is usually included in the methodology section of a report. It is important that you look for information specific to a study’s participants. When this information is missing, you may not be able to understand where the results come from or to whom the results apply. In addition, without this information, it is difficult to evaluate whether a study’s results are relevant to other populations. For example, studies with too few participants, participants who are homogeneous or participants recruited through convenience rather than appropriateness cannot be as broadly applicable as studies with high numbers of diverse participants who more closely resemble the general population. The following are important things to look for when you are reading about a study’s participants:
a. The study participants and/or data source (e.g., existing data from
IPEDS) are described in detail, including how many are engaged in the study.
As a report reader, you should be able to easily see who the participants were or where the data for the study originated. It should be clear who was asked to participate, how many chose to participate and the overall demographics of those participants (such as gender, race, etc.) who participated. In the case of existing data, it should be clear how the data were collected and when. Having this information allows you to determine how generalizable the study results may be to larger populations than the smaller sample included in the study results.
b. It is clear how the participants were recruited for the study.
In order to assess whether or not the study participants were individuals who were appropriate for the particular study, it is important to know how the participants were recruited or selected. You will want to look in the report to see what specific procedures were used for recruitment (e.g., email, word of mouth, etc.). It is also important to know whether participants received any form of incentive or compensation.
c. The participant sample represents an appropriate level of diversity for the study aims.
Take note of whether the participant sample resembles the population that is being studied. The population includes everyone that the study is supposed to apply to (i.e., a study about college student learning is supposed to apply to all college students). For example, if you are studying the habits of college students, does the participant sample include an appropriate number of students at all class levels? In order for you to evaluate the study and generalize the findings, you need to understand if the participant sample adequately represents the population under study. In other words, would the results likely hold if all people in the population had been included in the study?
d. If subgroups are included in analyses, they are appropriately defined and labeled.
If the report includes subgroups (e.g., participants organized by gender, race, age, institution type or other variables), they are clearly labeled in all places where data regarding the subgroups is presented. This includes the use of any graphs, charts or tables to describe subgroup results. When subgroups are included in results reported, the size of the group(s) should always be included for you to reference in relation to the larger study participant sample.
Examples
a. The study participants and/or data source (e.g., existing data from IPEDS) are described in detail, including how many are engaged in the study.
Venable, M. A. (2023). 2023 Online education trends report. BestColleges.com. [
link]
- Pages 6-7 outline the data collection strategy and authors define key terms on page 7 which helps with the interpretation of results of the report.
Magda, A. J., Capranos, D., & Aslanian, C. B. (2020).
Online college students 2020: Comprehensive data on demands and preferences. Wiley Education Services. [
link]
- See pages 49-54 for methodology and participant demographic information.
b. It is clear how the participants were recruited for the study.
Linder, K., & Dello Stritto, M. E. (2017).
Research preparation and engagement of instructional designers in U.S. higher education: Results from a national study. Oregon State University Ecampus Research Unit. [
link]
- See page 36 for a description of participant recruitment.
c. The participant sample represents an appropriate level of diversity for the study aims.
Venable, M. A. (2023). 2023 Online education trends report. BestColleges.com. [
link]
- Page 6 outlines data collection strategy which includes a quota sampling approach to ensure geographic diversity for the study aims. The study sampled post-secondary students or administrators. Online students were further differentiated by prospective online learners, online learners, online graduate learners, and remote learners.
Betts, K., Miller, M., Tokuhama-Espinosa, T., Shewokis, P., Anderson, A., Borja, C… Dekker, S. (2019).
International report: Neuromyths and evidence-based practices in higher education. Online Learning Consortium. [
link]
- See pages 48-56 for information on this study’s sample. This study recruited a large sample that represented a range of higher education personnel including instructors, instructional designers, and administrators.
d. If subgroups are included in analyses, they are appropriately defined and labeled.
Clinefelter, D. L., Aslanian, C. B., & Magda, A. J. (2019).
Online college students 2019: Comprehensive data on demands and preferences. Wiley Edu, LLC. [
link]
- Data visualizations on pages 11 and 12 clearly label undergraduate and graduate students, and separate sections report results for each group respectively (i.e. some results for undergraduate students are reported on pages 13-15, some results for graduate students are reported on pages 16-18).
Venable, M. A. (2023). 2023 Online education trends report. BestColleges.com. [
link]
- The results of the study are separated by post-secondary students or administrators. Reporting of student views are separated by prospective online learners, online learners, online graduate learners, and remote learners to provide deeper understanding of the results.
What are theoretical frameworks?
In research, theories are explanations for the kinds of associations that researchers expect to find in a study. These explanations are based on prior research and understanding of the topic. For example, past research may have suggested that class attendance and higher grades tend to go hand in hand. A researcher might look at this work and decide to do a study to see if giving incentives for class attendance could improve grades. This past research and understanding regarding attendance and grades can provide an explanation for what the researcher expects to happen in their study (e.g., “Since students who attend class often get higher grades, I expect that incentives for attendance will lead to higher attendance, and therefore, improve grades.”). Theories can also provide explanations for the expected association among variables or concepts. For example, students who attend class often may receive higher grades because they interact more with course content by participating in discussions and class exercises. In research, theories are more than guesses that the researchers make personally, but are based on (or situated in) past work and explain why certain patterns and associations may exist in the study.
What are qualitative research methodologies?
Qualitative research methodology generally refers to open-ended approaches that do not include numbers or statistics. Some common qualitative methodologies include open-ended survey items (e.g., “Describe some challenges you encountered when beginning your online program.”), one-on-one interviews, focus groups and participant observation. In qualitative methodology, the data collected includes participant responses (e.g., what they said in response to the interview or survey questions) as well as researcher notes (e.g., notes on behavior the researcher observed). Researchers analyze the data by identifying common themes and patterns in participant responses/behavior. While qualitative methodologies are often used with small sample sizes, they can provide descriptions of phenomena that are in-depth and specific to context.
What are quantitative research methodologies?
Quantitative research methodology generally refers to closed-ended approaches that seek to collect numeric data and often involve statistics. Some common ways to collect quantitative data include closed-ended survey items (e.g., “Rate your satisfaction with your educational experience on a scale from 1 (very dissatisfied) to 10 (very satisfied).”), assessments (e.g., percentage of questions answered correctly on an exam), and frequencies (e.g., number of times students viewed an instructional video). Since quantitative data includes numbers, statistics can be used to answer questions that describe single variables (e.g., “On average, how satisfied are students with their courses?”), differences between groups (e.g., differences in satisfaction between traditional and nontraditional students), and identify relationships between variables (“How does satisfaction relate to exam scores?”). Quantitative methodologies often allow for large sample sizes and can provide a big picture of tendencies and overall associations.
What are mixed methodologies?
Research that utilizes mixed methodologies, or “mixed methods” research, uses a combination of qualitative and quantitative methods to investigate the research question(s). By using mixed methods, researchers can take advantage of what both qualitative and quantitative methods have to offer. A report that uses mixed methodologies will include a description of coding processes and themes (qualitative) as well as a description of statistical analyses completed and results of statistical tests (quantitative). Some studies will use the same participants for both the qualitative and quantitative components (e.g., Thirty participants completed a quantitative questionnaire and qualitative interview.), while other studies may use different groups of participants for each component (e.g., Thirty participants completed a quantitative questionnaire, while eight participants completed qualitative interviews.).
What is validity?
Validity in research refers to whether the study is measuring what is meant to be measured. For example, if a researcher wants to measure academic success, they need to a) define what academic success is, and b) find a way to accurately measure participants’ academic success levels. If a researcher were to define academic success as “performing well in required academic courses,” and then measure participants’ GPA, it would be important for researchers and readers (including you) to evaluate whether GPA measured what the researchers intended to study. A research report should describe how data was collected so you can evaluate whether the study accurately measured what was intended.
What is the difference between a population and a sample?
A population is the entire group of people that researchers hope a study can apply to. For example, if a study intends to inform learning in online higher education, then the population for that study would include all online learners in higher education (across institutions). Typically, it is nearly impossible to recruit everyone from a population to participate in a study. Researchers usually recruit a smaller number of participants within the population’s limits. For example, researchers might recruit 50 online learners from three different institutions in higher education to participate in the example study mentioned earlier. This set of participants that is recruited for the study is called a sample. It is important for research reports to describe the sample (participants in the study) so that you can see how the sample of a study differs from the general population. For example, did the participants of a study come from one institution? Is the race/ethnicity of the participants in the sample similar to that of the population? Often researchers seek to recruit a sample that is representative (or closely resembles) the population so that the results can more accurately apply to everyone in the population.
What is generalizability in research?
In research, generalizability describes whether research findings can apply to a broader population. For example, a study may have been conducted using a sample of 30 undergraduate students. While the findings of that study will apply to those 30 students, it is likely that the researchers hope the findings can apply to a larger population. For example, they may want to apply the findings to all undergraduate students at a particular university, or undergraduate students in general. Research is considered generalizable if the findings can be applied to a broader population than the sample used for the study.
Information about a study’s participants, methods and limitations of the research can help you evaluate whether findings may generalize to broader populations. For example, if a report identifies that their entire sample of participants were recruited from one university, that information can identify a possible limitation to generalizing the findings to university students in general.
What is data visualization?
Data visualization is the method researchers will use to show the data and study’s results. For example, researchers might include a bar graph that compares academic success for different student groups or a line graph of students’ GPA over time. Data visualization should make it easier for you to see the story that the data or research findings tell. Good data visualization should be easy to understand and should complement and add to the descriptions in the text.
What is conflict of interest?
It is important for researchers to do their best to be neutral in the research process. In other words, researchers should not be too invested in any particular outcome of a study. A conflict of interest occurs when a researcher is involved in something that could lead to bias in the research process. For example, if an individual does research at a university and consulting at a company, there would be a conflict of interest if the company they consult for were to fund their university research. While there are steps that can be made to reduce bias if there is a conflict of interest (such as the researcher asking a collaborator to handle the data), it is important for this kind of information to be included in research reports.
