This section should describe the details and process of how a study was conducted from start to finish. This information should be descriptive enough that you could conduct the study again (or replicate the study) if you wanted to. When information about the methodology of a study is missing, you may have difficulty understanding specifically what the study investigated and how it was carried out. This lack of understanding might prevent you from being able to evaluate how and where the study’s results can be applied. The following are general components to look for relating to methodology. The Sample area of the checklist also contains components to look for about the participants in the methodology section.
a. The report has a methodology section.
It is important for you to understand how the study was conducted, including who were the participants (see Sample area of checklist), how were they recruited (see Sample area), and how were the data collected and analyzed (see b and c below). This level of detail gives you a better understanding of the results of the study. Often, this kind of information will be included in one section of the report, or in a report appendix, and labeled as the methodology section.
b. It is clear how data were collected.
Understanding how data were collected or how researchers went about gathering information from people for the study can provide you with a better understanding of the overall results of a study. For example, were the participants interviewed? Were the data collected through an anonymous online survey? Was the study based on archival (already-existing) data? Knowing more about the data collection helps you to better assess the validity of the study results.
c. It is clear how data were analyzed.
In addition to understanding how the data were collected, it is important to know how the data were analyzed, that is, how the researchers combined data from multiple participants and presented them as results. If the results are quantitative: Did the researchers provide frequency data? Did they calculate averages? Did they calculate new variables from the data? If the results were qualitative: Were the data coded? If so, what methods were used to code the data? Sometimes studies will use both or a mixed-methods approach. This kind of methodological information can be described prior to the presentation of the results in a section of the report detailing the methodology used to collect and analyze the data. Alternatively, if this detail is not in the main report, readers may be directed to an appendix that has this information. To fully understand the results that are being presented, it is important to know, in detail, how the data were analyzed.
d. If statistical analyses were used, specific tests are named.
If the data analysis included statistical testing, the authors should describe the statistical tests used to formulate the study results. It is important for you to understand what statistical test was used on which variables. This allows you to determine if the statistical analyses were appropriate for the data type and research questions. Good reports will describe why they chose a particular statistical test. This description might occur as the data are being presented or in a section of the report detailing the methodology used to collect and analyze the data.
e. If coding was performed, the coding procedure is described.
If the methodology for the study includes qualitative data (such as responses from interviews or focus groups), the report should address how that data were handled. Qualitative data is frequently analyzed through a process called “coding” where researchers look for patterns in the data to draw conclusions. To determine whether a report includes this kind of analysis, you will want to look for the following: Did the study author(s) begin with predetermined themes before they analyzed the study data or did themes emerge as they reviewed the data? Did more than one person code the data to ensure agreement on data themes? It is important for you to understand how the authors created categories or synthesized qualitative data to draw conclusions, and a description of the coding procedure is an important element of the study methodology.
Examples
a. The report has a methodology section.
Fox, K. Bryant, G., Lin, N., & Srinivasan, N. (2020).
Time for class – COVID-19 edition part 1: A national survey of faculty during COVID-19. Tyton Partners and Every Learner Everywhere. [
link]
- See page 33 for an example of a methodology section.
Jaschik, S. & Lederman, D. (2019).
2019 survey of faculty attitudes on technology. Inside Higher Ed & Gallup. [
link]
- See page 9 for an example of a methodology section.
b. It is clear how data were collected.
Pomerantz, J. (2019).
XR for teaching and learning: Year 2 of the EDUCAUSE/HP campus of the future project. ECAR. [
link]
- See page 41 for a description of data collection.
Magda, A. J., & Buban, J. (2018).
The state of innovation in higher education: A survey of academic administrators. The Learning House, Inc. [
link]
- See page 32 for a description of how data were collected.
c. It is clear how data were analyzed.
Brown, M., McCormack, M., Reeves, J., Brooks, D. C., Grajek, S., Alexander, B…Weber, N. (2020).
EDUCAUSE horizon report, teaching and learning edition. EDUCAUSE. [
link]
- Descriptive information about the data analysis procedures are included on pages 56-57.
Linder, K. (2016).
Implementation of and solutions for closed captioning in U.S. institutions of higher education: Results from a national study. Oregon State University Ecampus Research Unit. [
link]
- See page 30 for the data analysis description.
d. If statistical analyses were used, specific tests are named.
Linder, K. (2016).
Student uses and perceptions of closed captions and transcripts: Results from a national study. Oregon State University Ecampus Research Unit. [
link]
- See page 31 for a description of the statistical analyses.
e. If coding was performed, the coding procedure is described.
Linder, K., & Dello Stritto, M. E. (2017).
Research preparation and engagement of instructional designers in U.S. higher education: Results from a national study. Oregon State University Ecampus Research Unit. [
link]
- See page 36 for a description of how data were qualitatively coded.
What are theoretical frameworks?
In research, theories are explanations for the kinds of associations that researchers expect to find in a study. These explanations are based on prior research and understanding of the topic. For example, past research may have suggested that class attendance and higher grades tend to go hand in hand. A researcher might look at this work and decide to do a study to see if giving incentives for class attendance could improve grades. This past research and understanding regarding attendance and grades can provide an explanation for what the researcher expects to happen in their study (e.g., “Since students who attend class often get higher grades, I expect that incentives for attendance will lead to higher attendance, and therefore, improve grades.”). Theories can also provide explanations for the expected association among variables or concepts. For example, students who attend class often may receive higher grades because they interact more with course content by participating in discussions and class exercises. In research, theories are more than guesses that the researchers make personally, but are based on (or situated in) past work and explain why certain patterns and associations may exist in the study.
What are qualitative research methodologies?
Qualitative research methodology generally refers to open-ended approaches that do not include numbers or statistics. Some common qualitative methodologies include open-ended survey items (e.g., “Describe some challenges you encountered when beginning your online program.”), one-on-one interviews, focus groups and participant observation. In qualitative methodology, the data collected includes participant responses (e.g., what they said in response to the interview or survey questions) as well as researcher notes (e.g., notes on behavior the researcher observed). Researchers analyze the data by identifying common themes and patterns in participant responses/behavior. While qualitative methodologies are often used with small sample sizes, they can provide descriptions of phenomena that are in-depth and specific to context.
What are quantitative research methodologies?
Quantitative research methodology generally refers to closed-ended approaches that seek to collect numeric data and often involve statistics. Some common ways to collect quantitative data include closed-ended survey items (e.g., “Rate your satisfaction with your educational experience on a scale from 1 (very dissatisfied) to 10 (very satisfied).”), assessments (e.g., percentage of questions answered correctly on an exam), and frequencies (e.g., number of times students viewed an instructional video). Since quantitative data includes numbers, statistics can be used to answer questions that describe single variables (e.g., “On average, how satisfied are students with their courses?”), differences between groups (e.g., differences in satisfaction between traditional and nontraditional students), and identify relationships between variables (“How does satisfaction relate to exam scores?”). Quantitative methodologies often allow for large sample sizes and can provide a big picture of tendencies and overall associations.
What are mixed methodologies?
Research that utilizes mixed methodologies, or “mixed methods” research, uses a combination of qualitative and quantitative methods to investigate the research question(s). By using mixed methods, researchers can take advantage of what both qualitative and quantitative methods have to offer. A report that uses mixed methodologies will include a description of coding processes and themes (qualitative) as well as a description of statistical analyses completed and results of statistical tests (quantitative). Some studies will use the same participants for both the qualitative and quantitative components (e.g., Thirty participants completed a quantitative questionnaire and qualitative interview.), while other studies may use different groups of participants for each component (e.g., Thirty participants completed a quantitative questionnaire, while eight participants completed qualitative interviews.).
What is validity?
Validity in research refers to whether the study is measuring what is meant to be measured. For example, if a researcher wants to measure academic success, they need to a) define what academic success is, and b) find a way to accurately measure participants’ academic success levels. If a researcher were to define academic success as “performing well in required academic courses,” and then measure participants’ GPA, it would be important for researchers and readers (including you) to evaluate whether GPA measured what the researchers intended to study. A research report should describe how data was collected so you can evaluate whether the study accurately measured what was intended.
What is the difference between a population and a sample?
A population is the entire group of people that researchers hope a study can apply to. For example, if a study intends to inform learning in online higher education, then the population for that study would include all online learners in higher education (across institutions). Typically, it is nearly impossible to recruit everyone from a population to participate in a study. Researchers usually recruit a smaller number of participants within the population’s limits. For example, researchers might recruit 50 online learners from three different institutions in higher education to participate in the example study mentioned earlier. This set of participants that is recruited for the study is called a sample. It is important for research reports to describe the sample (participants in the study) so that you can see how the sample of a study differs from the general population. For example, did the participants of a study come from one institution? Is the race/ethnicity of the participants in the sample similar to that of the population? Often researchers seek to recruit a sample that is representative (or closely resembles) the population so that the results can more accurately apply to everyone in the population.
What is generalizability in research?
In research, generalizability describes whether research findings can apply to a broader population. For example, a study may have been conducted using a sample of 30 undergraduate students. While the findings of that study will apply to those 30 students, it is likely that the researchers hope the findings can apply to a larger population. For example, they may want to apply the findings to all undergraduate students at a particular university, or undergraduate students in general. Research is considered generalizable if the findings can be applied to a broader population than the sample used for the study.
Information about a study’s participants, methods and limitations of the research can help you evaluate whether findings may generalize to broader populations. For example, if a report identifies that their entire sample of participants were recruited from one university, that information can identify a possible limitation to generalizing the findings to university students in general.
What is data visualization?
Data visualization is the method researchers will use to show the data and study’s results. For example, researchers might include a bar graph that compares academic success for different student groups or a line graph of students’ GPA over time. Data visualization should make it easier for you to see the story that the data or research findings tell. Good data visualization should be easy to understand and should complement and add to the descriptions in the text.
What is conflict of interest?
It is important for researchers to do their best to be neutral in the research process. In other words, researchers should not be too invested in any particular outcome of a study. A conflict of interest occurs when a researcher is involved in something that could lead to bias in the research process. For example, if an individual does research at a university and consulting at a company, there would be a conflict of interest if the company they consult for were to fund their university research. While there are steps that can be made to reduce bias if there is a conflict of interest (such as the researcher asking a collaborator to handle the data), it is important for this kind of information to be included in research reports.
