Report Reader Checklist: Reporting results

Reporting results
Reporting results

The results of a study are often presented toward the middle-to-end of a report. This section should include the results of the data analysis (i.e., the statistics for quantitative research and the categories for qualitative research). When areas of the results section are missing or confusing, it can be difficult for you to understand what the study found.

 
a. All numbers used in the report are easy to comprehend.

When reading a report, you should never feel confused about how the authors arrived at a particular number or result. The numbers should add up correctly in both text descriptions and data visualizations. When report authors are sharing numbers based on calculations, those calculations should be described for you to assess their validity (appropriateness) in the context of the overall report. When looking at visual representations of the data (i.e., charts and graphs), they should be labeled so that it is easy to understand which numbers are represented in the figures.

[see examples]
b. An “N” is offered whenever data is being described in text, graph, table or chart.

Whenever data are being described, the total number of respondents or responses for all participants (or “N”) or for subgroups of participants (or “n”) should be clearly indicated. Percentages should include the context of the total number of participants. Similarly, the total N that is being broken down in any form of graph, table or chart should be labeled. Including Ns are important for the interpretation of the data. For example, if the results show 80 percent, it is important to know how many individuals this represents.

When the N and n are missing from a report, it can be difficult for you to evaluate whether there were enough participants to draw conclusions from the results. Additionally, it is important for you to see how many participants were in subgroups so you can further understand the sample and the results.

[see examples]
c. The report identifies missing data.

Sometimes, study participants choose to skip a question or they do not complete the entirety of a survey. Thus, a study result may contain missing data. Reports should include information about this missing data. This can be done when the sample is described; for example, missing data may result in a participant’s data being excluded from the analysis. Missing data can also be described at various points throughout the report when study results are shared (e.g., by noting missing data as a category in a graph or table).

When information about missing data is not included, it can be difficult for you to understand the numbers (see point a above) and the N and n (see point b above). Reporting missing data helps you easily see how numbers add up correctly and how many participants were included in each analysis.

[see examples]
d. It is clear where study findings fit in with the study’s purpose, research question(s) and methodology.

When results are presented in a report, you should be able to discern how the researchers arrived at the result and how the result fits in with the study’s purpose. For example, did the result answer a research question identified earlier in the paper? Was the data collection and analysis for the result described in the methodology section? If a finding is being presented and it is unclear how the finding resulted from the study’s purpose, research questions and methodology, then the result is without context. This lack of context can make it difficult for you to ascertain the finding’s importance to the report overall. For example, if you see a result in a report that doesn’t seem related to the study’s purpose, it is difficult to understand why the result is included and how/why it contributes to the goals of the study.

[see examples]
e. The data visualizations (graphs, charts and tables) enhance your understanding of the results.

Good data visualization will not only visually present the data but it will also add something to one’s understanding of the data. For example, a data visualization might represent trends or changes over time in the form of a line graph or show comparisons of different data points in a bar chart. Data visualizations can help you understand how results of the study relate to one another and relate to the study aims.

Data visualizations are quite diverse and certain methods of visualizing certain kinds of data can work better than others. Data visualizations are used to help explain data, so if they are confusing or hard to understand, then something is wrong. Data visualizations should be appropriate in size and scale so that the numbers represented are accurate. When reviewing data visualizations, you should also pay attention to whether the numbers add up correctly (e.g., a pie chart should add up to 100 percent). Also, pay attention to scale. When scale is not represented correctly, the relationship between data points can be exaggerated, distorted or just confusing. For example, if students could have scored between 0 and 100 on an exam, and the average scores are on a chart with numbers that range from 80 to 100, the differences between exam scores are going to look larger than if the chart’s numbers included the full range of 0 to 100 percent.

[see examples]

Examples

a. All numbers used in the report are easy to comprehend.

Dello Stritto, M. E., & Linder, K. (2018). Student device preferences for online course access and multimedia learning. Oregon State University Ecampus Research Unit. [link]
  • See results section (pages 8-22) for examples of how numbers are reported in text and data visualization.
[return to section]

b. An “N” is offered whenever data is being described in text or graph, table or chart.

Linder, K., & Dello Stritto, M. E. (2017). Research preparation and engagement of instructional designers in U.S. higher education: Results from a national study. Oregon State University Ecampus Research Unit. [link]
  • See pages 8-10 for examples in text and page 16 for an example of data visualization.
[return to section]

c. The report identifies missing data.

Dello Stritto, M. E., & Linder, K. (2018). Student device preferences for online course access and multimedia learning. Oregon State University Ecampus Research Unit. [link]
  • See data visualizations throughout the report (e.g. table on page 18) for information included about missing data.
[return to section]

d. It is clear where study findings fit in with the study’s purpose, research question(s) and methodology.

Gierdowski, D. C., & Galanek, J. D. (2020). ECAR study of the technology needs of students with disabilities. ECAR. [link]
  • See the study’s purpose on page 3, research question on page 7, information on methodology on pages 25-26, and the results on pages 8-22.
[return to section]

e. The data visualizations (graphs, charts and tables) enhance your understanding of the results.

Galanek, J. D., & Gierdowski, D. C. (2020). ECAR study of community college faculty and information technology. ECAR. [link]
  • See pages 10 and 12 for examples of quality data visualization.
Venable, M. A. (2023). 2023 Online education trends report. BestColleges.com. [link]
  • See pages 7 – 23 for examples of visualizations that add to the text description and help the reader interpret the results of the study.
[return to section]

Checklist areas

Context
Context
Methodology
Methodology
Sample
Sample
Reporting results
Reporting results
Transparency
Transparency
Reader experience
Reader experience

Download a one-page version of the checklist.

What are theoretical frameworks?

In research, theories are explanations for the kinds of associations that researchers expect to find in a study. These explanations are based on prior research and understanding of the topic. For example, past research may have suggested that class attendance and higher grades tend to go hand in hand. A researcher might look at this work and decide to do a study to see if giving incentives for class attendance could improve grades. This past research and understanding regarding attendance and grades can provide an explanation for what the researcher expects to happen in their study (e.g., “Since students who attend class often get higher grades, I expect that incentives for attendance will lead to higher attendance, and therefore, improve grades.”). Theories can also provide explanations for the expected association among variables or concepts. For example, students who attend class often may receive higher grades because they interact more with course content by participating in discussions and class exercises. In research, theories are more than guesses that the researchers make personally, but are based on (or situated in) past work and explain why certain patterns and associations may exist in the study.

What are qualitative research methodologies?

Qualitative research methodology generally refers to open-ended approaches that do not include numbers or statistics. Some common qualitative methodologies include open-ended survey items (e.g., “Describe some challenges you encountered when beginning your online program.”), one-on-one interviews, focus groups and participant observation. In qualitative methodology, the data collected includes participant responses (e.g., what they said in response to the interview or survey questions) as well as researcher notes (e.g., notes on behavior the researcher observed). Researchers analyze the data by identifying common themes and patterns in participant responses/behavior. While qualitative methodologies are often used with small sample sizes, they can provide descriptions of phenomena that are in-depth and specific to context.

What are quantitative research methodologies?

Quantitative research methodology generally refers to closed-ended approaches that seek to collect numeric data and often involve statistics. Some common ways to collect quantitative data include closed-ended survey items (e.g., “Rate your satisfaction with your educational experience on a scale from 1 (very dissatisfied) to 10 (very satisfied).”), assessments (e.g., percentage of questions answered correctly on an exam), and frequencies (e.g., number of times students viewed an instructional video). Since quantitative data includes numbers, statistics can be used to answer questions that describe single variables (e.g., “On average, how satisfied are students with their courses?”), differences between groups (e.g., differences in satisfaction between traditional and nontraditional students), and identify relationships between variables (“How does satisfaction relate to exam scores?”). Quantitative methodologies often allow for large sample sizes and can provide a big picture of tendencies and overall associations.

What are mixed methodologies?

Research that utilizes mixed methodologies, or “mixed methods” research, uses a combination of qualitative and quantitative methods to investigate the research question(s). By using mixed methods, researchers can take advantage of what both qualitative and quantitative methods have to offer. A report that uses mixed methodologies will include a description of coding processes and themes (qualitative) as well as a description of statistical analyses completed and results of statistical tests (quantitative). Some studies will use the same participants for both the qualitative and quantitative components (e.g., Thirty participants completed a quantitative questionnaire and qualitative interview.), while other studies may use different groups of participants for each component (e.g., Thirty participants completed a quantitative questionnaire, while eight participants completed qualitative interviews.).

What is validity?

Validity in research refers to whether the study is measuring what is meant to be measured. For example, if a researcher wants to measure academic success, they need to a) define what academic success is, and b) find a way to accurately measure participants’ academic success levels. If a researcher were to define academic success as “performing well in required academic courses,” and then measure participants’ GPA, it would be important for researchers and readers (including you) to evaluate whether GPA measured what the researchers intended to study. A research report should describe how data was collected so you can evaluate whether the study accurately measured what was intended.

What is the difference between a population and a sample?

A population is the entire group of people that researchers hope a study can apply to. For example, if a study intends to inform learning in online higher education, then the population for that study would include all online learners in higher education (across institutions). Typically, it is nearly impossible to recruit everyone from a population to participate in a study. Researchers usually recruit a smaller number of participants within the population’s limits. For example, researchers might recruit 50 online learners from three different institutions in higher education to participate in the example study mentioned earlier. This set of participants that is recruited for the study is called a sample. It is important for research reports to describe the sample (participants in the study) so that you can see how the sample of a study differs from the general population. For example, did the participants of a study come from one institution? Is the race/ethnicity of the participants in the sample similar to that of the population? Often researchers seek to recruit a sample that is representative (or closely resembles) the population so that the results can more accurately apply to everyone in the population.

What is generalizability in research?

In research, generalizability describes whether research findings can apply to a broader population. For example, a study may have been conducted using a sample of 30 undergraduate students. While the findings of that study will apply to those 30 students, it is likely that the researchers hope the findings can apply to a larger population. For example, they may want to apply the findings to all undergraduate students at a particular university, or undergraduate students in general. Research is considered generalizable if the findings can be applied to a broader population than the sample used for the study.

Information about a study’s participants, methods and limitations of the research can help you evaluate whether findings may generalize to broader populations. For example, if a report identifies that their entire sample of participants were recruited from one university, that information can identify a possible limitation to generalizing the findings to university students in general.

What is data visualization?

Data visualization is the method researchers will use to show the data and study’s results. For example, researchers might include a bar graph that compares academic success for different student groups or a line graph of students’ GPA over time. Data visualization should make it easier for you to see the story that the data or research findings tell. Good data visualization should be easy to understand and should complement and add to the descriptions in the text.

What is conflict of interest?

It is important for researchers to do their best to be neutral in the research process. In other words, researchers should not be too invested in any particular outcome of a study. A conflict of interest occurs when a researcher is involved in something that could lead to bias in the research process. For example, if an individual does research at a university and consulting at a company, there would be a conflict of interest if the company they consult for were to fund their university research. While there are steps that can be made to reduce bias if there is a conflict of interest (such as the researcher asking a collaborator to handle the data), it is important for this kind of information to be included in research reports.