Report Reader Checklist: Reader experience

Reader experience
Reader experience

It is important for a research report to be accessible to all readers. It is also important that reports are written in way that all readers can easily comprehend the information. If these elements are missing, readers may have a difficult time reading and applying the report to their work.

 
a. The report uses language that is easy to understand.

All of the different elements of the report, from the description of the research to the findings and analysis, should be comprehensible to the average reader. The report should not rely too heavily on jargon or technical terms, especially when definitions of those words are not provided. Theories, concepts, terms and acronyms that are used in the study should be defined so that a reader without advanced knowledge can understand the study aims and context.

[see examples]
b. The report meets ADA accessibility standards.

The Americans with Disabilities Act (ADA) requirements include standards to ensure that documents are accessible for all readers, including those with print and sensory disabilities. For example, these standards outline the need to format documents to ensure screen readability for those who are blind or who need assistive technology in order to read virtual documents.

[see examples]
c. The report includes an executive summary and/or abstract for ease of digesting study findings.

Often included toward the beginning of a report, an executive summary and/or abstract will allow you to see what the overall findings or key takeaways are for the study. Executive summaries and/or abstracts are high-level summaries of a study and its findings; therefore, read the entire report if you are looking for more detailed information.

[see examples]
d. The report is an appropriate length for the study scope and reporting of results.

If elements of the report seem disjointed or if the report seems to be very long, then it may be that it could have been broken into more than one report. On the other hand, shorter reports do not always provide the level of detail needed to understand a study and its results. For this criterion, you can consider if the report was too long, too short or just right.

[see examples]

Examples

a. The report uses language that is easy to understand.

BestColleges (2020). Trends in online student demographics. BestColleges. [link]
  • This report uses comprehensible language and provides definitions for readers. For example, see pages 6 and 7.
[return to section]

b. The report meets ADA accessibility standards.

Dello Stritto, M. E., & Linder, K. (2018). Student device preferences for online course access and multimedia learning. Oregon State University Ecampus Research Unit. [link]
  • See PDF version of the report, which uses headers and alt-text for figures and tables to ensure screen readability.
[return to section]

c. The report includes an executive summary and/or abstract for ease of digesting study findings.

Brown, M., McCormack, M., Reeves, J., Brooks, D. C., Grajek, S., Alexander, B…Weber, N. (2020). EDUCAUSE horizon report, teaching and learning edition. EDUCAUSE. [link]
  • See pages 5-6 for an executive summary at the beginning of the report.
Garrett, R., Simunich, B., Legon, R., & Fredericksen, E. E. (2023). CHLOE 8: Student Demand Moves Higher Ed Toward a Multi-Modal Future. Quality Matters & Encoura Eduventures Research. [link]
  • See pages 4-5 for the executive summary included in the beginning of the report.
[return to section]

d. The report is an appropriate length for the study scope and reporting of results.

Galanek, J., Gierdowski, D., & Brooks, D. (2018). ECAR study of undergraduate students and information technology, 2018. ECAR. [link]
  • This report is an example of a report that is neither too short nor too long for the content covered.
Means, B., & Neisler, J., with Langer Research Associates. (2020). Suddenly online: A national survey of undergraduates during the COVID-19 pandemic. Digital Promise. [link]
  • This report is an appropriate length for the study scope and reporting of results.
[return to section]

Checklist areas

Context
Context
Methodology
Methodology
Sample
Sample
Reporting results
Reporting results
Transparency
Transparency
Reader experience
Reader experience

Download a one-page version of the checklist.

What are theoretical frameworks?

In research, theories are explanations for the kinds of associations that researchers expect to find in a study. These explanations are based on prior research and understanding of the topic. For example, past research may have suggested that class attendance and higher grades tend to go hand in hand. A researcher might look at this work and decide to do a study to see if giving incentives for class attendance could improve grades. This past research and understanding regarding attendance and grades can provide an explanation for what the researcher expects to happen in their study (e.g., “Since students who attend class often get higher grades, I expect that incentives for attendance will lead to higher attendance, and therefore, improve grades.”). Theories can also provide explanations for the expected association among variables or concepts. For example, students who attend class often may receive higher grades because they interact more with course content by participating in discussions and class exercises. In research, theories are more than guesses that the researchers make personally, but are based on (or situated in) past work and explain why certain patterns and associations may exist in the study.

What are qualitative research methodologies?

Qualitative research methodology generally refers to open-ended approaches that do not include numbers or statistics. Some common qualitative methodologies include open-ended survey items (e.g., “Describe some challenges you encountered when beginning your online program.”), one-on-one interviews, focus groups and participant observation. In qualitative methodology, the data collected includes participant responses (e.g., what they said in response to the interview or survey questions) as well as researcher notes (e.g., notes on behavior the researcher observed). Researchers analyze the data by identifying common themes and patterns in participant responses/behavior. While qualitative methodologies are often used with small sample sizes, they can provide descriptions of phenomena that are in-depth and specific to context.

What are quantitative research methodologies?

Quantitative research methodology generally refers to closed-ended approaches that seek to collect numeric data and often involve statistics. Some common ways to collect quantitative data include closed-ended survey items (e.g., “Rate your satisfaction with your educational experience on a scale from 1 (very dissatisfied) to 10 (very satisfied).”), assessments (e.g., percentage of questions answered correctly on an exam), and frequencies (e.g., number of times students viewed an instructional video). Since quantitative data includes numbers, statistics can be used to answer questions that describe single variables (e.g., “On average, how satisfied are students with their courses?”), differences between groups (e.g., differences in satisfaction between traditional and nontraditional students), and identify relationships between variables (“How does satisfaction relate to exam scores?”). Quantitative methodologies often allow for large sample sizes and can provide a big picture of tendencies and overall associations.

What are mixed methodologies?

Research that utilizes mixed methodologies, or “mixed methods” research, uses a combination of qualitative and quantitative methods to investigate the research question(s). By using mixed methods, researchers can take advantage of what both qualitative and quantitative methods have to offer. A report that uses mixed methodologies will include a description of coding processes and themes (qualitative) as well as a description of statistical analyses completed and results of statistical tests (quantitative). Some studies will use the same participants for both the qualitative and quantitative components (e.g., Thirty participants completed a quantitative questionnaire and qualitative interview.), while other studies may use different groups of participants for each component (e.g., Thirty participants completed a quantitative questionnaire, while eight participants completed qualitative interviews.).

What is validity?

Validity in research refers to whether the study is measuring what is meant to be measured. For example, if a researcher wants to measure academic success, they need to a) define what academic success is, and b) find a way to accurately measure participants’ academic success levels. If a researcher were to define academic success as “performing well in required academic courses,” and then measure participants’ GPA, it would be important for researchers and readers (including you) to evaluate whether GPA measured what the researchers intended to study. A research report should describe how data was collected so you can evaluate whether the study accurately measured what was intended.

What is the difference between a population and a sample?

A population is the entire group of people that researchers hope a study can apply to. For example, if a study intends to inform learning in online higher education, then the population for that study would include all online learners in higher education (across institutions). Typically, it is nearly impossible to recruit everyone from a population to participate in a study. Researchers usually recruit a smaller number of participants within the population’s limits. For example, researchers might recruit 50 online learners from three different institutions in higher education to participate in the example study mentioned earlier. This set of participants that is recruited for the study is called a sample. It is important for research reports to describe the sample (participants in the study) so that you can see how the sample of a study differs from the general population. For example, did the participants of a study come from one institution? Is the race/ethnicity of the participants in the sample similar to that of the population? Often researchers seek to recruit a sample that is representative (or closely resembles) the population so that the results can more accurately apply to everyone in the population.

What is generalizability in research?

In research, generalizability describes whether research findings can apply to a broader population. For example, a study may have been conducted using a sample of 30 undergraduate students. While the findings of that study will apply to those 30 students, it is likely that the researchers hope the findings can apply to a larger population. For example, they may want to apply the findings to all undergraduate students at a particular university, or undergraduate students in general. Research is considered generalizable if the findings can be applied to a broader population than the sample used for the study.

Information about a study’s participants, methods and limitations of the research can help you evaluate whether findings may generalize to broader populations. For example, if a report identifies that their entire sample of participants were recruited from one university, that information can identify a possible limitation to generalizing the findings to university students in general.

What is data visualization?

Data visualization is the method researchers will use to show the data and study’s results. For example, researchers might include a bar graph that compares academic success for different student groups or a line graph of students’ GPA over time. Data visualization should make it easier for you to see the story that the data or research findings tell. Good data visualization should be easy to understand and should complement and add to the descriptions in the text.

What is conflict of interest?

It is important for researchers to do their best to be neutral in the research process. In other words, researchers should not be too invested in any particular outcome of a study. A conflict of interest occurs when a researcher is involved in something that could lead to bias in the research process. For example, if an individual does research at a university and consulting at a company, there would be a conflict of interest if the company they consult for were to fund their university research. While there are steps that can be made to reduce bias if there is a conflict of interest (such as the researcher asking a collaborator to handle the data), it is important for this kind of information to be included in research reports.