Interpreting Research Studies:
First read the
ief “Interpreting Research Studies” by the Guttmacher Institute and the
article “Eight questions to ask when interpreting academic studies: A primer for media”
published by the Journalist’s Resource.
After wards read the Powerpoint® presentation “Interpreting Research” written by
Professor Hayward. It includes many explanatory details of the items listed on the next
three pages. It analyzes the research paper “Alge
a for 8th Graders” which is also
included in the Final Exam folder in Canvas.
Finally create your own Powerpoint® or Slideshare® of a research paper which you
eviewed and analyzed. Use the information you gathered in the next two pages,
namely, the ‘Data Worksheet’ and the ‘Your Analysis’ portions of the FE Paper Outline
document. Use my Powerpoint® “Interpreting Research” as a guideline to create your
own slideshow.
When you submit the presentation, please submit your data worksheet as well. I should
already have a copy of the research paper you reviewed since it needed pre-approval, so
it is not necessary to submit with your data worksheet, your analysis, and your
slideshow.
The data worksheet can be completed with short answers and incomplete sentences.
The ‘Your Analysis” responses should be written in your prefe
ed essay style in
paragraph form with complete sentences. You may not need to answer every prompt in
these four sections, so do not feel compelled to- just answer the ones that make the
most sense for the research paper you are analyzing. Remember all 5 components (the
data worksheet and the four analytic responses) are contained in the ru
ic and that is
what you are being assessed on.
Format of studies:
• Author(s)/ Publisher
• Abstract
• Background/Context
• Method
• Results
• Implications/ Discussion
• Conclusion
• References/ Author(s)
Data Worksheet
Type of Study (Survey or Experiment; Social Science or Medical):
Design of Study: (Questionnaire, Review, Double blind, randomness, …)
Population being measured:
The unit of analysis: (Individuals, classrooms, hospitals, schools or states, for example.)
Sponsor of study/possible biases:
Type of data collected: (Qualitative, Quantitative, nominal, ordinal, …)
What are the independent and dependent variables: (Name them if an acronym is given.)
Types of graphs used:
Distribution type: (Binomial, Poisson, Chi-Square…)
Goodness of fit tests: (Chi-square, …)
Sample size(s):
Mean & Standard deviations:
Confidence Intervals: (if available)
Standardization test used: (z-test or t-test)
Hypothesis:
P-value(s)/strength:
Effect size: (if given)
Analysis of Variance and covariance used: (ANOVA, MANOVA, …)
Regression strength/Co
elation Coefficient:
Your analysis
Your analysis of the abstract:
• What makes the study important?
• Who conducted the research and wrote the report?
• Who published the report?
• What are the researchers’ (null & alternative) hypotheses?
Your analysis of the process/methods:
• Did the researcher select an appropriate group for study?
• If comparison groups are used, how similar are they?
• Are the methods appropriate to the research purpose?
• Was the sample size sufficient for the population being studied?
Your analysis of the results:
• What results did the researchers find?
• Do the findings make sense?
• How generalizable are the results?
• Does the study establish causation?
• Could the data be biased as a result of poor research design?
• How well does the study design address co
elation? causation?
• Are the results statistically significant?
Your analysis of the conclusion:
• What is the feasibility of replication?
• What limitations do the authors note?
• What has changed since the information was collected?
• Is the time frame long enough to identify an impact?
• What conclusions do similar studies draw?
Interpreting Research Studies
In Brief
Interpreting Research Studies
Key Questions to Ask When Reading a
Social Science Research Report
• What makes the study important?
• Do the findings make sense?
• Who conducted the research and wrote
the report?
• Who published the report?
• Did the researcher select an appropriate
group for study?
• If comparison groups are used, how
similar are they?
• What has changed since the information
was collected?
• Are the methods appropriate to the
esearch purpose?
• Does the study establish causation?
• Is the time frame long enough to identify
an impact?
• Could the data be biased as a result of
poor research design?
• Are the results statistically significant?
Yet valuable research is often communi-
cated in technical language and rigid
formats that make it difficult to interpret
and evaluate the findings. This document
is intended to help demystify social
science research for those who could make
use of the findings but lack specialized
training in research methods. It identifies
the key questions to ask when evaluating a
esearch report, explains why the answers
matter and offers tips on where to find the
information in the body of the report.
What makes the study important?
What makes a study newsworthy, or useful
for informing policies and programs? It
depends on how the study contributes to
what we already know. This information
may be summarized in the abstract of a
scientific journal article or the executive
summary of a longer report. More in-depth
information is usually found in the
“Discussion” or “Conclusions” sections at
the end of the report. Look for answers to
these questions:
• Does the study answer a previously
unaddressed question?
• Does it address an old question in a new
way or with surprising results?
• Does it confirm the results of previous
studies, strengthening the evidence o
showing that a program can be effective
in multiple settings?
• Does it build on past work to show
trends over time?
Reading through the abstract or executive
summary with these questions in mind
can help you evaluate the study’s rele-
vance even before you review the full
publication.
Do the findings make sense?
The abstract or summary will also present
the study’s key “findings” or “results.” Do
they make sense, given what you already
know about the subject? And are they
ooted in the existing body of research? A
scientific report should be properly refer-
enced, with original sources for all factual
statements and data from other research
clearly cited.
But just because a study’s findings chal-
lenge conventional wisdom, they are not
necessarily inco
ect. One function of
esearch is to test common assumptions
and reexamine earlier findings. A study
with unexpected results can be particu-
larly important or newsworthy, as it can
lead to new insights and approaches.
Findings that go against the conventional
wisdom, however, require more careful
evaluation.
Who conducted the research and wrote
the report?
It is important to consider whether the
study results could be influenced by a
esearcher’s conflict of interest. You
cannot always know this just by reading a
eport, but some knowledge of the field
can guide you. Are the authors well
egarded in the scientific community?
What are their professional credentials?
Have they published previously and, if so,
in what journals?
2006 Series, No. 2
Social science research, with its focus on human behaviors, rela-
tionships and social institutions, can be a rich source of material
for journalists, policymakers and program administrators.
Indeed, social science findings have their greatest impact when
they are useful to—and used by—groups such as these, who can
channel research into practice to improve people’s lives.
Studies generally indicate where
the authors work and who
funded their research. Are the
esearchers independent, o
could their work have been
influenced by the company,
government agency or advocacy
group that employed or funded
them? Who might stand to profit
from the findings? Any potential
conflict of interest should be
identified up front. That said,
esearchers have opinions and
eliefs just like everyone else;
good researchers committed to
a political or social agenda can
still conduct unbiased, trustwor-
thy studies that can withstand
independent evaluation,
provided they follow practices
designed to protect the quality
and integrity of research.
Who published the report?
Social science research is often
disseminated through journal
articles. An article published in a
peer-reviewed journal has been
evaluated by experts in the field
to help ensure that it meets high
scientific standards. Each field
has its own hierarchy of jour-
nals; if you are familiar with the
field, you can look to the pres-
tige of the journal as one indica-
tion of a study’s quality. If you
are not sure how a journal ranks,
look on its front pages for a
statement that it is peer-
eviewed and a list of who serves
on its editorial committee o
eview board (if one exists).
Studies from sources other than
journals (including reports that
esearch institutions publish
themselves) may also contain
solid, useful information. Look to
the “acknowledgments” (usually
at the very beginning of a report)
to see if the authors mention
outside sources of input and
advice, such as an expert advi-
sory panel or external reviewers.
With the exception of some
online journals, information on
the Internet is not reviewed as
igorously before being posted,
ut some sites do have a review
process. In general, if an exter-
nal review process is not
mentioned, you should assume
that one does not exist—which
means you will need to be
cautious about accepting the
study’s conclusions.
Did the researcher select an
appropriate group for study?
A social scientist’s work is
about people, either as individ-
uals or as part of a social insti-
tution (for example, a school,
a hospital, a religious group o
a
anch of government).
Although the question motivat-
ing a researcher’s work may be
general (“What does the public
think about abortion?” “At what
age do teenagers begin to have
sex?”), in practical terms a
study often focuses on a subset,
or sample, of the larger popula-
tion. This sample must be
selected carefully to ensure
that the study results are appli-
cable to the relevant general
population.
The selection of the study group
should be described in the
“Methods” section of an article
or report.
Using a representative sample
is the best way to ensure that
findings can be generalized to
all members of the target popu-
lation. If the researcher uses a
epresentative sample, the
eport will typically state this
specifically. There are many
ways to achieve a representa-
tive sample, and selecting a
true random sample is only one
of them. Other common
approaches are acceptable
and—with appropriate statisti-
cal adjustments for weight-
ing—can produce valid and
epresentative results.
Sometimes, however, a
esearcher may have good
easons to select the target
population in a different way.
Perhaps there is no list of the
general population available.
Perhaps the behavior in ques-
tion is particularly prevalent
among a subgroup, so it makes
the most sense to concentrate
the study among this group in