1
Enhancing Education Through Technology
Mandated Impact Evaluation
Evaluation of the Effectiveness of Educational Technology Interventions
Today nearly every school and a rapidly growing number of classrooms have computers and
internet connections, and student-to-computer ratios are reaching levels that permit sustained
instructional use of computers. Like most areas of education, there is a greater need for
more rigorous research using the scientific method in order to determine the effectiveness
of these tools in improving instruction and student achievement. There is some research
correlating teachers’ technology skills and their use of technology in classroom instruction with
higher academic achievement, but most researchers agree that a small minority of teachers use
technology as part of their instruction in academic subjects with sufficient frequency or skill to
improve student achievement.
After years of significant investments made in educational technology, policy makers and budget
decision makers are demanding evidence that this investment is improving student achievement
and instruction. As a result, rigorous evaluations are more important than ever before. Not only
to provide policy makers with the information they need to make decisions, but also for
practitioners to guide their practices.
Major Purposes of Evaluation and Key Evaluation Questions
The use of educational technology will be the focus, rather than the availability of technology,
since it is the integration of technology with academic instruction that has the potential to impact
achievement—not the presence of computers, for example, in the classroom. Technology should
not be valued for what it is, but rather for what it does. The evaluation will assess the impact of
using selected educational technology interventions that are intended to improve student
academic achievement in reading and/or in mathematics. The evaluation will address two key
questions:
1) Does the use of educational technology improve student achievement in reading
and/or mathematics?
This question will be addressed through an experimental methodology in which schools, teachers,
and/or students are randomly assigned to selected educational technology interventions. The
difference in reading or mathematics achievement between the group using the selected
educational technologies and the group not using those technologies will be the measure of the
impact of the technology on student achievement. The level of impact expected to be detected in
this evaluation is a 0.35 effect size with the rationale being that smaller effect sizes will not
substantially improve the achievement of students as expected by NCLB.
2) Under what conditions and practices does the use of educational technology improve
student achievement in reading and/or mathematics?
2
This question will be addressed by a correlational analysis in which the conditions in the school
and classroom as well as teacher practice will be measured. Those measures will be correlated
with the impacts of the technologies on student achievement.
Evaluation Design
The design of this evaluation is based largely on input received through a task order awarded to
Mathematica Policy Research in September 2002 (ED01CO0039, Task Order 4). The purpose of
this task order was to solicit the advice of experts in educational technology and evaluation to
design a rigorous evaluation of the effectiveness of educational interventions designed to improve
student achievement.
One of the major steps in designing an evaluation of the impact of educational technology is
selecting and defining the use of educational technology interventions to be evaluated. In order to
replicate a technology intervention, the intervention needs to be clearly delineated. Since the time
lines for this evaluation do not allow for development, the interventions to be evaluated will
consist of existing, specific educational interventions that can be replicated in multiple locations.
The interventions to be evaluated will be selected on a competitive basis under the above
referenced task order. Developers and publishers of educational technology interventions can
apply to participate in the evaluation. The process for selection of educational technology
interventions to be evaluated will be competitive. The type of interventions that will be
considered will be broad including, for example, software for direct instruction, curricula with an
emphasis on use of specific technology, or assessment systems for tracking student progress.
This competition will be held in late summer of 2003. Educational technology interventions will
be selected based on criteria including, but not limited to:
• Conceptual rationale and empirical evidence supporting the use of the intervention in
improving reading or mathematics achievement;
• Ability of the developer or publisher to support replication of the intervention in multiple
schools in multiple locations across the country;
• Ability of the evaluation to group the interventions into classes based on grade level and
type of intervention for reading and/or mathematics.
Neither specific educational technology interventions nor the number of interventions to be
evaluated will be known until fall 2003. The evaluation will group educational technology
interventions into classes of similar interventions at the same grade levels in reading and/or
mathematics. The impact of classes of interventions relative to a control group will be reported,
not the impact of any specific educational technology intervention relative to a control group or
another intervention.
For example, depending on the applications received from educational technology developers and
publishers, the evaluation could assess the impact of a class of technology interventions used to
improve decoding skills for first and second graders, the impact of a class of technologies used to
improve reading comprehension for fourth and fifth graders, the impact of a class of technologies
used to teach early numeracy skills to first and second graders, the impact of a class of
technologies to teach proportionality to fourth and fifth graders, and/or the impact of a class of
technologies for teaching algebra to eighth. Each class could have two or more specific
interventions. The sample sizes given in this scope of work assume that four classes of
interventions will be evaluated in reading and four in mathematics.
3
There are numerous evaluation issues that remain to be resolved after the educational technology
interventions have been selected including:
• Number of specific interventions and classes of interventions to be replicated overall and
in each district and school. Information on the application process for educational
technology developers and vendors will be available in mid August, 2003. Information
on the effectiveness of their educational technologies is an incentives for developers and
vendors to participate in the evaluation. The effectiveness of classes of technologies will
be reported by the evaluation, not for any specific educational technology intervention.
The data collected by the evaluation on the implementation and effectiveness of each
technology educational intervention will be provided to the developers and vendors of
that intervention for their use.
• Recruitment of districts and schools to implement the educational technology
interventions and to participate in the evaluation. Since this is not a study of a federal
program (although Title II(D) may be used to support such uses of educational
technology), there is not a direct link to federal requirements to participate in program
evaluations. In addition to the value of evidence on effectiveness of the educational
technology interventions, incentives for districts and schools to participate in the
evaluation include access to educational technologies and professional development for
teachers to use those technologies.
• Level of random assignment. Distinct interventions that are not available to all students
at a given grade level in a school enhance feasibility of randomly assigning students to a
treatment or control group within a school. Random assignment of students is
statistically more efficient than random assignment at the school level. If interventions
are to be made available to all students within a grade, due to the nature of the
intervention or negotiations with school districts to gain their participation in the study,
random assignment will need to take place at the school level which would require
relatively more schools than random assignment at the student level.
While there are numerous issues yet to be resolved, these are the parameters of the evaluation
design as of August 2003. This description will be updated periodically when additional
information becomes available.

Extracted via my personal research. because I undergo the true definition and importance of "Enhancing Education Through Technology". Thanks
From: Adewumi Tope Bunmi