Status
of Program Components
The following
narrative addresses the project’s progress in meeting each of its four program
components during the third year of implementation.
Component
1a: Student Academic Achievement
The evaluation questions for student academic
achievement are:
-
Do students (in the “experimental group”) who were enrolled in at
least one course taught by a teacher who participated in the Urban Dreams
program, on average, perform better on the SAT/9 subtests (reading, language
arts, and social studies) and STAR subtests (English language arts and history)
than students who were not taught by such teachers (the “comparison” group)?
-
What is the correlation between program participation and standardized
test scores?
-
Do students who perform better on the SAT/9 and STAR subtests also
self-report higher levels of technology proficiency?
-
Is
there a statistically significant difference between the experimental
and comparison groups’ standardized test performance after controlling for
background factors (i.e., those that are not attributed to program impact)
between groups that might influence the attainment of technology competencies
(an intermediate outcome hypothesized to impact achievement) and/or represent a
selection threat that gives one group an initial advantage with regard to
standardized test performance?
The ultimate goal of
Urban Dreams is improved student achievement.
The project components are designed to contribute to academic gains.
The project objectives call for measurable student achievement in core
academic areas by the end of the 2001-2002 academic year.
The project evaluators and staff instituted a more
rigorous “quasi-experimental” design this year to meet the demands of the No
Child Left Behind statute and to better understand the impact of the project on
students within Urban Dreams’ classrooms related specifically to academic
achievement and technology proficiency. To
accomplish this, the evaluators developed representative samples of Urban Dreams
and non-Urban Dreams students. For
the purpose of this experimental study, being treated was operationally defined
as having taken one or more classes during either or both the current or past
school year (2000-2002) from at least one teacher who was associated with the UD
program.
A survey instrument (attached), described in detail
under Component 1b (see page 22), was administered to students who attended one
of six sites where teachers had the opportunity to participate in the Urban
Dreams program. For each school
site, a list of the language arts and social studies teachers was developed.
Stratified sampling resulted in the random selection of six teachers (one
from each site – 24 teachers total) for each of the following four groups:
-
a language arts teacher involved in the Urban Dreams
program
-
a social studies teacher involved in the Urban Dreams
program
-
a
language arts teacher who was not involved in Urban Dreams
-
a social studies
teacher who was not involved in Urban Dreams
During the 2001-2002
academic year, the staff and evaluators collected data on the Stanford
Achievement Test-Ninth Edition (SAT/9) and STAR state proficiencies in English
and social studies for a randomly selected cohort of Urban Dreams students and
non-Urban Dreams students. This
data demonstrates that students who participated in Urban Dreams classrooms
scored significantly higher on both assessments than students who were not in
Urban Dreams classroom. The
following report was presented in 2003 to the American Educational Research
Association (AERA) in Chicago.
AERA
Report
Objectives
or Purposes
The
purpose of this study was to examine evidence related to the reliability and
validity of scores derived from the Student Technology Proficiency Inventory (STPI),
a new instrument intended for use with a population known to include high
numbers of disadvantaged and limited English-speaking students in need of access
to technology who had participated in Urban Dreams (referred to as UD), a
Technology Innovation Challenge Grants (TICG) Program.
It includes evidence about the usefulness of the score inferences within
the context of evaluating the impact of UD on the technology proficiency of
students. The purpose of the paper
session/ discussion is to present and receive feedback regarding our approach to
developing the inventory’s items and the results of the validation study.
Perspective(s)
or Theoretical Framework
Oakland
Unified School District (OUSD) aimed to improve high school students' academic
achievement in social studies and language arts through the system-wide
implementation of UD an innovative, standards-based and technology-embedded
reform program. One
of the Project’s major goals related to student achievement reads, “Students
will increasingly use educational technology for learning in core academic
subjects…. [Those] who have participated in the Technology Innovation program
for three years will demonstrate literacy and proficiency in the use of
technological systems, operations, communications, research resources,
problem-solving and decision-making tools.”
The
logic model for UD (available at: http://californiaschools.net/ud
) shows the overarching goals to be increased student achievement and technology
proficiency. Thus, a measure of
students’ proficiency with technology that yields valid scores for this
population and is practical to administer was needed in order to evaluate
program impact.
The
project evaluators conducted a web search to examine assessments designed to
measure student technology proficiency based on the National Educational
Technology Standards for Students (NETS-S) published by the International
Society for Technology in Education (ISTE, 2000). The
search did not generate student technology skills assessments that would meet
the needs of UD. However, several
websites provided curriculum scope and sequence technology plans, benchmarks, or
skills continuums. Many of the
technology skills and proficiencies expected of teachers were applicable to
students. The California Technology
Assistance Project (CTAP) Technology Assessment Profile was examined for the
types of technology skills required by teachers and some of these questions were
adapted for students (http://ctap2.iassessment.org)
in constructing our measure.
In
a recent search for related literature, ERIC Resources and the online database
of proposals to AERA for the 2002 Annual Meeting were consulted (http://edtech.connect.msu.edu/searchaera2002/searchsessions.asp).
Again, measures of teacher technology competence were found (e.g.,
Flowers & Algozzine, 2000; Ropp, 2001).
But, as noted by Toyama and Crawford (2002), “while technology
proficiency standards for students are becoming increasingly common at the state
level… assessing educational technology proficiency in statewide testing
programs is still uncommon” (p.21). Furthermore, they found that information regarding the
technical quality of instruments was not available for the majority of student
technology proficiency instruments they reviewed (p.20).
This highlights the need for the current study in which evidence related
to the reliability and validity of scores derived from the STPI, the newly
constructed instrument, is examined.
Methods,
Techniques, or Modes of Inquiry
This
paper focuses on the quantitative research methodology employed to evaluate the
quality of data produced by the STPI as it relates to evaluating the UD goal of
increasing students’ technology proficiency.
The instrument validation study involved factor analyzing the
inventory’s items, calculating Cronbach’s alpha, a measure of internal
consistency reliability, and calculating various correlation coefficients
between proficiency scores derived from STPI and indicators hypothesized to
relate to technology proficiency.
Validation
also involves evidence pertaining to the usefulness of score inferences (AERA/
APA/ NCME, 1999). Like Flowers and
Algozzine (2000), we include the results of an experimental study to determine
whether the STPI is sensitive to “UD,” the technology-embedded reform
program intervention. To examine
the impact of UD on students’ self-reported technology proficiency, students
who were or were not taught by teachers involved in UD were compared. In addition, the number of UD teachers who the student had
taken during the prior two school years (“treatment exposure level”) served
as a predictor of STPI scores.
Hierarchical
regression analysis was employed where blocks of variables are entered
successively and those in prior blocks are controlled when examining the effects
of variables entering in later blocks (see Table 1, following page).
Table
2. Variable blocks used in
hierarchical regression analysis of program impact.
Data
Sources or Evidence
Selection
of Teachers/Classes:
OUSD serves more than 11,000 urban secondary students in an ethnically
and linguistically diverse community. The
sample of students who responded to the instrument attended one of six sites
where teachers had the opportunity to participate in UD.
For each school site, a list of the language arts and social studies
teachers was made. Stratified
sampling resulted in the random selection of six teachers (one from each site)
for each of the following 4 groups: LA
UD teacher, SS UD teacher, LA non-UD teacher, and SS non-UD teacher (where LA=
language arts, SS= social studies).
Description
of Instrument Being Investigated:
The STPI items were contained within a paper-and-pencil survey that also
asked students to report gender, typical course grades, grade level, ethnic
background, whether they a) have a computer in their home, b) received a free
computer, c) have taken a technology class at school, and d) plan to go to
college. They also indicate which, if any, of the UD teachers they have taken
classes from, whether their teachers encourage the use of the computer for
school assignments, whether they have cooperated with a group of students to
create a class project using computer technology, and whether they believe that
knowing how to use the computer will be important for them in their future.
The 21 technology proficiency items concern (a) basic operations and
concepts; (b) social, ethical and human issues; (c) communication tools; (d)
productivity tools; (e) research tools; and (f) problem-solving and decision
making tools (with 5, 3, 3, 5, 2, and 3 items, respectively).
Description
of Sample:
A
total of 929 high school students responded to the STPI.
The percentage of females was slightly higher than that of the males (54%
vs. 46%). The grade level
distribution was 42% freshmen, 33% sophomores, 17% juniors, and 8% seniors.
The ethnic distribution was 30% African American, 28% Asian, 4%
Caucasian, 26% Hispanic, 7% Multiethnic (or a qualified response), and 5%
“other.”
Description
of “Experimental” and “Comparison” Groups:
Ninety-six percent of the students completed the question on the survey
which allowed classification into the “experimental” vs. “comparison”
group. For the purpose of this part
of the experimental study, being treated was operationally
defined as having taken one or more classes during either or both the current or
past school year (2000-2002) from at least one teacher who was associated with
UD. The comparison group consisted of the 23% of the sample who were
students at the same sites but who did not have an UD teacher within the last
two years. To gauge whether the
experimental and comparison groups varied in systematic ways that might threaten
the internal validity of the study, a series of t-tests for independent groups
and chi square tests of association were conducted.
Results
and/or Conclusions/ Point of View
To
determine the reasonableness of combining responses to a specific set of items
(namely, numbers 4-24) into a single score intended to reflect students’
technology proficiency, the items were initially subjected to a factor analysis.
Although 4 factors were extracted, the first eigenvalue was nearly four
times as large as second. Also, the
factors that emerged did not align well with the categories articulated in
content standards (ISTE, 2000) on which they were based.
This is not surprising or problematic considering our intent to construct
a quickly administered measure where the number of items written to address each
area was necessarily limited. Our
concern was that no clear competing second factor be present that might weaken
the internal consistency of scores derived from combining all items together.
Cronbach’s
alpha was .86 for the scale comprised of items 4-24 which were completed by 835
students.
Preliminary
evidence of content validity relies upon knowing that the item writer is a
professor of educational technology who relied upon the NETS-S (a standards
framework that is highly regarded).
For
additional evidence of validity, we looked to see whether higher scores were
found among students (a) who had higher school grades, (b) at higher grade
levels, (c) with computers in their homes, (d) who had taken a technology class
at their school, (e) who planned to go to college, and (f) who believed that
knowing how to use the computer would be important for their future.
Weak correlations, but statistically significant and in the predicted
direction (p<.001), were found between the total number of technology skills
the student self-reported and all the indicators listed above (r’s= .25, .20,
.27, .32, .24, and .27, respectively), lending support for the validity of
scores derived from the instrument.
Results
of the experimental study, investigating the usefulness of the STPI for program
evaluation purposes, suggest that students in the treatment group (i.e., whose
teachers participated in UD) did report significantly more technology
proficiency skills than the students in the comparison group after controlling
for demographic, academic achievement/ aspiration, and computer-specific
background variables. Though
statistically significant, the change in the proportion of variance for the
outcome (self-reported proficiency level) was only 1%.
It should be recognized that this estimate of program impact is
conservative in that we control for computer-specific variables that the UD
could, in fact, have impacted (e.g., the acquisition of a home computer, the
decision to take a computer class, beliefs in the importance of having computer
skills). The regression results for the model outlined above with UD
treatment dichotomously indicated are shown in Table 2 below.
Table
3. Hierarchical regression results
Taken
together, the results of this validation study provide evidence that this
quickly administered, checklist-type survey instrument can be usefully employed
to measure self-reported technology proficiency among high school student
populations consisting of high numbers of disadvantaged and limited
English-speaking students.
Although
the validity evidence is moderate, there are three limitations that suggest the
estimates could be stronger if some modifications were introduced. Due to space limitations, they are not discussed in this
proposal but are included, along with suggestions for further research, in the
paper to be presented.
Educational
or Scientific Importance of the Study
This
study contributes to our knowledge of how an assessment of student technology
proficiency may be developed and validated for use within the context of a TICG
program evaluation. Moreover, the
results of the study suggest that significant progress has been made toward
developing a valid measure for this purpose.
The significance of the study can be appreciated in light of the current
status of large-scale technology proficiency assessments as reported by Toyama
and Crawford (2002). They suggest
that technology proficiency assessment is “still in a very early stage” and
that there is a “scarcity of technical quality evidence” which is needed in
order that fair and valid decisions can be reached (p. 21). This study provides such evidence while heeding their advice
that “test developers need to evaluate OTL [opportunity to learn] and
determine the extent to which the instruments are sensitive to the instruction
that students receive in school versus outside of school” (p. 21). Toyama and Crawford explain why there will be increasing
demand for high quality technology proficiency assessment and encourage their
development (p. 22). The need to
address “the lack of appropriate student outcome measures that can capture the
impact of technology use in a broad set of technology-using classrooms” was
underscored in a 2002 AERA symposium according to the proposal, “Evaluating
Technology Impacts: Assessing
Student Technology Outcomes,” that was accepted by Division H (Hamilton,
Hinojosa, Quellmalz, Crawford, Toyama, and Zalles, 2001). This work represents an attempt in addressing this critical
need.
References:
American
Educational Research Association (AERA), American Psychological Association (APA),
& National Council on Measurement in Education (NCME). (1999). Standards for
educational and psychological testing.
Washington, DC: American Educational Research Association.
Flowers,
C. P., & Algozzine, R. F. (2000). Development and validation of scores on
the Basic Technology Competencies for Educators Inventory.
Educational and Psychological Measurement, 60 (3), 411-418.
Hamilton,
E., Hinojosa, T., Quellmalz, E. S., Crawford, V., Toyama, Y., & Zalles, D.
(2001). Evaluating Technology Impacts: Assessing Student Technology Outcomes. (Proposal to Division H
of AERA) [Online] Retrieved July 15, 2002 from http://edtech.connect.msu.edu/searchaera2002/viewproposaltext.asp?propID=5753
International
Society for Technology in Education (2000).
National Educational Technology Standards for Students (NETS-S):
Technology Foundation Standards for All Students. [Online] Retrieved July 17,
2002 from: http://cnets.iste.org/sfors.htm
Ropp,
M. M. (2001). Technology standards:
Too much to learn, too little time.
(Proposal to Division K of AERA) [Online] Retrieved July 15, 2002 from
http://edtech.connect.msu.edu/searchaera2002/viewproposaltext.asp?propID=6850
Toyama,
Y., & Crawford, V. (2002). Assessment
of student proficiency with educational technology: State
of the States’ large-scale technology proficiency assessment.
Paper presented at the Annual Meeting of the American Educational
Research Association, New Orleans, LA, April 2002.
© Copyright 2002 Center for Evaluation and
Research, LL
