PROGRAM EVALUATION
Executive Summary
Evaluation Report
Program Components
Outcomes,
Recommendations, and Exemplars STUDENT ACHIEVEMENT
Academic Achievement
Technology
Proficiency
Student Technology
Proficiency Inventory (STPI)
PROFESSIONAL DEVELOPMENT
Professional Development
Teacher Interviews
Technology Survey
District Technology Report
Curriculum Plan
Draft Rubric
Collaborating Agencies
Video Case
Studies
Video Supplemental Evaluation Report
PARENT AND COMMUNITY
Parent and Community
Parent Survey
Community Partners
Parent Technology Workshops
Parent Summative Evaluation
VIDEO SUPPLEMENTAL
VIDEO
CASE STUDIES
|
|
Overview of CER Project Evaluation Activities
Year Three: 2001-2002
The
following is an overview of the major evaluation activities for each component
area:
1.
Student Achievement:
Objective
1.1 Language Arts and Social Studies:
By
June of 2002, Oakland high school graduates who have participated in the
Technology Integration Program for three years will demonstrate statistically
significant (p<.05) pre and posttest gains in district, state and national
performance standards in social studies and language arts as prescribed by NESIC
and the California Frameworks Committees, and/or as measured in district and
state assessments, including but not limited to the state-prescribed
standardized assessment (Stanford Achievement Test 9).
Objective
1.2 Technology Systems:
By June 2002, Oakland high school graduates who have participated in the
Technology Innovation program for three years will demonstrate literacy and
proficiency in the use of technological systems, operations, communications,
research resources, problem-solving and decision-making tools as measured by
pre- and post- program Likert-style questionnaires, projects produced, and a
limited number of interviews and demonstrations.
Activities
|
Products
|
Timelines
|
 |
Access to the district QSP database
|
|
Creation of quasi-experimental design [i]
|
|
Development of a comparison group of district students
|
|
Analysis of SAT/9 scores for 2002 and high school exit exam
|
|
Matched
score (pre/post) analysis of changes for UD students; comparison of UD
students with other non project students in the district
|
July
– August 2002
|
|
Creation
of a student portfolio system
|
|
Collections
of student work as evidence of teachers’ best practices
|
This
is made available to the project.
|
|
Collection of student work during teacher interviews and videotaping
|
|
Collections
of student work as evidence of teachers’ best practices
|
Winter
and spring 2002
|
|
Development and administration of an exiting student survey of
technology skills for graduating students
|
|
Creation of quasi-experimental design [ii]
|
|
Development of a comparison group of district students
|
|
Analysis of student survey of technology skill |
|
Matched
score (pre/post) analysis of changes for UD students; comparison of UD
students with other non project students in the district
|
Spring
2002
|
|
Objective
2.1 Curriculum Development:
By June 2001 and annually
thereafter, Oakland teachers who have participated in the Technology
Integration Professional Development Program for at least two years will
produce interdisciplinary, multicultural curricula reflecting the cultures
and languages of Oakland’s student body through the themes of human and
civil rights.
Objective
2.2 Staff Development II:
By June 2002, 100% of social studies, language arts teachers and other
staff members, as appropriate, working in project high schools will have
completed year I staff education activities delineated in the
“Technology Integration Professional Development Program.”
Objective
2.3 Staff Development I: By June 2001 and annually thereafter, each cohort of Oakland teachers
who have participated in the Technology Integration Professional
Development Program for at least two years will demonstrate augmented
content knowledge in the disciplines of social sciences and literature,
technology uses in the classroom, and constructivist pedagogy as assessed
through pre-, post- and ongoing professional development surveys, and
through participant-developed curriculum units, thematic lessons and
demonstration lessons.
|
Activities
|
Products
|
Timelines
|
|
Interviews
and survey with cohort 1 and 2 teachers (stipend provided) – we will
make modifications of the previous year’s survey
|
Interim
reports on teacher knowledge, attitudes and behaviors
|
Interviews
beginning in December and continuing through February
|
|
Develop
an assessment rubric for teacher developed
Curricula
plans/subsequent product emerging from the expository writing workshops
|
Assessment
rubric
|
Ongoing
|
|
Review
of curricula and instructional materials developed by participating
teachers (use with video case studies)
|
Collection
of curriculum and instructional materials that illustrate promising
practices
|
Ongoing
|
|
Pre/post
assessment of teachers in the supplemental grant video program
|
Measurement
of the impact of supplemental grant video trainings
|
Baseline
taken prior to video training and in the spring of 2002
|
|
Pre/post
assessment of students in the supplemental grant video program
|
Measurement
of the impact of supplemental grant video trainings
|
Baseline
taken prior to video training and in the spring of 2002
|
|
Interactive
video studies
|
Demonstration
how UD professional development activities impact what occurs in the
classroom
|
Beginning
in November of 2001 and ongoing
|
|
Pre/post assessment of project sponsored workshops
|
Measurement
of the participant attitudes and perceived benefit of project sponsored
workshops
|
During
summer institute trainings
|
|
Objective
3 .1 Parent Involvement: By
June 2001, 75% of program participants' parents and/or guardians will
participate in at least one parent activity appropriate to their parenting
situation.
Objective
3.2 Community Engagement:
By June 2002, members from collaborating Institutes for Higher Education (IHEs)
and identified community partners in business and/or social service
agencies will participate in at least one school-community activity
appropriate to improvement of instructional service delivery to students.
Objective
4.1 Equal Access to Technological Instructional Tools:
By June 2004, every high school student in Oakland Unified School District
will have consistent access to computer technology as instructional and
learning tools in their social studies and language arts classes and
classrooms.
|
Activities
|
Products
|
Timelines
|
| Structured
interviews with project staff and collaborating partners
|
Interview
transcriptions and follow-up summaries
|
Ongoing
|
| Administration
to principals of the STaR school-based technology survey
|
Analysis
of system changes at the school level
|
Fall
2001 and summer 2002
|
| Analysis
of community-based technology trainings
|
Pre
and post assessments and summative evaluation summaries
|
Spring
2002
|
| Follow-up
telephone interviews with cohort 2 families who have received refurbished
computers.
|
Report
on family usage of computers
|
November
2001 through January 2002
|
Other
evaluation activities include: 1) assistance with the annual TICG performance
report database; 2) direct contacts with teachers and project collaborators; 3)
planning and development of interim review in Washington; 4) participation in
Western Cluster meetings;
[i]
A quasi-experimental comparison design is appropriately used for estimating the
impact of partial impact programs like Urban Dreams which is unable to randomly
place students in treatment and control groups. Because of the past difficulty accessing student level data,
the project has settled for an ex post design; the comparison group is being
developed after the start of the program. The project evaluators are identifying
those variables that may represent significant differences between the
intervention and comparison group. The
evaluators are creating a statistical representation of the overall
relationships among the variables through the use of a multivariate statistical
design.
[ii] Given that it is not
within the evaluators’ power to randomly place students in to treatment or
control groups (let alone politically or ethically acceptable), the evaluators
have chosen to use a quasi-experimental comparison design when comparing student
SAT/9 and graduation test scores. A quasi-experimental comparison design is
appropriately used for estimating the impact of partial impact programs like
Urban Dreams which is unable to randomly place students in treatment and control
groups. Because of the past
difficulty accessing student level data, the project has settled for an ex post
design; the comparison group is being developed after the start of the program.
The project evaluators are identifying those variables that may represent
significant differences between the intervention and comparison group.
The evaluators are creating a statistical representation of the overall
relationships among the variables through the use of a multivariate statistical
design.
A quasi-experimental comparison
design is appropriately used for estimating the impact of partial impact
programs like Urban Dreams which is unable to randomly place students in
treatment and control groups. Because of the past difficulty accessing student level data,
the project has settled for an ex post design; the comparison group is being
developed after the start of the program. The project evaluators are identifying
those variables that may represent significant differences between the
intervention and comparison group. The
evaluators are creating a statistical representation of the overall
relationships among the variables through the use of a multivariate statistical
design.
Given that it is not within the
evaluators’ power to randomly place students in to treatment or control groups
(let alone politically or ethically acceptable), the evaluators have chosen to
use a quasi-experimental comparison design when comparing student SAT/9 and
graduation test scores. A
quasi-experimental comparison design is appropriately used for estimating the
impact of partial impact programs like Urban Dreams which is unable to randomly
place students in treatment and control groups.
Because of the past difficulty accessing student level data, the project
has settled for an ex post design; the comparison group is being developed after
the start of the program. The project evaluators are identifying those variables
that may represent significant differences between the intervention and
comparison group. The evaluators
are creating a statistical representation of the overall relationships among the
variables through the use of a multivariate statistical design.
Index
© Copyright 2002 Center for Evaluation and Research, LL
|
