Oakland Unified School District

n

Overview of Project Evaluation Activities 2001-2002

The following is an overview of the major evaluation activities for each component area:

	Student Achievement
	Professional Development
	Technology Access and Parent Involvement
	Copy of Preliminary Plan (Fall 2001)

1. Student Achievement:

Objective 1.1 Language Arts and Social Studies: By June of 2002, Oakland high school graduates who have participated in the Technology Integration Program for three years will demonstrate statistically significant (p<.05) pre and posttest gains in district, state and national performance standards in social studies and language arts as prescribed by NESIC and the California Frameworks Committees, and/or as measured in district and state assessments, including but not limited to the state-prescribed standardized assessment (Stanford Achievement Test 9).

Objective 1.2 Technology Systems: By June 2002, Oakland high school graduates who have participated in the Technology Innovation program for three years will demonstrate literacy and proficiency in the use of technological systems, operations, communications, research resources, problem-solving and decision-making tools as measured by pre- and post- program Likert-style questionnaires, projects produced, and a limited number of interviews and demonstrations.

Activities

Products

Timelines

	Access to the district QSP database
	Creation of quasi-experimental design [i]
	Development of a comparison group of district students
	Analysis of SAT/9 scores for 2002 and high school exit exam

Matched score (pre/post) analysis of changes for UD students; comparison of UD students with other non project students in the district

July – August 2002

Creation of a student portfolio system

Collections of student work as evidence of teachers’ best practices

This is made available to the project.

Collection of student work during teacher interviews and videotaping

Collections of student work as evidence of teachers’ best practices

Winter and spring 2002

	Development and administration of an exiting student survey of technology skills for graduating students
	Creation of quasi-experimental design [ii]
	Development of a comparison group of district students
	Analysis of student survey of technology skill

Matched score (pre/post) analysis of changes for UD students; comparison of UD students with other non project students in the district

Spring 2002

2. Professional Development:

Objective 2.1 Curriculum Development: By June 2001 and annually thereafter, Oakland teachers who have participated in the Technology Integration Professional Development Program for at least two years will produce interdisciplinary, multicultural curricula reflecting the cultures and languages of Oakland’s student body through the themes of human and civil rights. Objective 2.2 Staff Development II: By June 2002, 100% of social studies, language arts teachers and other staff members, as appropriate, working in project high schools will have completed year I staff education activities delineated in the “Technology Integration Professional Development Program.” Objective 2.3 Staff Development I: By June 2001 and annually thereafter, each cohort of Oakland teachers who have participated in the Technology Integration Professional Development Program for at least two years will demonstrate augmented content knowledge in the disciplines of social sciences and literature, technology uses in the classroom, and constructivist pedagogy as assessed through pre-, post- and ongoing professional development surveys, and through participant-developed curriculum units, thematic lessons and demonstration lessons.
Activities	Products	Timelines
Interviews and survey with cohort 1 and 2 teachers (stipend provided) – we will make modifications of the previous year’s survey	Interim reports on teacher knowledge, attitudes and behaviors	Interviews beginning in December and continuing through February
Develop an assessment rubric for teacher developed Curricula plans/subsequent product emerging from the expository writing workshops	Assessment rubric	Ongoing
Review of curricula and instructional materials developed by participating teachers (use with video case studies)	Collection of curriculum and instructional materials that illustrate promising practices	Ongoing
Pre/post assessment of teachers in the supplemental grant video program	Measurement of the impact of supplemental grant video trainings	Baseline taken prior to video training and in the spring of 2002
Pre/post assessment of students in the supplemental grant video program	Measurement of the impact of supplemental grant video trainings	Baseline taken prior to video training and in the spring of 2002
Interactive video studies	Demonstration how UD professional development activities impact what occurs in the classroom	Beginning in November of 2001 and ongoing
Pre/post assessment of project sponsored workshops	Measurement of the participant attitudes and perceived benefit of project sponsored workshops	During summer institute trainings

3. Technology Access and 4. Parent Involvement:

Objective 3 .1 Parent Involvement: By June 2001, 75% of program participants' parents and/or guardians will participate in at least one parent activity appropriate to their parenting situation.

Objective 3.2 Community Engagement: By June 2002, members from collaborating Institutes for Higher Education (IHEs) and identified community partners in business and/or social service agencies will participate in at least one school-community activity appropriate to improvement of instructional service delivery to students.

Objective 4.1 Equal Access to Technological Instructional Tools: By June 2004, every high school student in Oakland Unified School District will have consistent access to computer technology as instructional and learning tools in their social studies and language arts classes and classrooms.

Activities	Products	Timelines
Structured interviews with project staff and collaborating partners	Interview transcriptions and follow-up summaries	Ongoing
Administration to principals of the STaR school-based technology survey	Analysis of system changes at the school level	Fall 2001 and summer 2002
Analysis of community-based technology trainings	Pre and post assessments and summative evaluation summaries	Spring 2002
Follow-up telephone interviews with cohort 2 families who have received refurbished computers.	Report on family usage of computers	November 2001 through January 2002

Other evaluation activities include: 1) assistance with the annual TICG performance report database; 2) direct contacts with teachers and project collaborators; 3) planning and development of interim review in Washington; 4) participation in Western Cluster meetings;

[i] A quasi-experimental comparison design is appropriately used for estimating the impact of partial impact programs like Urban Dreams which is unable to randomly place students in treatment and control groups. Because of the past difficulty accessing student level data, the project has settled for an ex post design; the comparison group is being developed after the start of the program. The project evaluators are identifying those variables that may represent significant differences between the intervention and comparison group. The evaluators are creating a statistical representation of the overall relationships among the variables through the use of a multivariate statistical design.

[ii] Given that it is not within the evaluators’ power to randomly place students in to treatment or control groups (let alone politically or ethically acceptable), the evaluators have chosen to use a quasi-experimental comparison design when comparing student SAT/9 and graduation test scores. A quasi-experimental comparison design is appropriately used for estimating the impact of partial impact programs like Urban Dreams which is unable to randomly place students in treatment and control groups. Because of the past difficulty accessing student level data, the project has settled for an ex post design; the comparison group is being developed after the start of the program. The project evaluators are identifying those variables that may represent significant differences between the intervention and comparison group. The evaluators are creating a statistical representation of the overall relationships among the variables through the use of a multivariate statistical design.

A quasi-experimental comparison design is appropriately used for estimating the impact of partial impact programs like Urban Dreams which is unable to randomly place students in treatment and control groups. Because of the past difficulty accessing student level data, the project has settled for an ex post design; the comparison group is being developed after the start of the program. The project evaluators are identifying those variables that may represent significant differences between the intervention and comparison group. The evaluators are creating a statistical representation of the overall relationships among the variables through the use of a multivariate statistical design.

Given that it is not within the evaluators’ power to randomly place students in to treatment or control groups (let alone politically or ethically acceptable), the evaluators have chosen to use a quasi-experimental comparison design when comparing student SAT/9 and graduation test scores. A quasi-experimental comparison design is appropriately used for estimating the impact of partial impact programs like Urban Dreams which is unable to randomly place students in treatment and control groups. Because of the past difficulty accessing student level data, the project has settled for an ex post design; the comparison group is being developed after the start of the program. The project evaluators are identifying those variables that may represent significant differences between the intervention and comparison group. The evaluators are creating a statistical representation of the overall relationships among the variables through the use of a multivariate statistical design.