UD - 4th Year Evaluation

FOURTH YEAR FINAL EVALUATION REPORT (Acrobat Reader)
Final Report
Standardized Test Performance of Students Report
Technology Proficiency of Students Report
CTAP Teacher Survey
CTAP Teacher Technology Use Survey
Complete Parent Training Survey Results

4TH YEAR EVALUATION
2002-2003 Evaluation Plan Executive Summary
Introduction
Evaluation Methods
Results
Recommendations and Exemplars

STUDENT ACHIEVEMENT
Cluster Meeting Powerpoint

AERA
AERA Research
AERA Powerpoint

CALIFORNIA TECHNOLOGY ASSISTANCE PROJECT (CTAP)
CTAP Teacher Survey Results

TECHNOLOGY ACCESS and PARENT INVOLVEMENT
Parent Winter Survey
Computer Training Workshops

Urban Dreams Project Evaluation Site

Overview of Project Evaluation Activities

Year Four: 2002-2003

Oakland Unified School District’s Urban Dreams Technology Challenge Grant

Download Fourth Year Reports - Require Acrobat Reader

	Oakland Unified School District Technology Innovation Challenge Grant - Final Report
	The Impact of Participating in the Urban Dreams Technology Innovation Challenge Grant Program on the Standardized Test Performance of Students - Standardized Test Performance of Students Report
	The Impact of Participating in the Urban Dreams Technology Innovation Challenge Grant Program on Technology Proficiency of Students - Technology Proficiency of Students Report
	CTAP Technology Proficiency Survey of Urban Dreams and Non-Urban Dreams English and Social Studies Teachers - CTAP Teacher Technology Survey Results
	Teacher Technology Use Survey of Urban Dreams and Non-Urban Dreams English and Social Studies Teachers - CTAP Teacher Technology Use Survey
	MAFEI Parent Computer Training Survey Results (2002-2003)

Overview of Project Evaluation Activities

	Student Achievement
	Professional Development
	Technology Access and Parent Involvement

The following is an overview of the major evaluation activities for each component area:

1. Student Achievement:

Objective 1.1 Language Arts and Social Studies: By June of 2002, Oakland high school graduates who have participated in the Technology Integration Program for three years will demonstrate statistically significant (p<.05) pre and posttest gains in district, state and national performance standards in social studies and language arts as prescribed by NESIC and the California Frameworks Committees, and/or as measured in district and state assessments, including but not limited to the state-prescribed standardized assessment (Stanford Achievement Test 9).

Objective 1.2 Technology Systems: By June 2002, Oakland high school graduates who have participated in the Technology Innovation program for three years will demonstrate literacy and proficiency in the use of technological systems, operations, communications, research resources, problem-solving and decision-making tools as measured by pre- and post- program Likert-style questionnaires, projects produced, and a limited number of interviews and demonstrations.

Activities	Products	Timelines
Access to the district QSP database in August with the 2002 SAT-9 results and high school exit exam results Creation of quasi-experimental design[i] Development of a comparison group of district students based on teacher participation in UD professional development Analysis of SAT/9 scores for 2002 and high school exit exam results	Matched score (pre/post) analysis of changes for UD students; comparison of UD students with other non project students in the district Comparison of student results looking at both teacher integration (self-reported on teacher survey)	July – September 2002 and 2003
· Dissemination of student portfolio system – provide information to UD teachers to see if any teachers are willing to field test the portfolio system – choose 2 to 3 teachers with stipends to field test the system	Collections of student work as evidence of teachers’ best practices	This is made available to the project. We can let teachers know that the portfolio system exists. If there is interest in its use than the project can provide workshops on its use.
Collection of student work during teacher interviews and videotaping	Collections of student work as evidence of teachers’ best practices	Winter and spring 2002 and 2003
Development and administration of an student survey of technology skills for graduating students Creation of quasi-experimental design[ii] Development of a comparison group of district students Analysis of student survey of technology skills	Matched score (pre/post) analysis of changes for UD students; comparison of UD students with other non project students in the district	Spring - summer 2002 and 2003

2. Professional Development:

Objective 2.1 Curriculum Development: By June 2001 and annually thereafter, Oakland teachers who have participated in the Technology Integration Professional Development Program for at least two years will produce interdisciplinary, multicultural curricula reflecting the cultures and languages of Oakland’s student body through the themes of human and civil rights.

Objective 2.2 Staff Development II: By June 2002, 100% of social studies, language arts teachers and other staff members, as appropriate, working in project high schools will have completed year I staff education activities delineated in the “Technology Integration Professional Development Program.”

Objective 2.3 Staff Development I: By June 2001 and annually thereafter, each cohort of Oakland teachers who have participated in the Technology Integration Professional Development Program for at least two years will demonstrate augmented content knowledge in the disciplines of social sciences and literature, technology uses in the classroom, and constructivist pedagogy as assessed through pre-, post- and ongoing professional development surveys, and through participant-developed curriculum units, thematic lessons and demonstration lessons.


Activities	Products	Timelines
Interviews and survey with cohort 1,2 and 3 teachers (stipend provided) – we will make modifications of the previous year’s survey – we will also make a comparison to non-UD teachers through representing sampling	Interim reports on teacher knowledge, attitudes and behaviors Matched score (pre/post) analysis of changes for UD teachers; comparison of UD students with other non project teachers in the district	Interviews beginning in December 2002 and continuing through February 2003
Development and implementation of an assessment rubric for teacher developed curricula plans/subsequent product emerging from the expository writing workshops	Assessment rubric and quality curricula posted on project website	Ongoing
Review of curricula and instructional materials developed by participating teachers	Collection of curriculum and instructional materials that illustrate promising practices	Ongoing
Follow-up interviews with teachers in the 2001-2002 supplemental grant video program	Measurement of the impact of supplemental grant video trainings	Baseline taken prior to video training and in the spring of 2002; follow-up interviews in the spring of 2003
Pre/post assessment of project sponsored workshops	Measurement of the participant attitudes and perceived benefit of project sponsored workshops	During summer institute trainings

3. Technology Access and 4. Parent Involvement:

Objective 3 .1 Parent Involvement: By June 2001, 75% of program participants' parents and/or guardians will participate in at least one parent activity appropriate to their parenting situation.

Objective 3.2 Community Engagement: By June 2002, members from collaborating Institutes for Higher Education (IHEs) and identified community partners in business and/or social service agencies will participate in at least one school-community activity appropriate to improvement of instructional service delivery to students.

Objective 4.1 Equal Access to Technological Instructional Tools: By June 2004, every high school student in Oakland Unified School District will have consistent access to computer technology as instructional and learning tools in their social studies and language arts classes and classrooms.

Activities	Products	Timelines
Structured interviews with project staff and collaborating partners	Interview transcriptions and follow-up summaries	Ongoing
Administration to principals of the STaR school-based technology survey	Analysis of system changes at the school level	Summer 2002, fall 2002 and spring 2003
Analysis of community-based technology trainings	Pre and post assessments and summative evaluation summaries	Spring 2002 and 2003
Follow-up telephone interviews with families who have received refurbished computers.	Report on family usage of computers	Winter 2003

Other evaluation activities include: 1) assistance with the annual TICG performance report database; 2) direct contacts with teachers and project collaborators; 3) planning and development of interim review in Washington; 4) participation in Western Cluster meetings;

[i] A quasi-experimental comparison design is appropriately used for estimating the impact of partial impact programs like Urban Dreams which is unable to randomly place students in treatment and control groups. Because of the past difficulty accessing student level data, the project has settled for an ex post design; the comparison group is being developed after the start of the program. The project evaluators are identifying those variables that may represent significant differences between the intervention and comparison group. The evaluators are creating a statistical representation of the overall relationships among the variables through the use of a multivariate statistical design.

[ii] Given that it is not within the evaluators’ power to randomly place students in to treatment or control groups (let alone politically or ethically acceptable), the evaluators have chosen to use a quasi-experimental comparison design when comparing student SAT/9 and graduation test scores. A quasi-experimental comparison design is appropriately used for estimating the impact of partial impact programs like Urban Dreams which is unable to randomly place students in treatment and control groups. Because of the past difficulty accessing student level data, the project has settled for an ex post design; the comparison group is being developed after the start of the program. The project evaluators are identifying those variables that may represent significant differences between the intervention and comparison group. The evaluators are creating a statistical representation of the overall relationships among the variables through the use of a multivariate statistical design.

http://californiaschools.net/ud/