II

Year Two

Methods of Evaluation

	Local Evaluation Activities
	Local Evaluation Plan
	Discrepancy Evaluation Model
	Selected Instruments: Appropriateness, Reliability and Validity

Local Evaluation Activities

The Oakland Unified School District contracted with the Center for Evaluation and Research (C.E.R.) from the outset of the project to facilitate the evaluation of the Urban Dreams Project. C.E.R. coordinated local evaluation efforts to furnish process and summative information to the project staff with the goal of validating successful practices and providing for informed decision-making. Four C.E.R. evaluators currently work on the project along with a four person support staff. Several C.E.R. staff members participated in national evaluation and Western Cluster meetings over the last two years.

The evaluation of the Urban Dreams project is a cooperative undertaking by district personnel, partner agencies, and C.E.R. staff. Together, staff members from these agencies have been involved in most of the evaluation activities including the development of annual local evaluation plans and the project’s discrepancy evaluation model.

Local Evaluation Plan

	Go to Logic Model
	Go to Plan for Year Three

The local evaluation plan consists of an outline of general evaluation activities that are to be accomplished each year. The plan serves as a general guide that directs evaluation efforts. The following is an overview of the major evaluation activities for each component area:

Staff Development: Urban Dreams is providing ongoing professional development for over 100 teachers in the areas of technology, language arts and history. Evaluation activities include: a) surveys and interviews with teachers, b) analysis of curricula and instructional materials developed by the participating teachers, c) the creation of preliminary video case studies, and d) analysis of project sponsored workshops.

Student Achievement: Student achievement is principal goal of Urban Dreams. The evaluation is involved in tracking and analyzing students participating in Urban Dreams’ classrooms at all nine participating high schools.

Technology Access: The project is providing technology directly to classrooms and homes. The evaluation includes: a) structured interviews with project staff and collaborating partners providing these services, and b) analysis of community-based technology trainings.

Parent Involvement: Urban Dreams has delivered over 500 refurbished computers and accompanying technology training to families of students in classrooms with teachers who are participating in the Urban Dreams project through the Computers for Families Program. The evaluation includes: a) structured interviews with project staff and collaborating partners providing these services, b) analysis of community-based technology trainings, and c) follow-up telephone interviews with families who have received refurbished computers.

Discrepancy Evaluation Model

Evaluators and project staff developed a discrepancy evaluation model (DEM) that outlined in more detail each of the evaluation activities. The DEM serves as an on-going narrative of the project’s activities related to its original goals and objectives, and serves as a roadmap that evaluators use to analyze the status of the process and summative evaluations.

The DEM created by project staff and the evaluation team includes program narratives, component maps and input, process, and output (IPO) statements for each program component on two increasingly detailed levels. The plan is currently available on the evaluation website that is maintained by C.E.R. (http://ns1.californiaschools.net/~ud). The plan is periodically updated to reflect not only original resources, goals (outputs) and processes, but to reflect discrepancies in intended outputs (which can be positive or negative in relationship to the original intents). The plan helps to detail the nature of those discrepancies and other formative data, thus insuring that project staff’s decisions for change are grounded in valid research.

DEM Level II

Selected Instruments: Appropriateness, Reliability and Validity

The project evaluation utilized a variety of instruments to gather information on program processes and impacts. Data collection methods included surveys, workshop evaluation forms, telephone questionnaires, observation protocols, one-to-one interviews and focus groups. Evaluators, project staff and teachers collect the data.

During the 1999-2000 academic year, a survey was created to determine changes in skills and attitudes of participating teachers. The teacher surveys were made available on the evaluators’ Urban Dreams evaluation web site for respondent input. The evaluators also made scannable versions of the same surveys. Every high school English and social studies teacher was asked to complete a teacher survey delivered to their classrooms by evaluators. In addition, students in first and second period social studies were asked to complete a student survey. Evaluators made survey results available online. The results of this survey were used ultimately to direct the project’s activities.

During this past year (2000-2001), the evaluators did a follow-up survey and held interviews with several of the Cohort 1 teachers. The evaluators made direct contact with teachers and provided stipends to enhance the participation in onsite interviews. The goal of this data collection effort was to determine how teachers were integrating their new resources within their instructional programs.

Other measures that are used in the evaluation are the:

	Stanford Achievement Test (SAT/9), California’s mandatory state assessment system, includes reading, language arts and social studies sub-tests. Analysis will include: a) simple pre-post two tale test of significance of matched scores; and b) comparison of participating students with a similar population of students in the district using analysis of variance (ANOVA) analytic techniques;
	Video case studies of how teachers are integrating technology into their content areas;
	Teacher and parent participation records and workshop evaluation forms; and
	Observations, interviews and focus groups with project staff, teachers, parents and collaborating agency personnel.

Another major focus of the evaluation has been the collection and analysis of student data. Over a ten-month period, the evaluators worked with district’s technical staff to obtain access to district demographic and test data. The evaluators hired on contract a district employee to work on an overtime basis to provide some of the requested information. As a result, the evaluators now have much of the requested information on students who attended Cohort 1 teacher classrooms during the past year.

	Staff Development: Urban Dreams is providing ongoing professional development for over 100 teachers in the areas of technology, language arts and history. Evaluation activities include: a) surveys and interviews with teachers, b) analysis of curricula and instructional materials developed by the participating teachers, c) the creation of preliminary video case studies, and d) analysis of project sponsored workshops.
	Student Achievement: Student achievement is principal goal of Urban Dreams. The evaluation is involved in tracking and analyzing students participating in Urban Dreams’ classrooms at all nine participating high schools.
	Technology Access: The project is providing technology directly to classrooms and homes. The evaluation includes: a) structured interviews with project staff and collaborating partners providing these services, and b) analysis of community-based technology trainings.
	Parent Involvement: Urban Dreams has delivered over 500 refurbished computers and accompanying technology training to families of students in classrooms with teachers who are participating in the Urban Dreams project through the Computers for Families Program. The evaluation includes: a) structured interviews with project staff and collaborating partners providing these services, b) analysis of community-based technology trainings, and c) follow-up telephone interviews with families who have received refurbished computers.