Student Academic Achievement

Component 1a: Student Academic Achievement

The ultimate goal of Urban Dreams is improved student achievement.  The project components are designed to contribute to academic gains.  The project objectives call for measurable student achievement in core academic areas by the end of the 2001-2002 academic year.

The project evaluators and staff instituted a more rigorous “quasi-experimental” design this year to meet the demands of the No Child Left Behind statute and to better understand the impact of the project on students within Urban Dreams’ classrooms related specifically to academic achievement and technology proficiency.  To accomplish this, the evaluators developed representative samples of Urban Dreams and non-Urban Dreams students.  For the purpose of this experimental study, being treated was operationally defined as having taken one or more classes during either or both the current or past school year (2000-2002) from at least one teacher who was associated with the UD program. 

A survey instrument was administered to students who attended one of six sites where teachers had the opportunity to participate in the Urban Dreams program.  For each school site, a list of the language arts and social studies teachers was developed.  Stratified sampling resulted in the random selection of six teachers (one from each site – 24 teachers total) for each of the following four groups:   a language arts teacher involved in the Urban Dreams program, a social studies teacher involved in the Urban Dreams program, a language arts teacher who was not involved in Urban Dreams, and a social studies teacher who was not involved in Urban Dreams.

The purpose of the following analysis is to summarize performance on standardized tests (specifically, the SAT/9 and STAR tests) administered during spring 2002 to students at high school sites where teachers participating in the Urban Dreams program work.  The test scores for these students on the Stanford Achievement Test, Version 9 (SAT/9) and California State Standards Test (STAR) were retrieved for students in both the “experimental” and “comparison” groups from the records kept by the school district.

Ninety-six percent of the students completed the question on the survey that allowed classification into the “experimental” vs. “comparison” group.  For the purpose of this experimental study, being treated was operationally defined as having taken one or more classes during either or both the current or past school year (2000-2002) from at least one teacher who was associated with the program.  The comparison group consisted of the 23% of the sample who were students at the same sites but who did not have a UD program teacher within the last two years.

The size of the sample used in each statistical analysis varies reflecting the percentage of students for which data related to each of the dependent variables has been obtained.  Of the 890 students that could be classified as belonging to either the experimental or control group, 90.8% had useable scores reflecting their total number of technology proficiency skills.  In contrast, only 73.9%, 71.3%, 70.9%, 70.6%, and 72.7% of these group-classified students had accessible NCE SAT/9 reading, language, and social studies scores and STAR English language arts and history scores, respectively.

Evaluation Questions

The evaluation questions for student academic achievement are:

Evaluation Question One:

To address evaluation question one, a series of independent samples t-tests were performed using the normal curve equivalent (NCE) scores of each of the three SAT/9 subtests.  In the analysis of performance on the STAR, the levels were compared between the groups using the non-parametric Mann-Whitney U test since the data represent ordinal measurement (i.e, lacking the property of equal widths between successive levels as needed for t-tests.).  For all 5 analyses, group membership (experimental versus comparison) served as the independent variable. 

For each of the five standardized test scores analyzed, students in the experimental group, on average, performed higher than those in the comparison group (see table below).  It should be kept in mind, however, that these statistically significant differences may, in part, be explained by initial group differences in background factors having little, if anything, to do with the Urban Dreams program itself.  Thus, greater attention should be paid to the results of evaluation question four.

It should be noted that for approximately 30% of the students who were classifiable as belonging to the experimental or comparison group, no standardized test score data was available.  Thus, the samples sizes used in the analyses ranged from a low of 628 to a high of 658.  (The degrees of freedom indicated in each statistical summary reflect a statistical adjustment that is made when the homogeneity of variance assumption is not met, as true in these instances.) 

Standardized SAT/9 achievement test score performance by group (experimental vs. comparison)

Level on STAR English language arts test by group membership

Level on STAR history test by group membership