Student Technology Proficiency

Component 1b: Student Technology Proficiency

Evaluation Questions

The evaluation questions addressed under Component 1b are:

Evaluation Question One:  

To address evaluation question one, an independent samples t-test was performed using the total number of skills (potentially ranging from 0 to 21) reported by each student as the dependent variable and group membership (experimental versus comparison) as the independent variable.

The average number of technology skills that students in the experimental group indicated they had was 15.56 (SD= 4.42) whereas the average for the comparison group was 13.72 (SD= 4.74).  The mean difference of 1.84, unadjusted for background factors that differ between the groups, was statistically significant, t(279.272)= -4.684, p< .001.  A 95% confidence interval for the difference between the two population means suggests that the experimental group’s population mean lies between 1.07 to 2.61 skills higher than that of the comparison group.  Allowing for rounding, this difference is a matter of at most 3 more skills, on average, for those who have taken a course with a UD teacher.  It should be kept in mind that this statistically significant difference may, in part, be explained by initial group differences in background factors having little, if anything, to do with the UD program itself.  Thus, greater attention should be paid to the results of evaluation question three.

Evaluation Question Two:  

To address evaluation question two, differences between the experimental and comparison groups in terms of gender, ethnicity, having plans to attend college, having a home computer, having taken a technology class, and belief in the importance of computer skills were investigated with chi square tests of independence (also known as tests of association and tests of homogeneity).  The proportion of experimental group students who belong to each category (e.g., male versus female) is compared to the proportion of comparison group students who do.  Also, in addressing evaluation question two, differences between the experimental and comparison groups in terms of grade level and student-reported GPA were investigated with independent samples t-tests. The purpose of this evaluation question was to gauge whether the experimental and comparison groups varied in systematic ways that might threaten the internal validity of the study. 

The two groups were found to differ with respect to gender, grade level, ethnicity, having taken a technology class, and plans for college but to not differ with respect to having a computer at home, their beliefs about the importance of computer skills for their future, and typical course grades.  Specifically, the experimental group had a larger proportion of students who were females, who had taken a technology course, who planned to go to college, who were sophomores, and who were African American.  In contrast, the comparison group had a larger proportion of males, of freshmen, of Asian and Hispanic students.  (In gauging program effects, an effort is made to control for group differences by employing hierarchical regression.)  

Evaluation Question Three:  

To address evaluation question three, hierarchical regression analysis was employed where blocks of variables are entered successively and those in prior blocks are controlled when examining the effects of variables entering in later blocks.

Students in the treatment group (i.e., whose teachers participated in the project) did report significantly more technology proficiency skills than the students in the comparison group after controlling for demographic, academic achievement/ aspiration, and computer-specific background variables.  Though statistically significant, the change in the proportion of variance for the outcome (self-reported proficiency level) was only 1%.  It should be recognized that this estimate of program impact is conservative in that we control for computer-specific variables that the Urban Dreams program could, in fact, have impacted (e.g., the acquisition of a home computer, the decision to take a computer class, beliefs in the importance of having computer skills).  The regression results for the model outlined above with program treatment dichotomously indicated are shown in Table 7 below. 

Hierarchical regression results

Evaluation Question Four:  

To address evaluation question four, the proportions of each group who reported “yes” to the having the competency were obtained though a cross-tabulation of group membership (experimental versus comparison) and response to the individual survey item (numbers 4 through 24, separately) and a chi square test of independence was performed.

In addressing this question, items appearing as number 4 through 24 on the survey are considered.  We find that a higher proportion of those in the experimental group report having the individual competencies (listed as separate items) than the proportion of those in the comparison group who do.  This is true for all but two of the 21 items (see Table 8, next page).  For item #10 ( i.e., “I am honest, polite and respectful when I email and chat.”) 82% of the comparison responded “yes,” whereas 78% of the experimental group did.  Also, for item #14 (i.e., “I use chat or instant messaging often.”) 63% of the comparison group responded “yes,” whereas 57% of the experimental group did.  However, when chi square tests of independence were performed, neither of these two unanticipated differences, where the comparison group’s proportion exceeded that of the experimental group’s, was found to be statistically significant (p > .05).  In contrast the differences where the proportion of the experimental group exceeded that of the comparison group were found to all be statistically significant (p < .05) except for items 12, “I send e-mail to friends or family frequently,” and 13, “I am able to send e-mail attachments.”  It should be kept in mind that the statistically significant differences may, in part, be explained by initial group differences in background factors having little, if anything, to do with the Urban Dreams program itself. Thus, greater attention should be paid to the results of evaluation question five.  

Evaluation Question Five:  

To address evaluation question five and six, a strategy analogous to addressing evaluation question three was employed but the type was a logistic (rather than linear) regression due to the binary (rather than continuous) outcomes involved.

The competencies (i.e., survey items) are grouped in Table 8 on the basis of the second use listed above:  To explain the variation among student responses to a particular competency (i.e., marking “yes” or “no”), does it help to know whether students are in the experimental versus comparison group if you already know their status on each of the background variables?  If so, this would suggest that being in the experimental group (i.e., having been a student in a course with a teacher involved in the UD program) may have helped develop that particular competency.  This seems plausible with respect to items 4, 6, 9, 11, 16, 17, 18, 21, and 24.  There is insufficient evidence, however, to suggest that program participation impacted competencies listed as items 5, 7, 8, 12, 13, 15, 19,20, 22, 23, and 14.  (As noted earlier regarding item #14, a smaller proportion of the experimental group reported using chat or instant messaging often.)  In addition, we unexpectedly find that after controlling for group differences in background factors, the comparison group is more likely to respond that they are, “honest, polite and respectful when I email and chat” (Item #10; p = .028).

Differences between (unadjusted) proportions of program participants versus non-participants who report having the competency identified in individual survey item numbers 4 through 24 and odds ratios from logistic regression controlling for background variables^a