The Science Center has always believed that FirstHand has a positive impact on Philadelphia students. As a science-focused organization, though, we wanted to know – does our assumption hold up under objective, external evaluation?
The Education Innovation and Research grant we received from the U.S. Department of Education for FirstHand has allowed us to explore this question in partnership with AnLar LLC, a community-minded organization specializing in understanding how programs like FirstHand impact students’ science learning and feelings about science, and CUNY’s Lehman College.
The team has been conducting a rigorous evaluation aimed at understanding how participating in FirstHand benefits students’ achievement and feelings about science and to identify areas of strength and growth, ensuring FirstHand continues to maximize its positive impact.
The design incorporates three tiers of evaluation to achieve these goals:
- Formative evaluation to facilitate an understanding of the structure of the program and inform its continuous improvement
- Implementation evaluation to explore the key components of FirstHand
- A summative evaluation to examine the impact of the program on students
While we’re only entering year three of a five-year study, the evaluation team’s second year-over-year progress report has arrived, and the interim findings are promising. Here’s what their research – comprised of numerous student, mentor, teacher, and guardian interviews, observational assessments, and program surveys - uncovered.
Hands-on STEM Learning in the FirstHand Lab
While both approaches have their place in education, almost anyone will tell you that they find hands-on learning more engaging than sitting in a lecture. FirstHand students learn almost entirely through hands-on experiences, and they notice the differences between their science class at school and the interactive environment provided by FirstHand.
One student highlighted this by saying: “It’s like an actual lab where you get to do actual stuff. In science class, yeah, I have fun, but here, I get to move around and experiment with things, which is really exciting.”
This hands-on approach not only makes science more engaging but also helps students feel more connected to the subject than they might in a traditional classroom setting. In fact, FirstHand’s differentiation from the typical classroom was a common theme in student feedback.
“Here (FirstHand), you get to use more tools that we don't use in our science class like microscopes and stuff,” one student said.
These sentiments held when the evaluation team spoke with school leaders. One principal shared, “FirstHand is a window to the world for my students. The program provides exposure to something beyond their norm, encouraging the children to wonder about what could be.”
Echoing these sentiments, a coordinator at another school shared, “The opportunity for students to learn in a new atmosphere and step away from the traditional methods excited the students and helped to create a better culture among the grade."
Building STEM Competency
Not surprisingly, one of the main goals of FirstHand is to make students comfortable with – and therefore more competent in – STEM. Does the data actually support this as an outcome, though?
So far, the answer seems to be yes. By asking teachers to rate students’ STEM competencies (both students involved with FirstHand and control students who have yet to participate in FirstHand), it becomes clear that students taking part in FirstHand were more likely to have their competencies rated as “exceptional” compared to the control group, which had more competencies rated as “poor” or “N/A.”
One teacher from John F. McCloskey Elementary school expressed pride in witnessing the students develop their knowledge and also overcome their fears when it came to presenting their projects. “It’s always good to see what they’ve learned and the knowledge that they picked up, they usually are really nervous but they know their topics and they know how to present them well, and [with] the quiet ones it’s good to see them shine.”
Belonging in Science and Valuing Science
A student’s beliefs about science are half the battle when it comes to encouraging STEM participation. The adage “if you can’t see it, you can’t be it” holds true here and is one of the reasons that FirstHand encourages students to see themselves as scientists. FirstHand students build relationships with STEM professionals called “mentors,” which helps them understand what it takes to succeed in a STEM career. Students also expressed in the interview process that being able to see mentors who looked like them (such as a black woman) helped set them at ease and feel like they could be in her role one day. One student even said that meeting their mentor “made me feel more open-minded because I looked at their faces and saw why they wanted to be a scientist.”
Student surveys demonstrate that participants feel a greater sense of belonging in science after participating in FirstHand; they were also more likely to report seeing value in science in their everyday lives.
And this was backed by numerous student interviews which revealed that participating in FirstHand challenged the assumptions many students had about science: “FirstHand made me think about a lot of things, like, especially with bioplastic. But it’s like now, maybe I do want to become a scientist,” mused one student.
“I thought I really hated science until I came here, I feel like the only reason why I hated it was because we didn’t get to do stuff like this,” explained another.
What Families Observed
Encouragement from family members is critical when it comes to nurturing a lifelong interest in STEM. FirstHand student showcases have been a way for students to share what they’re doing with their family members, ranging from younger siblings, to parents, to grandparents.
Some family members homed in on their child’s attitude toward FirstHand.
One mother said, “She talked about it a lot. To be honest [student name] doesn’t smile like that at home. I’ve never seen her smile so much. I’m like wow, she’s like totally different when she’s here because she seems 100% happy here. She could be herself here versus in school, or her neighborhood. So this brings out the best in her.”
While others noted a difference in growth and development: “I saw it from her schoolwork from last year to this year, she really did improve a lot. We will be signing her up for the summertime.”
Pride was a common theme amongst parents finally seeing what the students had been working on: “Proud, proud. He was telling me we can come down here because I didn’t quite understand what he was doing now to see face to face, just a proud parent.”
The Lasting Impact
"It changed how I feel about science jobs by showing me that becoming a scientist may not be easy but it is fun to put your mind to it and just get into it,” one student noted after participating in FirstHand.
These interim findings affirm that FirstHand is indeed making a positive impact on students' science journeys, fostering a deeper engagement and a sense of belonging in the STEM field. By continuing to refine the program based on formative insights, FirstHand can strengthen its role in inspiring the next generation of scientists. The journey of improvement and adaptation is ongoing, ensuring that FirstHand remains a transformative experience for all students.
