Preliminary evaluation of the IDEAS program "Service Learning in Science: Astronomy Camp-Ins" by Caren Oberg, 20 Aug 2000 -------------------------------------------------- Describe the evaluation component of the project. a. Briefly describe the type of evaluation that was conducted. b. Briefly describe the data collection procedures. [Please note that I have included a discussion about the data collection methods used with each component] Three different types of evaluation were used. Component 1 was process oriented. We sought to evaluate the participants comfort level with discussing topics of astronomy with elementary, middle, and high school students, their familiarity with inquiry-based teaching methods, and their familiarity with the Service-Learning philosophy. Results of the questionnaire found that of the 14 teachers and pre-service teachers who took part in the training: 14 of the 14 participants felt their comfort levels with discussing astronomy improved, as long as they were teaching their most comfortable grade level (elementary, middle, or high school). However, these teachers were not later tracked to determine how they used their new knowledge once they returned to their classrooms (as this was not the focus of the program evaluation). 9 of the 14 participants had already had experience with Inquiry Based methodology and could give examples 12 of the 14 participants could define Service Learning. 8 of those 14 associated Service Learning with the student experience. 1 of the 14 associated Service Learning specifically to high school students. Initially, the questionnaire was to serve as a benchmark to compare the experiences of those pre-service teachers who were going to take part in the camp-ins later on. However, it was found that although 7 questionnaires were filled out by pre-service teachers, only four pre-service teachers actually attended camp-ins. [Two additional pre-service teachers would join the program later and attend a separate training that was not evaluated]. It was also found that the training was discussed during the in-depth interviews with the four pre-service teachers. Component 2 was outcome oriented. Using in-depth interviews, all six of the pre-service teachers who participated in the camp-ins were asked to discuss their experiences. The discussion focused on their use of inquiry methodology with the campers, use of astronomy and comfort levels with astronomy and the SpaceLink facility, and the impact of the Service Learning approach. Due to the small numbers, this was a qualitative study. A full-scale evaluation report has been given to MdSCI staff. Component 3 is impact oriented. To be conducted in 6 months, this component will also consist of in-depth interviews. Each of the 6 pre-service teachers will be contacted and asked to reflect further on their experience and how they have applied their experience to their classrooms. A two-page appendix with these results will be forwarded to MdSCI staff in the future. All evaluation tools are included at the end of this report. c. Were the data collection procedures effective in measuring the project's outcome? Component 1: The questionnaire was not as successful as it could have been. As explained above, its original purpose was to serve as a benchmark tool. Had it remained as such, it would have been a successful tool. However, since it later became a tool merely to measure what participants took away from the training, it could have been considerably shorter. Component 2: The in-depth interviews were very successful. Lasting almost 20 minutes in length per interview, the open-ended questions allowed the pre-service teachers to tell their entire experience plus any additional information they wanted to discuss. In depth interviews are far more successful than questionnaires, especially with a very small group, because the interviewer can probe for additional information. Component 3: As above. Additionally, in depth interviews ensure a higher rate of return than mailing a questionnaire. Component 1: Questionnaire Assessment procedures have been established in order to evaluate this program's effectiveness. Evaluation methods will include this survey and two future interviews. This survey is the first step in this assessment process. Please fill this form out completely. Your knowledge and opinions will assist us in providing the best experience. Please answer the following questions: Science/Space Literacy 1. BEFORE THIS TRAINING how comfortable were you in discussing the following space/astronomy topics with the following age groups: Elementary School? (K-5) Hubble Space Telescope Comfortable Somewhat Not Comfortable Space Shuttle Equipment Comfortable Somewhat Not Comfortable Stars Comfortable Somewhat Not Comfortable Middle School/Jr. High ? (6-8) Hubble Space Telescope Comfortable Somewhat Not Comfortable Space Shuttle Equipment Comfortable Somewhat Not Comfortable Stars Comfortable Somewhat Not Comfortable High School ? (9-12) Hubble Space Telescope Comfortable Somewhat Not Comfortable Space Shuttle Equipment Comfortable Somewhat Not Comfortable Stars Comfortable Somewhat Not Comfortable 2. AFTER THIS TRAINING how comfortable are you in discussing the following space/astronomy topics with the following age groups: Elementary school ? (K-5) Hubble Space Telescope Comfortable Somewhat Not Comfortable Space Shuttle Equipment Comfortable Somewhat Not Comfortable Stars Comfortable Somewhat Not Comfortable Middle School/Jr. High ? (6-8) Hubble Space Telescope Comfortable Somewhat Not Comfortable Space Shuttle Equipment Comfortable Somewhat Not Comfortable Stars Comfortable Somewhat Not Comfortable High School ? (9-12) Hubble Space Telescope Comfortable Somewhat Not Comfortable Space Shuttle Equipment Comfortable Somewhat Not Comfortable Stars Comfortable Somewhat Not Comfortable Teaching Methodology 3. Are you familiar with inquiry based teaching (i.e.: teaching through facilitated learning/questioning) (Circle One) Familiar Some-what Not Familiar 4. Have you used inquiry based teaching with students before today? (Circle one) Yes No 5. Describe your experiences with inquiry-based teaching (size of class, topic discussed, planning...) (Use the back of this page for additional space) 6. Describe a time when you had difficulty teaching a family member/friend to do an action or understand a concept. (Use the back of this page for additional space) Did you use knowledge that both of you shared in order to help your family member/friend understand? Explain. 7. BEFORE THIS TRAINING how comfortable were you with using inquiry based teaching with the following age groups: (Circle One) Elementary School (K-5) Comfortable Somewhat Not Comfortable Middle School (6-8) Comfortable Somewhat Not Comfortable High School (9-12) Comfortable Somewhat Not Comfortable 8. AFTER THIS TRAINING how comfortable are you with using inquiry based teaching with the following age groups: (Circle One) Elementary School (K-5) Comfortable Somewhat Not Comfortable Middle School (6-8) Comfortable Somewhat Not Comfortable High School (9-12) Comfortable Somewhat Not Comfortable Service-Learning 9. Please define service-learning in your own words? 10. Give an example of how service-learning can be incorporated into the classroom? 11. How often since, but including, the 12th grade have you participated in a service-learning project? As the teacher/leader (Circle one) 0-1 2-4 5+ As a student (Circle One) 0-1 2-4 5+ 12. Discuss a service-learning project that you have been a part of (either as a teacher/leader or as a student). What did you gain from the experience? 13. BEFORE THIS TRAINING how comfortable were you with using service-learning in the following grades: (Circle One) Elementary School (K-5) Comfortable Somewhat Not Comfortable Middle School (6-8) Comfortable Somewhat Not Comfortable High School (9-12) Comfortable Somewhat Not Comfortable 14. AFTER THIS TRAINING how comfortable are you with using service-learning in the following grades: (Circle One) Elementary School (K-5) Comfortable Somewhat Not Comfortable Middle School (6-8) Comfortable Somewhat Not Comfortable High School (9-12) Comfortable Somewhat Not Comfortable General Information and Goals 15. Why did you choose to participate in this program? 16. Please list three (3) goals you wish to meet during this program? 17. What is your specialty? (Science, History, ESL...) ________________________________________ 18. What grade levels do you teach or intend to teach? (Circle all that apply) K-2 3-5 6-8 9-12 19. What grade levels are you most comfortable teaching? (Circle all that apply) K-2 3-5 6-8 9-12 Thank you for taking the time to fill out this survey. Component 2: In-Depth Interviews Interview Protocols 1. How did you get involoved with the SLS-Camp-in Program? 2. What were your objectives? Program Objective 1: Use of Inquiry-Based Teaching Methods 3. Describe your camp in experience. 4. If you attended more than one camp-in, describes changes from camp-in to camp-in. 5. Describe examples of inquiry based teaching used during the camp-ins. 6. Speculate how you will use inquiry based methodology in your classroom. Program Objective 2: Use of astronomy to teach other areas of science 7. Describe discussions between students and yourself during camp-ins 8. Describe your comfort level with astronomy before and after the training session 9. Describe how used your astronomy knoweldge to assist students during the camp-ins 10. Speculate how you will use astronomy in your classroom, next year, to teach other types of science (or other subjects) Program Objective 3: Service-Learning 11. Can you define Service-Learning? 12. How do camp-ins reflect the Service Learning philosophy? 13. How will you use Service Learning in your classroom next year? General Questions 14. How do you plan to use the camp-in experience in the future? 15. What would you change logistically? 16. Would you recommend this program to next year's class? Why or why not?