Standard 4: Program Impact
Standard 4
The provider demonstrates the impact of its completers on P-12 student learning and development, classroom instruction, and schools, and the satisfaction of its completers with the relevance and effectiveness of their preparation.
The Purdue University Northwest (PNW) Educator Preparation Program (EPP) demonstrates the impact of its completers on P-12 student learning and development, classroom instruction, and schools, and the satisfaction of its completers with the relevance and effectiveness of their preparation in several ways. In preparation for this self-study, the EPP undertook efforts to design and implement a research project that resulted in a case study. The Application of Educators’ Knowledge, Skills and Dispositions to Impact Student Learning: A Case Study of an Educator Preparation Program represents efforts by the EPP to use multidimensional measures to determine the impact of the EPP on P-12 learning and development. This study, in concert with resources provided by the Indiana Department of Education (i.e., Federal Accountability Rating of Schools Participating in Case Study Table; State Accountability Rating of Schools Participating in Case Study Table; Statewide Average of Ratings of Teacher Effectiveness; EPP Comparative and Performance Data– Effectiveness Ratings; EPP Comparative and Performance Data– Principal Survey for PNW Completers; and EPP Comparative and Performance Data: Teacher Survey for PNW Completers), indicates that completers positively impact P-12 learning and development, are considered to be effective in the classroom, employers and completers are satisfied/highly satisfied with the preparation they have received.
Case Study
Making instructional decisions is key to providing education that positively impacts the learning and development of students. Therefore, many Educator Preparation Programs (EPPs) focus on providing candidates with opportunities where they develop instructional decision-making skills concerning what content and processes to teach and how to best organize and deliver content in the most effective ways possible. Inherent in this process is an educator’s ability to analyze the learning occurring within their classroom, and to reflect on their role in the learning process. The Application of Educators’ Knowledge, Skills and Dispositions to Impact Student Learning: A Case Study of an Educator Preparation Program demonstrates the impact of completers of one EPP on the learning of P-12 students in their classrooms. Specifically, it investigated completers’ use of instructional-decision making, analysis of student learning, and reflection and self-evaluation. Participants were asked to: 1) engage in a Focus Group Discussion led by one of the researchers; 2) be observed teaching in their classroom (the observation was scored by researchers using the Skills of Teaching Observation Tool (STOT)); and, 3) supply their previous year’s Indiana Teacher Evaluation Tool as well as student performance data on Indiana statewide assessments of student performance (ILEARN, IAM, IREAD-3, and ISTEP+). Additionally, input from students in completers’ classrooms was gathered using the Praxis Performance Assessment for Teachers – Student Surveys (PPAT). Further, measures reported by the Indiana Department of Education were used: Principal Survey, Teacher Survey, and Effectiveness Rating.
Across all measures, evidence indicates that completers of the EPP are able to identify approaches and pedagogical moves necessary to create engaged learning environments. Principals evaluated the completers to be either “effective” or “highly effective” in planning and providing instruction; indicating that their classroom environments are safe spaces where learners understand what is expected of them. The data also demonstrate that completers felt prepared to use observation and other informal and formal assessments to understand the learners in their classroom. Principals overwhelmingly indicated that completers were prepared to analyze data to improve instruction within their classrooms. These novice educators further asserted that they were prepared to understand how students learn. However, the completers indicated that opportunities to learn more about state testing and how to alleviate the pressure of these assessments for learners would have been helpful.
When considering the roles of reflection and self-evaluation in the teaching and learning process, completers indicated that they were prepared to meet the challenges of classrooms by their preparation program. They further asserted the import and value of professional development in their ability to be effective educators. Even though principals assessed completers as being “effective” or “highly effective” in providing inclusive, rigorous, and challenging instruction, completers indicated a desire to have spent more time in classrooms working with a variety of learners (i.e., learners with exceptionalities, multilingual learners) prior to having their own classroom.
Throughout the data, completers emphasized the value of relationships, working one-on-one with learners, and providing a variety of learning experiences for students in their classrooms. Their ability to connect content to relevant learning tasks and design activities for learners to engage with the content from various perspectives was observed. Principals indicated that they were “satisfied” or “very satisfied” in the training completers received from the EPP.
Overall, the data from the case study support that completers perceive and are perceived to be making instructional decisions that positively impact the learning in their classrooms. They indicate and are perceived to use analysis of student data in this decision making process. Their understanding of content and pedagogical content knowledge is seen in their classroom practices, the expectations of performance they communicate to learners, and learner performance on assessments.
Federal Accountability System
In concert with the case study (The Application of Educators’ Knowledge, Skills and Dispositions to Impact Student Learning: A Case Study of an Educator Preparation Program), the EPP examined information reported by the state of Indiana Department of Education for the schools in which completers who participated in the Case Study were employed. Developed by the state, the Federal Accountability System is an effort to improve student achievement, increase student success, and assure the state’s compliance with the Every Student Succeed Act (ESSA). This system provides information about student achievement of statewide goals using indicators that are fundamental measures of school performance and success (i.e., student achievement, student growth, and attendance; “Closing the Gaps” (elementary and middle schools); graduation, strength of diploma, and English learner performance (high schools). While these indicators operate at the schoollevel, they provide an indication of the impact on student learning for PNW completers working within these environments. A school’s performance on each indicator is measured and reflected by a designation of exceeds expectations, meets expectations, approaches expectations or does not meet expectations. An overall score for the school is derived through a combination of student performance on all indicators.
Each indicator encompasses a broad array of information that is used to determine a school’s designation on it. For example, measures of students’ ability to meet or exceed grade-level standards and expectations along with their participation in statewide assessments is used to establish a school’s designation on the student achievement indicator. This indicator is reported for Academic Achievement in English/Language Arts and Academic Achievement in Mathematics. The student growth indicator measures students’ ability to become or maintain proficiency on English/Language Arts and Mathematics academic standards within a four year period of time through the use of student growth percentiles which are used to determine adequate growth annually for both English/Language Arts and Mathematics. Additionally, the student attendance indicator measures whether students are considered “model attendees” (i.e., a student who is in attendance for at least 96% of enrolled time) or “improving attendees” (i.e., students who have improved attendance by at least three percentage points from the prior school year). The “Closing the Gaps” indicator used for elementary and middle schools ensures that attention is paid to the performance of students who fall within the lowest performing 25% of students at a school in relation to the state’ long term goal for student growth proficiency. It is reported in both English/Language Arts and Mathematics. For High Schools, additional indicators are used in determining the federal rating (graduation, strength of diploma, and English learner performance). The graduation indicator measures whether students successfully complete all requirements for graduation and includes an indicator of diploma strength (i.e., Indiana’s Core 40 diploma designation or higher). The strength of diploma indicator measures the rigor of the completed high school program by students. Finally, the English language proficiency indicator measures whether students learning the English language are on target to develop or attain English language proficiency within six years. For this measure, student growth percentiles are used to determine whether students are making adequate growth annually to meet these targets. Again, these indicators operate at the school-level and offer some indication of the impact on student learning for PNW completers working within these environments.
The Federal Accountability Rating of Schools Participating in Case Study Table provides information for schools where completers who participated in the case study work. Of the five schools included in this analysis, two are designated as meeting expectations, two are approaching expectations, and one does not meet expectations. Areas of strengths in the elementary and middle schools are found in the indicators of Academic Achievement in the English/Language Arts and the “Closing the Gap- English Language/Arts” where all schools meet expectations. The majority of schools (75%) also were determined to meet expectations in the Student Growth in English/Language Arts, the “Closing the Gap-Mathematics,” and attendance indicators. Indicators where only half of the schools meet expectations are in Academic Achievement in Mathematics and Student Growth in Mathematics.
State Accountability Rating System
Similarly, the state of Indiana has also created a state accountability rating system. This system assigns grades to schools using a five-letter system (i.e., A, B, C, D, or F; where A is the highest and F is the lowest). A school’s grade is determined by a combination of scores on the determined indicators (i.e., student proficiency, student growth overall, student growth of lowest performing 25% of students in English Language Arts, student growth of highest performing 75% of students in English Language Arts, student growth of lowest performing 25% of students in Mathematics, student growth of highest performing 75% of students in Mathematics, and graduation and college and career readiness). The state accountability rating uses a broad array of information when calculating scores for each indicator. Specifically, the student proficiency indicator measures student proficiency of grade-level standards in both English/Language Arts and Mathematics at the School. The student growth indicator measures student growth towards proficiency at the school; each student receives points based on the type of growth demonstrated (i.e., low growth, typical growth, high growth) and points are assigned using a growth to proficiency table. On this indicator, it is possible for a school to achieve a score higher than 100. Finally, the graduation and college and career ready indicator measures students completion of the Indiana graduation requirements within four and five years at the school, and student credentialing indicating postsecondary readiness.
With this in mind, the State Accountability of Schools Participating in Case StudyTable provides additional information for schools where completers who participated in the case study work. The five schools analyzed received grades of A, B, or C from the state. Further, these grades appear to be relatively stable (i.e., they are the same as the previous year’s rating). When examining the data a bit closer, an area of strength for these schools is found in the indicator, English Language Arts: Lowest 25% of students, where all schools received a score of greater than 100 points. Other areas of strength are found in the indicators of Student Growth and Mathematics: Lowest 25% of students where 80% of the schools received scores greater than 100 points. Indicators where schools performed the lowest were English/Language Arts: Highest 75% of students, Mathematics: Highest 75% of students, and Student Proficiency. Forty percent of schools received scores of 100 points or higher in English/Language Arts: Highest 75% of students and 20% of schools received scores of 100 points or higher in Mathematics: Highest 75% of students. In terms of Student Proficiency, the majority of schools were rated between 62 and 78 points, with one school rated at 17. From this analysis, it is clear that schools in which completers work demonstrate that they are effective and having a positive impact on the learning and development of students.
State Comparative and Performance Data
To further support the assertion that PNW completers are effective teachers, the EPP examined comparative and performance data provided by the state of Indiana for its completers. Specifically, two tables were examined: Statewide Average of Ratings of Teacher Effectiveness; EPP Comparative and Performance Data and Effectiveness Ratings. The Statewide Average of Ratings of Teacher Effectiveness provides efficacy ratings (i.e., effective or highly effective) for novice educators as measured by their annual performance evaluations at one-, two-, and three-years into their professional practice. As indicated on this table, the statewide average of novice educators deemed to be effective or highly effective is 96%. Completers from PNW are rated similarly, with 96% of completers being rated as effective or highly effective (EPP Comparative and Performance Data and Effectiveness Ratings). Further, this table indicates that as PNW completers continue in the profession the number rated as highly effective increases (i.e., 16% year one, 24% year two, and 37% year three).
Principal Survey Data
Principal Survey data provided by the state indicate that 94% of principals are satisfied or highly satisfied with the training and preparation that completers from PNW received (Comparative and Performance Data–Principal Survey for PNW Completers). When examining responses related to the domain of knowledge, 100% of principals indicated that completers are prepared to adhere to ethical and legal requirements of the teaching profession; and 96% indicated that completers understand how students learn and meet expectations for content preparation and knowledge. When examining principals’ perceptions of completer’s pedagogical preparation, 100% indicated that completers are prepared to analyze assessment data to improve instruction and that they are able to integrate technological tools to advance student learning. Further, 98% indicated that completers are prepared to provide an inclusive learning environment and work effectively with exceptional learners. Of the respondents, 96% indicated completers are prepared to provide appropriate and challenging learning experiences, and to use a variety of assessment methods to guide and improve instruction. When considering completers’ preparation for providing a rigorous learning environment, the development of content specific assessments, and ability to differentiate instruction, 94% indicated that they are “satisfied” or “very satisfied.” The lowest area of rating in pedagogical preparation of completers on the survey is found in their ability to use effective strategies to manage the learning environment with only 92% of respondents indicating that “satisfied” or “very satisfied” ratings. An examination of principals’ responses related to the domain of professional disposition preparation indicates that 100% of respondents were “satisfied” or “very satisfied” with completers’ preparation to openly accept suggestions/constructive feedback and to exhibit ethical practice. The responses show that 98% of principals indicate that completers are prepared to work effectively with school leaders and within the school culture, while 96% indicate that they are prepared to work effectively with other professionals. The lowest ratings in dispositional preparation was found in completers’ ability to work effectively with parents/guardians with 94% of respondents indicating “satisfied” or “very satisfied.”
When viewed in aggregate, data from the EPP Comparative and Performance Data: Principal Survey for PNW Completers indicate that building administrators perceive completers to be applying the knowledge, skills, and dispositions they learned in their preparation program. It also indicates that principals see completers exhibit these understandings ethically and in adherence with the legal requirements of teaching; that completers analyze student assessment data to improve classroom instruction; that they integrate technological tools to advance student learning; and that they are open and accepting of suggestions/constructive feedback. While no area in the data indicates perceived levels of performance that are concerning, it does appear that consideration for how candidates might be afforded more opportunities to develop and practice effective strategies for creating productive learning environments and to work effectively with parents/guardians should occur.
Teacher Survey Data
An examination of the Teacher Survey data provided by the state indicate that 92%of completers indicate that their preparation program at PNW was either “good” or “excellent (Comparative and Performance Data– Teacher Survey for PNW Completers). When examining responses related to the domain of knowledge, 97% of respondents indicate that they felt prepared to adhere to the ethical requirements of the teaching profession; 95% indicate that they were prepared to adhere to the legal requirements. 94% of completers indicate that they were prepared to understand how students learn and that they recognized the importance of continued professional development. Only 92% felt that they were prepared to meet expectations for content preparation and knowledge. When examining completers’ perceptions of their pedagogical preparation, 95% indicate that they were prepared to provide inclusive learning environments and to work collaboratively with school leaders and/or colleagues to create a safe and positive learning environment. Further, 94% indicate that they were prepared to differentiate instruction to meet the needs of all learners; 92% indicate that they felt prepared to work effectively with exceptional learners, to develop quality assessments to assess learners’ understandings of lessons, and to provide a rigorous learning environment; and 91%indicate that they were prepared to provide appropriate and challenging learning experiences. The lowest areas of rating in pedagogical preparation is found in their perceived preparedness in the use of appropriate strategies to effectively manage learning experiences and the integration of technological tools to advance student learning with only 89% of respondents indicating that they were “satisfied” or “very satisfied;”while, 88% indicated that they felt prepared to analyze student assessment data to improve classroom instruction. An examination of completers’ responses in the domain of professional disposition preparation indicates that 97% of respondents were “satisfied” or “very satisfied” with their preparation to openly accept suggestions/constructive feedback, to exhibit ethical practice, and to work effectively with other professionals. Of the respondents, 94% of completers indicated that they felt prepared to work effectively within the school culture, while 92% indicated that they were prepared to work effectively with school leaders. The lowest ratings in dispositional preparation is found in completers perceived ability to work effectively with parents/guardians with 88% of respondents indicating “satisfied” or “very satisfied.” When viewed in aggregate, data from the EPP Comparative and Performance Data: Teacher Survey for PNW Completers indicate that completers perceive that they are applying the knowledge, skills, and dispositions they learned in their preparation program. It also indicates that completers exhibit these understandings ethically; that they provide inclusive learning environments; that they work collaboratively with school leaders and/or colleagues to promote safe and positive learning environments; that they are open and accepting of suggestions/constructive feedback; and that they feel that they were prepared to work effectively with other professionals. Data from the Teacher Survey and the Principal Survey demonstrates some alignment between the two and that no area in the data indicates perceived levels of preparedness that are concerning. However, as revealed in the Principal Survey, it does appear that consideration for how candidates might be afforded more opportunities to develop and practice effective strategies for creating productive learning environments and ones in which they are able to work effectively with parents/guardians should occur.
Efforts to Improve Impact on Student Learning
Towards this end, the EPP is piloting a year-long residency experience during the 2020 spring semester. The year-long experience, called the Professional Year, places candidates in classrooms for extended periods of time (three-consecutive days the first semester and five-consecutive days the final semester). This extended period of time seeks to address completers’ desires for more time in classrooms and has the opportunity for candidates to learn more about statewide assessments and tools for mitigating the pressure students feel during this time. Additionally, it provides opportunities that address concerns raised by providing more opportunities for candidates to develop and practice effective strategies for creating productive learning environments where they learn to work effectively with parents/guardians. Further, the EPP has employed the Site Tracker for the EPP (STEPP), a new field experience plan and monitoring system to ensure that candidates are placed in a variety of grade levels, within diverse school settings (e.g., rural, urban, suburban; ethnic, socioeconomic, linguistic, etc.). This new system, STEPP, has formalized placement processes, created a streamlined process for candidate placement that involves interviewing, and seeks to provide a multitude of learning opportunities from which candidates are able to learn and grow.