A B S T R A C T
The focus of this article is the development and evaluation of an assessment program for measuring senior teachers’ competences in secondary schools. The goals of the developed instrument were measuring senior teachers’ competences and providing the opportunity for self-reflection for the teachers assessed. This instrument was developed and evaluated in four steps: (1) the content of assessment was determined, defined in senior teacher competences; (2) criteria and standards were specified for the assessment ofthe competences; (3) the assessment methods were determined; and (4) the assessment program was evaluated by means of a pilot study. The target group consisted of eight potential senior teachers, who were assessed with the new instrument. In total, eleven teachers and 70 pupils evaluated the new assessment instrument. The assessment seems fit for the purpose. Pupils are positive about the assessment program, whereas the teachers are more sceptic about it.
Introduction
For many years, the quality of education in general and of teachers in particular has been the object of discussion and research. Indeed, teacher quality is important because teachers play a crucial role in realizing the quality of the learning environment (Hattie, 2009) and determine to a great extent the school’s quality (Marzano, 2011). In this respect, Rasmussen and Friche (2011) state that schools experience a pressure to increase and demonstrate the quality oftheir education and teachers. In theNetherlands, this pressure to increase the quality of education in general and of teachers in particular has been addressed by the Teaching Advisory Board of the Dutch government. As a way to increase teacher quality, they advised to create more opportunities for career development and differentiation within the teaching profession. This should increase the attractiveness of the teaching profession and prevent good teachers from leaving schools and choosing other career paths (Teaching Advisory Board, 2007). The Dutch Ministry of Education decided that secondary schools should introduce integral personnel management in order to (1) stimulate teachers’ development; (2) offer opportunities for differentiation in the teacher profession; and (3) raise the quality of Dutch secondary education. It was assumed that the introduction of integral personnel management in secondary education would lead to increased educational quality. It might help putting the best teachers on the most complex tasks and pupil groups, and the possibility to address weak teaching practices (Borko, Whitcomb, & Liston, 2009).
To integrate an effective and fair integral personnel management system, instruments are needed that validly and reliably assess teacher quality (van der Schaaf, Stokking, & Verloop, 2005). At the moment, no specific standardized procedures or guidelines for teacher assessment are available and Dutch secondary schools emphasize those aspects which are important for their particular schools. The common practice is that teachers gain a raise ofsalary each year, simply by having worked a year more as a teacher. In order to effectuate this, one annual dialogue between teacher and management takes place. This can hardly be looked upon as an assessment method for teacher quality. The question then arises whether there are possibilities to assess teacher quality validly. Whereas assessment and development of student teachers has quite often been studied (e.g. Hegender, 2010; Noell & Burns, 2006), summative assessment of teachers working in schools has been studied distinctively less often. Therefore the aim of the current study is to develop and evaluate a summative assessment program for senior teachers in secondary education. Besides this summative function, the assessment program should have a formative function to enable and stimulate teachers to reflect on their own competence development.
Indeed, literature shows different perspectives on how teacher competence is defined and measured, many of these focusing on the effectiveness of teachers in accomplishing high student learning outcomes (e.g. Chen, Mason, Staniszweski, Upton, & Valley, 2011; Mangiante, 2011; Praslova, 2010; Seidel & Shavelson, 2007). These studies rely on the assumption that certain teacher behaviour (den Brok, Brekelmans, & Wubbels, 2004) and teachers’ (pedagogical) content knowledge (e.g. Baumert et al., 2010; Kleickmann et al., 2013; Shulman, 1986) have an influence on student achievement. Research results about teacher competences were used as input for the ‘assessment development team who would construct a schoolspecific assessment method’ and based on this together with input from the teachers themselves, an assessment instrument was developed
The focus of this instrument was on senior teachers, because of their important role in the school; they have the most important (teaching) positions in schools and are responsible for coaching starting teachers, for example. It is assumed that these senior teachers determine the quality ofthe school to a large extent. Next to this, there was additional funding from the government, meant for the best teachers, in order to motivate them additionally and keep them in school. For the integral personal management ofa school it is important to be able to ‘spot’ and assess these key teachers in a valid way, presumably in a way that is accepted by the school team. In the study described in this paper, senior teachers have already been effective teachers for many years and for the new program to be developed, competences were needed that would have an additional value beyond ‘being a very effective teacher’
Thus, the main focus of our study was to develop and test an assessment program for distinguishing average senior teachers from very good senior teachers. The assessment program should contribute to an opportunity of self-reflection for the teachers as well. Therefore, the central question of our study is: How can senior teachers’ competence in secondary education be assessed, while providing the opportunity for self-reflection by the senior teachers? The assessment program was developed in close collaboration with a large secondary school and a pilot study was organized in which we carried out and evaluated the assessment program. In order to do so, the following steps were carried out: first, literature was explored on what ‘good teachers’ are and the content of the teacher competence had to be determined. Second, criteria and standards were defined in order to validly assess the competences of senior teachers. Third, the program sections of the assessment were determined. The final step was to carry out a pilot with eight participating senior teachers. The new assessment program was evaluated. Below, these four steps are described in detail.
Theoretical background Defining good teachers The ability to distinguish average senior teachers from very good senior teachers depends on how senior teacher competence is defined and what assessment criteria and standards are set (Uhlenbeck, 2002). In general, all assessments require a clear notion of the construct to be assessed (Messick, 1995; Sadler, 1998). This is especially important for the development process described in this article because the assessment program being developed in this study can be considered a ‘high stake’ assessment. A positive assessment result would lead to a salary raise, while negative outcomes of the assessment program would lead to a ‘frozen salary’. Senior teachers, as we focus on in this study, ought to be the school’s best teachers. Defining good teachers is complex and there is no consensus on this topic, yet (e.g. Berliner, 2001; Fenstermacher & Richardson, 2005).
Contemporary educational research on good teachers is scattered across a variety of research traditions, showing a diversity of definitions, instruments and results related to the issue of good teaching. These traditions can be broadly categorized as: (1) perception studies of ideal teaching, including learning environment research (Allen & Fraser, 2007); (2) effectiveness research (e.g. den Brok et al., 2004; Seidel & Shavelson, 2007), (3) studies on teachers’ professional knowledge (e.g. Berliner, 2004; Darling-Hammond & Snyder, 2000; Verloop, 2005), and (4) research on teachers’ professional identity (e.g. Beijaard, Meijer, & Verloop, 2004; Day, Sammons, Stobart, Kingston, & Gu, 2007). These four traditions have their own specific perspective of studying good teaching practices.
The first perspective, perception studies of ideal teaching, for example, show that students (aged 7–16 years) value a nice personality and teaching ability very important (e.g. Beishuizen, Hof, van Putten, Bouwmeester, & Asscher, 2001 ), as well as competent instructing, focusing on transfer of knowledge and skills. Kutnick and Vena (1993) mentioned physical presentation, teachers’ care for students, and trustworthiness as being important for good teachers whereas Hamacheck (1969) adds being helpful in schoolwork, clear explanation and humour.
The second tradition, effectiveness research, mainly focusses on the results of teachers’ actions on students’ learning processes, achievement or attitude towards learning (Seidel & Shavelson, 2007). Seidel and Shavelson (2007) used an interesting framework of teachers’ effectiveness based on cognitive models of teaching (and student learning) in their meta-analysis on teachers’ effectiveness studies. One of their conclusions was that domainspecific components of teaching resulted in the largest effects for students’ learning. Studies within this perspective show that the combination of teaching skills with communicative competence are important for gaining positive achievement by the students (e.g. Hattie, 2009; Marzano, 2003; Ryan & Deci, 2000; Scheerens, 2007). Further, Brophy and Good (1986) stated instruction and classroom management techniques are very important teacher behaviours. This is in line with findings from learning environment research as described above. More specifically, effectiveness studies show that in order to gain high student outcomes, teachers should be able to realize an appropriate level of difficulty for the instruction, continuous progress at a high success rate, effective diagnosis of learning needs and prescription of learning activities and monitoring of progress and continuous practice, integrating new learning (Brophy & Good, 1986; Marzano, 2003). This also fits the perception perspective, in which students also state that teaching ability is important and that they are preferably taught by competent instructors, who can transfer knowledge and skills (Beishuizen et al., 2001 ).
The third tradition described, concerns the (practical and theoretical) professional knowledge required for good teaching. Teachers’ domain-specific knowledge is important for explaining properly and asking the stimulating, specific, subject-related questions (Darling-Hammond, 1999). In order to be able to instruct well (professional) knowledge of teachers is considered a requirement (Clausen, Reusser, & Klieme, 2003; Wise & Okey, 1983). More specifically, teachers’ subject matter knowledge and pedagogical content knowledge have been argued to be essential for realizing quality of education (e.g. Hill, Rowan, & LoewenbergBall, 2005; Shulman, 1986). Teachers’ pedagogical and subjectrelated knowledge are often linked to their quality of instruction (Elbaz, 1991; Shulman, 1987).
Finally, the fourth tradition concerns research on teachers’ professional identity, taking the teacher as a person as focus for research, stating that the teacher’s personality is ‘omnipresent’ in his way of teaching and professional learning (Beijaard et al., 2004). The identity perspective claims that teachers perceive teaching as a combination of different roles regarding the teaching job and a certain hierarchy concerning these roles. Teachers view themselves as subject matter experts, learning experts, and pedagogical experts (Beijaard, Verloop, & Vermunt, 2000). The perceived hierarchy in roles determines teachers’ professional identity and their behaviour they show. Taking results from the above described research traditions together, an ‘integrated expertise’ comes forward, combining professional (pedagogical and subject-related) knowledge and instructional skills (Beijaard et al., 2000; Darling-Hammond, 1999; Stronge, 2007). From the perception studies and the studies on professional identity of teachers, personality-related characteristics could be added, but up to now little empirical evidence has been found that this leads towards better student outcomes
Summarizing, good teaching and good teachers have been studied from different perspectives, leading to different foci, like instructional quality and classroom management, professional knowledge and teachers’ personality as possible main factors for good teaching. As Gage (1964) stated, it might be the case that teaching and teacher quality cannot be described by a single theory at all. Vanderlinde and Van Braak (2010, p. 303) stated that ‘‘the traditional top-down model ofthe development and dissemination of educational innovations should be replaced by a model where teachers share a primary role with educational researchers in the development of innovative practices (Englert & Tarrant, 1995)’’. Vanderlinde and Van Braak (2010) refer to a research-practice gap. With this, they emphasize the fact that the use of and reflection upon and use of academic research by teachers are usually less than optimal. It seems that teachers and other educational professionals do not seem to see an additional value of educational research, or they are unable to use results from educational research in their practice. Vanderlinde and Van Braak (2010) found in their study a possible solution to bridge this gap, by realizing more cooperation between researchers and educational practitioners. In our study, for example, we have translated their insights into a more bottom up approach, by giving the information from literature as input to the development team.
Instead of sticking to one perspective of teacher quality, we choose for a more eclectic approach, using multiple perspectives. This was done in order to increase the school team’s commitment, leading towards an underlying competence profile for senior teachers that would be recognized and accepted by the school team, partly because this would be developed by the school team itself instead of driven by one single theoretical perspective. Teachers’ position towards an educational innovation is more positive when given ownership, agency and logical sense-making (Ketelaar, Beijaard, Boshuizen, & den Brok, 2012). This is why we decided to take our knowledge on good teachers with us as a starting point for the development of the assessment instrument, while not having these theoretical findings dominating the discussions with the development team of the instruments (see below). The way of bottom up working – instead of top down – on the development of a high stake assessment program, as described in this paper, contributes to the body of knowledge of assessment development as well as educational innovation, when striving for optimal commitment of the school team.
Development and evaluation of the teacher assessment program
Step 1: determining the content: teacher competence
As this study was carried out in a large secondary school in the Netherlands, the competence profile had to be locally valid. That is, the Dutch government has adopted the ‘‘Professions in Education Act’’ (2005), which specifies seven teacher competences as the minimum quality for certified teachers. Teacher-training colleges have to use these competences to assess their teacher students and a logical consequence is to use these same competences to assess working teachers in a personnel management system. These seven teacher competences are (Snoek et al., 2009): (1) interpersonal competence; (2) pedagogic competence; (3) subject knowledge and methodological competence; (4) organizational competence; (5) competence for collaboration with colleagues; (6) competence for collaboration with the working environment; and (7) competence for reflection and development. These seven competences refer to a ‘basic level’ or a starting point for junior teachers. For this, quick scans and internet self-assessment tools for teachers are available. However, more than this starting level is expected from senior teachers, being already the more effective teachers in school. Furthermore, instruments to assess teachers’ behaviour in the classroom already exist, like the Questionnaire on Teacher Interactions (den Brok et al., 2003). Unfortunately, in Dutch secondary schools, personnel management is a rather underdeveloped area and has not received much attention from the school management so far (Seezink & Poell, 2009). There are no systematically documented experiences from other Dutch schools, yet, which could be used in order to develop the assessment program aimed for. Therefore, a competence profile for seniorteachers – extending the basic and effective level – has been developed in this study.
School development team
In this study, a school-specific competence profile was developed for senior teachers. The (school) context was explicitly taken into account because of the specific demands of the school environment for senior teachers (Berliner, 2005) and because of the commitment of the school team to this new assessment program. Next to this, the described theoretical perspectives were brought into the development team by the authors of this paper, as well as specific trends within teachers education, for example, the teacher-research as being a competence teachers need for their ongoing professional development (van der Linden, Bakx, Ros, & Beijaard, 2012)
A development team consisting of ten teachers from the secondary school (10% of the entire school team) and the management were brought together with the assignment to specify the competences which were needed for by their senior teachers. Four out of the eight senior teachers to be assessed participated voluntarily in the development team. A large team was chosen in order to create a valid profile as well as a commitment for the use of the new competence profile for senior teachers as the underlying basis for the assessment program. For the acceptance of this new assessment program, the team of teachers should be confident that it is a valid and fair way of judgement (Baartman, Prins, Kirschner, & Van der Vleuten, 2007). During one year, the development team worked on the competence profile. They started with two open brainstorm sessions and they investigated literature on teacher quality and recent educational developments. The teachers themselves mainly used literature, which they frequently used for their own professional development. This was followed by discussion-sessions of first, second and third drafts of the competence profile for senior teachers. Eventually, based on consensus, the competence profile for senior teachers was accepted by all members of the development team (see Table 1 ).
The competence profile for senior teachers consists of two parts. First, the development team agreed that the seven competences determined by the Dutch government as described above relate to senior teachers as well as to starting teachers. These competences mainly focus on effective classroom activities, but also include cooperation with colleagues and stakeholders in the educational environment, even though this is a relatively small part of the competence profile. In order to assess the classroom activities as well, the school management decided to use the existing questionnaire on teacher behaviour, the QTI (den Brok et al., 2003). Second, senior teachers take on many outside-theclassroom activities, such as innovation projects and coaching younger teachers. Therefore, the development team formulated a competence profile for senior teachers with eight competences, specifically, having an added value above the seven national competences for teachers.
In total, the competence profile for senior teachers thus consisted of the seven teacher competences developed by the government and the eight competences developed by the school development team. The competence profile for senior teachers includes competences like cooperation, dealing with stress, problem solving and coaching. These were chosen because of the fit with the specific school-situation (Berliner, 2005) and because literature shows that these competences add towards educational quality (e.g. Brophy & Good, 1986). Next to this, competences like innovating, learning, anticipating and resultsbased acting were put into the competence profile because of the innovative developments within the educational field, like the teacher-researchers creating the opportunity to realize a critical, reflective attitude towards their practice (Zeichner & Noffke, 2001 ). Personality-related characteristics were not included in the competence profile for senior teachers because of the speculative relation with teaching ability (Damon, 2007). Finally, the meaning of the eight competences was described by specifying a number of ascending levels distinguished within each competence, in order to describe these as specific as possible. For senior teachers the highest levels are relevant.
Step 2: specification of criteria and standards
For the seven basic competences for (junior) teachers, quick scans and internet self-assessment tools for teachers are available. The focus of our paper was on the newly developed competences for senior teachers. Following, criteria were needed in order to validly assess the additional competences of senior teachers (Uhlenbeck, 2002). Assessment is a comparative process which requires a frame of reference, with unambiguous definitions of assessment criteria and standards (Damon, 2007). Therefore, assessment criteria and standards had to be developed, matching the eight competences for senior teachers. Standard-setting studies show that standards are often contingent on the local situation (Price, 2005) and are always subjective to some extent, for example if they are determined by a group of experts in the domain and thus rely on human judgement (Norcini & Shea, 1997). This is the case in our study. A way of specifying standards as objectively as possible, is the use of exemplars and verbal descriptions (Sadler, 1998). Exemplars are key examples that
describe the desired level of proficiency and are mostly used for product evaluations. Verbal descriptions or qualitative rubrics (Scriven, 1980) describe the properties characterizing the desired level of proficiency. These standards are context-specific and are often the most feasible to use, especially when multiple criteria are used (Sadler, 1998). A rubric is a scoring tool for qualitative rating of authentic work and it includes criteria for rating important dimensions of performance. It describes levels of performance on a particular task and thereby denotes what is considered important to both assessors and assessees. For assessors, it helps determine what to look for when assessing (Jonsson & Svingby, 2007; Tigelaar, Van Tartwijk, Janssen, Veldman, & Verloop, 2009). A review of studies investigating the use of scoring rubrics shows that rubrics can enhance the reliable scoring of performance assessments, especially if they are analytic and topic-specific (Heldsinger & Humphry, 2013; Jonsson & Svingby, 2007; Panadero and Jonsson, 2013). Consistent with the Sadler’s (1998) ideas, this study also shows the benefits of exemplars and adds the importance of rater training. Rubrics, on the other hand, do not automatically enhance the validity of performance assessments. This requires not only that the content of the rubric adequately represents the content of the construct to be assessed (in this case, senior teachers’ competences), but also that, for example, the mental processes used during the assessment are incorporated. According to Jonsson and Svingby, very few studies on the use of rubrics provide this kind of validity evidence, which implies that the effect of rubrics on the validity of performance assessments is not clear at the moment. In this study, we decided to describe a set of rubrics for each competence, as research seems to mainly show advantages of the use of rubrics. The teacher development team has described a set of rubrics for each competence defined in the competence profile for senior teachers, and they have also developed the competence profile in the same way. Eventually, based on consensus, the rubrics were fixed for each level (see Table 1 for the eight competences and the rubrics).
Step 3: determination of the assessment program parts
The competence profile and rubrics were the starting point for the further development of the assessment program for senior teachers. The first two steps described the development of the competence profile and the rubrics, defining the content (‘what is assessed’). The next, third, step is about the way how this content could be assessed. The choice of assessment methods largely determines the validity of the assessment process, as the methods should adequately measure the construct at stake (Messick, 1995). A single assessment would probably not be sufficient to validly assess senior teachers’ competences. A mix of methods should be used instead (Baartman et al., 2007; van der Vleuten & Schuwirth, 2005), because it reveals additional insights in comparison with one single assessment method, gaining input from qualitative as well as quantitative data (e.g. Spillane, Pareja, Dorner, Barnes, & May, 2009). Others propose a longitudinal process involving various methods in order to gain a rich picture of teachers’ knowledge and performance (Berliner, 2005; Darling-Hammond & Snyder, 2000). Ideally, measurements like observations, interviews and questionnaires should be repeated over time. The reliability of assessment can also increase when using different information sources, like peers, the teachers themselves, management and experts (Uhlenbeck, 2002). In this study a combination of assessment methods was chosen in order to combine the strong aspects of the different assessment methods. This study uses an assessment program (Baartman et al., 2007), consisting of a mix of methods: (1) observations and questionnaires; (2) interviews; and (3) portfolio assessment. Different groups of stakeholders (management, pupils, colleagues, the senior teachers themselves and two experts), were involved in the judgement of the senior teachers’ competencies in a pilot study of the assessment program. Table 2 summarizes the different methods used in the assessment program. These assessment methods and the rationales for choosing each method are described more in-depth below.
Observations and questionnaires: background and rationales
Observations are a powerful method to assess teachers’ quality because authentic teacher behaviours can be judged in the in vivo context (Chen et al., 2011; Landy & Conte, 2013). However, observations by schooled assessors are not very practical and quite expensive (Bakx, van der Sanden, Sijtsma, & Taconis, 2002). Pupils and colleagues can also play a role observing their teachers. Questionnaires can be used in order to judge competences of the potential senior teachers. The use of questionnaires, when using transparent, clear and uni-dimensional items, can be a rich and ‘standardized’ method for assessing teachers’ competences (Landy & Conte, 2013). Also, for the judgement of teachers’ behaviour in class validated questionnaires are often used, like the QTI (questionnaire on teacher interaction) (den Brok, Brekelmans, & Mainhard, 2010; Levy, den Brok, Wubbels, & Brekelmans, 2003; Telli & den Brok, 2012). Using questionnaires helps the observers tuning their perspectives towards certain aspects of the senior teachers’ competences. Having pupils observe their teachers using a standardized questionnaire can lead to richer insights in teachers’ classroom behaviour and can reveal aspects which might other ways remain implicit (Burden, 2010). In order to use pupils’ observations as part of the assessment of teachers’ competence, a large group of pupils is needed to prevent bias (Damon, 2007). The other competences, like cooperation within the teaching team, can be validly judged by observations by colleagues and managers. The observation of teachers by their colleagues can be seen as ‘peer assessment’. Different studies show benefits of peer assessment, with regard to professional development and reflection by the peer-assessor as well as the assesse (Sadler & Good, 2006), even though observations done by peers always have the problem of sympathy of the observed person. That is why at least four peers, working together with the teacher on a regular basis, should be involved in order to gain reliable and valid ratings (e.g. Sluijsmans, Brand-Gruwel, & Van Merrie¨nboer, 2002).
Practice of observations by pupils in the pilot study
For the observation by pupils, a standard questionnaire was used, measuring pupils’ perceptions of interpersonal teacher behaviour (Wubbels, Brekelmans, & Hooymayers, 1991). This questionnaire assesses teacher’s interpersonal behaviour, using nine aspects. Table 3 presents the scales, the number ofitems and a typical item for each QTI-scale (den Brok et al., 2003). These nine aspects are related to the competencies from the seven basic competencies (especially interpersonal competence and pedagogic competence) as well to newly defined competencies (especially flexibility/anticipating, dealing with stress and problem solving). For each senior teacher being assessed in the new assessment program (eight teachers in total), 20 pupils carried out the observations and filled out the QTI-questionnaire. The QTIquestionnaire (Wubbels et al., 1991) could be filled out by the pupils anonymously. Therefore, during twelve weeks, the pupils observed their teachers. After this period they completed the questionnaire online in the computer-classroom under guidance of an ICT-assistant for the first time. This was repeated half a year later
0 Response to "Studies and Eavaluation part 1"
Post a Comment