Gillies Haughton
Moray House Institute of Education
Heriot-Watt University, Edinburgh

The paper reports on the results of an investigation into attitudes towards self-assessment of Malaysian pre- and in- service trainees at a UK Institute (Moray House). Trainees were invited to take part in a self assessment exercise, where they tried to assess their own B.Ed dissertations, identifying strengths and weaknesses. The results of this exercise are discussed and the implications for self-assessment projects explored in the light of' the 'reflective practitioner' model of teacher behaviour.

Introduction

This paper concerns itself with the issues of the willingness and capacity of teachers to engage in self-evaluation of their professional activities. It may be regarded as a laboratory rather than a field study since the teachers in question were all engaged in courses of professional development (either pre-service or in-service B.Ed TESOL courses at Moray House Institute of Education, Edinburgh) and the professional activity was the production of a professional project at the end of their courses. Nevertheless, the assumption of such courses is that they prepare participants for the trials of professional life and that behaviour on such courses is both a simulation of and a development of desirable features of real-life professional behaviour.

The course - in common with many modern teacher education courses - seeks to develop professional autonomy and the capacity to reflect on professional action. It actively encourages participants to reflect on the choices open to them and on the results of the choices made by them . As such it seeks to develop "reflective practitioners" (Schon, 1987) If participants have indeed developed such capacities it would seem reasonable to expect them to demonstrate this at the end of their course. One way in which they might do so is by demonstrating that they are willing to engage in critical self evaluation. Furthermore, they may be expected to be capable of conducting such self evaluation in a way that would conform with the evaluation of fellow-professionals. This would mean not only that they arrive at the same sort of final decision - i.e. end up with the same grades - but also that the way in which they arrive at that decision would be broadly similar - that they attend to and value the same sort of features in roughly the same degree.

To clarify these three questions, participants were invited to engage in a self-assessment of the professional project they produced at the end of their courses. Willingness to participate would be taken as some indication of willingness to engage in self-evaluative behaviour. Those who did take part in the self-assessment were asked to grade their assignments using criteria familiar to them and identical with those used by their course tutors. It was hoped that there would be a close match between self-assessed and tutor awarded grades and this match would be evidence of capacity to undertake self-evaluative behaviour. Finally, participants were asked to complete a questionnaire. In this they were invited to identify the reasons why they thought the assignment merited the grade they had awarded, what features they thought tutors would particularly attend to and what were the strengths and weaknesses of their own assignments. This was intended to clarify whether the processes of self-evaluation were substantially similar to the processes employed by tutors.

Self-Assessment and Reflective Practice

A particularly appropriate locus for the study of reflection is the professional project. Such extended assignments- variously titled 'dissertations" professional projects' or 'extended studies' are common at the end of professional development courses. Characteristically they require participants to synthesise theory and practice, by reviewing literature or concepts discussed in workshops and seminars and relating these to a particular context - in this instance Malaysian secondary schools. Students are encouraged to identify their own areas of investigation, to decide why it is worthy of investigation, to design their own means of carrying out the investigation, to implement it, analyse the results and evaluate the findings. In all of this they are engaged in what Schon (1987) calls 'problem-forming' and in reflecting on knowledge-in-action.

Furthermore, the process as a whole requires to be critically evaluated. Participants are expected to note the shortcomings of their design, implementation and analysis. In effect the entire process is a simulation of the desired behaviour of the professional in action.

In Moray House students are invited to identify an area of interest at the end of their third year. In consultation with a supervisor they prepare a project proposal before the end of the academic year. If they then return to Malaysia during the vacation they. may make use of the opportunity to gather data. Alternatively they may request assistance with data collection from friends or colleagues back in Malaysia. The main part of the work is carried out in the fourth year. The project which results from this extensive gestation period is the final piece of work submitted by the students.

There are two submission dates - a preliminary one two weeks in advance of the final submission. The preliminary one is to ensure that all the necessary pieces are drawn together and to allow the supervisor an early opportunity to identify remediable errors. At this stage such errors are largely a matter of presentation - missing references, obscure expressions etc., but there may also be more substantive issues - e.g. linking between different parts is unclear and recruires to be articulated or some part requires greater or less emphasis.

The finally submitted work is given to another tutor to mark and comment on. The s~pervisor also grades the assignment. Both tutors then meet to discuss their impressions and agree on a final grade. In the event of disagreement the work is referred to a third marker. The first marker is responsible for the feedback notes which are returned to the student. Although the first marker writes these comments they are produced after the two markers have conferred and represent their deliberated opinion. There were altogether six markers involved in the sample under consideration.

Target Groups

Two groups were invited to take part in the exercise. Both were following the B.Ed TESOL programme at Moray House. Both were entirely Malaysian in composition. One was a group of 11 pre-service students - all female, all in their twenties. The other was a group of 20 in-service students - 12 female and 8 male, all with a minimum of eight years teaching experience. The pre-service group had attended a two year preparatory course in Edinburgh and were about to complete a four year course at Moray House. The in-service group were completing a link programme, which involved doing the two final years of the course at Moray House.

Both groups were required to do a "professional project". The specifications and criteria for these projects were identical. However, in the case of the pre-service group, there was a stronger emphasis on the collection of data and the trialling of materials. This was because this group had completed a period of school placement in Malaysia just before commencing their fourth year and were thus in a favourable position to collect data or conduct trials. The criteria for assessment of the project were the same criteria which had been used throughout the course for the assessment of other assignments. Both groups were thoroughly familiar with these criteria and fairly familiar with the ways in which tutors interpreted the criteria, although, clearly, the pre-service group had much greater familiarity with both.

Design of the Investigation

The investigation was closely modelled on one described by Penny and Grover (1996) in which they attempted to discover "the extent to which final year students following a four year degree in education have a realistic sense of' the strengths and weaknesses of an independent research study they are required to undertake."

Penny and Grover discovered a poor match between student self-assessment and tutor grades. They also noted that students emphasised "lower order criteria such as style and presentation" and paid little or no attention to criteria concerned with theoretical and conceptual understanding and the quality of discussion.

The present study sought to replicate Penny and Grover's work, though on a considerably smaller scale. (Penny and Grover studied some 160 students over two years.) Whilst the extent to which self-assessments would tally with tutor assessments was of interest, the main focus was on the reported perceptions of what was valuable in the extended study both by self-assessors and by tutors.

The major point of difference in the two cases was that Penny and Grover's subjects were ali native speakers whereas the Moray House group were not. Previous studies of self assessment of academic writing by non-natives (Wangsotorn, 1981; Blue, 1988; Haughton and Dickinson, 1988) have suggested that students tend to overestimate in assessing their own writing. However, Penny and Grover noted that although underestimates did occur among their self-assessors, most of the mismatch was due to overestimation.

Blue's study compared performance of different nationality groupings and noted that only European non-natives consistently underestimated performance in academic writing. Although Asians (Chinese, Japanese, Thais, Indonesians) overestimated less than other non-Europeans they were still more prone to overestimate writing performance than performance in other skill areas.

The difference between natives and nonnatives may, however, be less significant than might appear. Cummins (1979) has argued that basic language proficiency is separable from academic competence and that the skills of writing academic prose must be developed separately.

Two further points of difference with the Penny and Grover study are significant. First Penny and Grover required the whole cohort of students to respond, whereas the Moray House students were free not to take part if they wished. Secondly the grades in Penny and Grover's study were percentage marks, where Moray House uses letter grades.

In the present study both groups were invited to reflect on their professional projects as soon as they had submitted them. They were provided with a questionnaire (Appendix 1) and with the standard assessment sheet (Appendix 2) which accompanies all assignments and which identifies the criterion areas. These areas might not all be applicable - for example, some projects did not rely on the collection of data, so the criterion with regard to data would not apply to them. The sheet also provides a system for tutors to estimate performance under relevant headings. When tutors mark other assignments this is the sheet they use to provide feedback to students. Students were told that they could use this for guidance, but did not need to follow it precisely. This in fact is what happens when tutors mark the professional projects - they are guided by the criteria, but do not complete the sheet. Nevertheless, some students did return the completed sheets along with their questionnaires.

The questionnaire contained three parts. First it invited the students to allocate a grade. Possible grades are A - E with E as a fail and D as a borderline pass. Split grades such as A/B are not possible, nor are 'grades-within-grades' such as A+ or B-. The primary interest here was in how well the self-assessed grades matched with the actual grades awarded by tutors.

There were then a series of open-ended questions about why the students had graded the way they had, what they thought tutors would pay attention to in marking and what they thought were the strengths and weakness of their project. This was intended to clarify how the students went about the self-assessment, in particular which areas they attended to or considered should be attended to.

The final section of the questionnaire contained a series of closed questions where students were asked about the degree of confidence they felt in their judgements, how long it had taken them to reach their judgements and how difficult they had found it. The main interest here was to discover whether the investigation needed to be amended before it was repeated the following year, but it was also useful to note whether there was any correlation between the accuracy of self-assessment and the speed, ease and confidence with which it was carried out. There were also two questions about whether students would want self-assessment to be taken into account in awarding grades either for the project or for other work. There was no question of this actually happening - students had in effect just completed their course - but it was included as a check on the reports on confidence and as an indicator of further willingness to engage in such activity.

Results

The results clearly indicated quite marked differences between the pre-service and the in-service groups.

1. Response to the investigation

Of the pre-service group 7 (83%) returned the questionnaire compared to 11 (55%) of the in-service group. In the in-service group only 2 men (25%) responded, where 75% of the women responded. Willingness to respond related to some extent to actual performance - 15 out of the 18 did actually score A's or B's for the project. However, among those who did not respond were 3 A's and 5 B's.

2. Self assessed grades

Self assessed grades were distributed as shown in Table 1. As might be expected, most students predicted A's or B's. Accurate predictions have been printed in bold. The in-service group included a number of people who were uncertain - two thought it was 'A or B' and one thought it was 'B or C. (All three were nevertheless 'fairly' or 'very' confident about the grade.)

The pre-service group were notably more accurate predictors. Accurate predictions among the in-service group include the two 'A or B' predictions. Overestimation was more common than underestimation, though both occurred. However, in only one instance was the estimate out by more than one grade.

3. Features mentioned in determining self-assessed grades

Students were asked to identify why they thought their projects merited the grade they had awarded themselves. The responses covered a wide range of features, but these have been reduced, for ease of reporting, to a number of common headings. These are summarised in Table 2. Two of the in-service group specifically referred to the established criteria ("The project fulfills all of the criteria required... ", "I have considered criterion 1,2,3,5 & 6...") and the pre-service group generally used the established criteria as a guide and employed the terminology used in them.

Most of the comments were positive - i.e. the project merits a good grade because it possesses these excellent quaJ.ities - but some also mentioned reasons why the project could not merit a better grade. One of the in-service group admitted ignorance - "No idea. Just instinct and hope."

Student accounts of what was notable in their projects were then compared with the markers' comments. In general there was a measure of agreement in tutor and student perceptions of what was important, though tutors tended to reveal in their comments both a wider range of issues and much more attention to the specifics of the methods employed or the content of the project. In some cases the match was very close indeed. Compare, for example, these two accounts:

Student

I think I have managed to present my project most efficiently by provi ding a detailed literature review on the subject, analysing the data collected in great depth and also suggesting some positive steps to be taken, as a result of the research.

Tutor

This is a lucid and attractively written account of the project, well supported by a range of readings. A particular strength is the critical awareness which is demonstrated throughout, both in the course of the investigation and the conclusions drawn from it.
The standard of presentation is excellent. This seems to be an excellent example of 'action research' which marries theory and practice extremely well.

Student

I feel that the discussion was not as in depth as I would like it to be

Tutor

Ch 1: well presented background
Ch 2: not a very clear account of the teaching of reading - not much relevant literature; satisfactory on assessment.
Ch 3: clearly explained - though the issue of comparison between texts is avoided - not an entirely satisfactory research design
Ch 4: not very sophisticated statistics - no use of significance tests; loses sight of criterion referencing Ch 5: rather trivial conclusions - perhaps more could be made of the research findings.
Overall: this study is coherently presented and wello explained. It falls short of being outstanding in respect of its statistical handling and its disregard of the criterion referenced dimension of assessment in the analysis and discussion.

4. Student perceptions of tutor criteria

Unlike the situation investigated by Penny and Grover, the students in this case were all fully aware of the criteria employed. Most employed the terminology of the grade sheet criteria. The results are summarised in Table 3.

The question here was largely to see whether there was a match between what students reported as the criterial values in their own assessment and what they expected tutors to assess for. Notably three of the frequently mentioned features in Table 2 - argumentation, practicality and use of reading - are also frequently mentioned in Table 3. Critical evaluation - mentioned by all of the pre-service group as an important tutor criterion - does not figure at all in student accounts of their own assessments. The somewhat vague heading of 'content' is never clearly articulated in accounts of self-assessment though in some cases the word does appear e.g. "a thoroughly researched piece of work consisting of relevant content, discussion and reading sources."

Although several students did identify hard work and guidance from the supervisor as important factors in assessing their own work, none suggested that tutors would reward hard work or obedience to supervisors. In fact, although tutors do mention hard work in their comments, these tend to be in a negative context e.g. "Although the data has been very thoroughly examined the results are disappointing".

Again the student reports were compared with the comments of tutors. Thtor comments revealed that all tutors did in fact attend to the stated criteria. Features which appeared in all comments included the use of reading and the quality of the discussion or argumentation. Comments about practicality, content, critical evaluation and research or data occurred in 80% or more of tutor comments. Although originality and relevance to the Malaysian context were mentioned, they were much less frequent. The sole feature listed in Table 3 which did not appear in tutor comments was "ability to work independently", though there were mentions of "independence of thought' "an original and independent approach", "an independent and critical stance" and "a thoughtful personal commentary".

5. Strengths and weaknesses of projects

When it came to identifying strengths and weaknesses students were less forthcoming. Many of the responses to both were extremely short e.g. "Practicality of the project" (a strength) or "Recommendations" (a weakness). Comments on strengths tended to be less informative e.g "providing relevant details/contents" or "It has been specifically focussed". Responses which were more elaborated did not always reflect earlier answers. However, for the sake of comparison the same terminology has been employed in summarizing these features in Table 4.

Only one respondent actually identified a match between strengths and the features listed in determining self-assessed grades - "All the points mentioned in Q2". Practicality and research consistently appeared as both factors in self-assessment and as perceived strengths. None of those who had identified originality as a factor in self assessment mentioned this as a strength, whilst those who mentioned it as a strength did not include it as a factor in their self assessment.

Again estimates of strengths were compared to tutor comments. Frequently the student statements were too terse to be usefully compared, but on occasion there were very close matches.

Student

I think that my project is well organized and is fairly easy to follow. I also feel that I have written closely to my title and produced recommendations relevant and applicable to my professional needs.

Tutor

A highly readable and well-organized project inevitably perhaps, given the focus on teacher qualities, the Recommendations take the form of ~pious vows'

Students wrote at more length and in more diverse ways regarding weaknesses. Many were aware of shortcomings in the data they had collected or in the means they had employed to collect it. Several explicitly blamed the word limit imposed on the project for either reducing the scope of their investigation - "The word length limits the number of activities that could be included" - or limiting the extent or depth of the discussion "insufficient in-depth coverage in Research and Discussion due to the word limitation". Table 5 lists the weaknesses mentioned.

Two students affirmed that there were no faults because of their supervisors' help:-

"with continuous guidance from the tutors especially my supervisor all the weaknesses have been avoided"

"none - my supervisor would have advised me regarding this lack if there is any"

6. Ease and speed of self-assessment

The in-service group was notably more relaxed in their approach to self assessment. 72% considered it 'very' or 'fairly easy' to self-assess, compared to 28% of the pre-service. Only 2 of the in service group (18%) spent more than 15 minutes on the process of grading and reflecting on the process of grading, where 4 out of 7 pre-service (57%) did. Ease and speed did not relate to accuracy of prediction -whereas the two pre-service students who found the exercise easy, were accurate predictors, this was true of less than half of those in the in-service group. Of the six students who took more than 15 minutes over the exercise four were inaccurate in their predictions.

7. Confidence and enthusiasm for self-assessment

Most of the participants were confident of their grades (Table 6).

Confidence, however, did not match accuracy - only 2 of 5 very confident in-service participants were accurate. The in-service and pre-service groups differed considerably in their enthusiasm for taking self-assessed grades into account in the grading of projects: 64% of the in-service group reported strong agreement where most of the pre-service group (57%) were neutral.

Conclusions

The study reported here was undertaken as a pilot for a more extensive investigation which will seek to clarify some of the tentative conclusions sketched below.

1. Willingness to engage in self-assessment

The response was less enthusiastic than might be hoped. If reflectiveness is a core professional attribute and one which the course seeks to promote among all the participants, then it would be hoped that most students would wish to participate. There appeared to be greater willingness to participate among high scoring students. This may be because it is more pleasant to reflect on perceived good performance, but reflective practitioners should be prepared to reflect on problematic areas. Indeed there is most need for these areas to be the subject of reflection. Since the exercise necessarily took place at the very end of the course, many people may understandably have had other priorities and failure to take part need not indicate unwillingness to undertake self assessment in general. It is hoped to investigate reasons for non-participation more thoroughly with succeeding groups.

2. Accuracy of self-assessments

The accuracy of the self assessed grades, among the preservice group, was impressive. It is obviously easier to obtain agreement on a five point scale than on a percentage scale, but Penny and Grover reported 66% of student assessed marks falling within 10 marks ( 10%) of the tutor marks. On that comparison the pre-service group does well (71%).This is almost certainly due to this group's much greater familiarity with the interpretation put upon criteria by the tutors.

The in-service group were much less precisely accurate and their tally of 'direct hits' (45% including "A or B" assessments, 27% excluding them) compares poorly. Although these students had not received any formal practice in self-assessment and other studies suggest that practice does certainly improve accuracy, yet they had been exposed to an extensive process of criterion referenced assessment where the criteria were explicitly stated, where the application of the criteria to specific assignments was generally discussed by tutors in advance of assignments and where the feedback on assignments made use of the same formally stated criteria.

In common with other studies overestimation was much more frequent than underestimation.

3. The quality of the self-assessment process

Penny and Grover reported that students mainly used lower order criteria of style and presentation and ignored higher order criteria such as conceptual understanding and quality of discussion. They noted that students who did refer to higher order criteria all scored highly and that 75% of high scoring students did in fact refer to such criteria in reporting on their self-assessing. Although the Malaysian students did in fact produce many of the types of comment that Penny and Grover noted - emphasis on reading, practical usefulness, hard work they did also in substantial numbers refer to the way in which they had handled the argument as an important factor in determining grades. It must, however, be conceded that the vast majority of the students (83%) who responded scored A's and B's. Style and presentation was not much mentioned by the Malaysian groups.

Students' self-perceptions, as stated in their reasons for awarding self-assessed grades and in their accounts of strength and weakness, were generally coherent and consistent, but varied extensively in the extent of their detail and the clarity of their expression. They appeared to be sincere on the whole. Nevertheless, the student who did get an 'A', self-assessed as a 'B' and wrote "No idea. Just instinct and hope" was either simply not tempting providence or depressingly truthful.

Comparison of student self-perceptions with reported perceptions of tutor criteria revealed a significant gap. Students knew what tutors would assess - knew for example that critical evaluation would be an important factor - but did not translate that into their own assessments. This may mean that there is a difference of opinion - tutors think this is important, but I don't - or a lack of comprehension - tutors think this is important, but I'm not sure what it is - or a lack of confidence - I'm sure it's important, but I don't know if I did it very well. Comparison of tutor comments with self assessments revealed that students were generally accurate in their predictions of what tutors would value.

On the whole, the study provides some evidence that students at the end of a course in professional development are capable of reflecting on major work they have undertaken. High scoring students are more likely to engage in this activity and more likely to perform well. They reflect in a way that reveals significant consonance of views and values with their fellow professionals - both peers and tutors - and they can accurately assess the outcome of their work. The marked difference in pre-service and in-service groups -both in willingness to participate, and accuracy of prediction and quality of reflection - is disquieting - one would have anticipated that the mature professional would be more able and willing to engage in reflection. Finally, if we regard anticipated that the mature professional would be more able and willing to engage in reflection. Finally, if we regard 'reflective practice' as a virtue, then the fact that high scoring students are more willing and able to reflect is also disquieting - we can not all aspire to greatness, but we can all aspire to goodness.

References

Blue, G. 1988. Self-Assessment: the limits of learner independence In: A. Brookes & P. Grundy (eds) Individualization and Autonomy in Language Learning (ELT Documents 131) MEP & the British Council.

Cummins, J. 1979. Cognitive/academic language proficiency, linguistic interdependence, the optimal age question and some other matters Working Papers on Bilingualism (19).

Haughton, 0. & Dickinson, L. 1988. Collaborative assessment by masters' candidates in a tutor-based system Language Testing 5: 2.

Penny, A.J. & Grover, C. 1996. An Analysis of Student Grade Expectations and Marker Consistency Assessment & Evaluation in Higher Education 21: 2.

Schon, D.A 1987. Educating the Reflective Practitioner: Towards a New Design for Teaching and Learning in the Professions. Jossey-Bass.

Wangsotorn. 1981. A Self-assessment in English skills by undergraduate and graduate students in Thai universities In: J.A. Read (ed) Directions in Language Testing SEAMEO, Regional Language Centre, Singapore University Press.

Questions put to self-assessment participants

	Name:	  	 Pathway:

Section One Grading the professional project

1. Using the criteria published for the B.Ed course, decide on a final grade for your professional project.

2. Why do you believe your project merits the grade you have awarded it?

3. Which qualities do you expect tutors to focus on when they grade your professional project?

4. What are the particular strengths of your project?

5. What are the particular weaknesses of your project?

Section Two Self Assessment

1. How easy was it to grade your project?
   very easy    fairly easy    average    fairly hard    very hard

2. How confident are you in the grade you have awarded yourself?
   very confident    fairly confident    not very confident    not at all

3. How long did it take you to answer questions 1-5?
   Less than 15 minutes    15-30 minutes    30-45 minutes
   45-60 minutes    more than 60 minutes

4. Would you want a self-assessed grade to be taken into account in awarding your final grade for your professional project?
   strongly agree    agree    neutral    disagree    strongly disagree

5. Would you want a self-assessed grade to be taken into account in awarding grades for other types of course work?
   strongly agree    agree    neutral    disagree    strongly disagree

Note to Student: Please fill in details for the sections marked * on this page and overleaf.

* NAME___________________________ * MODULE ______________________

Satisfactory Length. Please note that the specified and actual length of the assignment must be recorded. Any assignment which is more than 25% above the specified length will be automatically downgraded by one grade.

Grades With respect to the specified criteria:

A signifies that the work is outstanding, or as good as could reasonably be expected from someone operating at this level.
B signifies that the work is good.
C signifies that the work is satisfactory.
D signifies that the work meets a minimally acceptable standard.
E signifies that the work is of an unacceptably low standard.

1. Expression and Discourse Organisation
   How well organised is the material? Is it clearly presented and easy to follow? Is It helpfully laid out, and is there visual support where appropriate? Is it cogently expressed?

2. Content, Analysis and Discussion
   Is the content accurate? Is it relevant to the topic? How well is the material analysed? Is the discussion convincing? How well are the topics handled in terms of range and coverage?

3. Use of Reading and Other Sources
   How well has the prescribed reading been covered and understood? How well have the sources been used to further the discussion? Are the sources properly presented and documented? Is there evidence of independent reading?

4. Primary Data Collection
   As far as can be judged, is the primary data accurate? How relevant is it to the topic? How well has the Primary Data been integrated and the discussion? Has it been properly documented and (where appropriate) made retrievable for inspection?

5. Practical Areas and Application
   How well has the practical aspect been related to theory? Is the practical work appropriate and feasible in the given context?

6. Critical Judgenient (Advanced Credits Only)
   How well has the student been able to bring together (synthesise) different readings and different kinds of experience etc. in order to provide a coherent point of view? How well has the student been able to evaluate sources and experiences in a thoughtful and practical way? Is there evidence of independence of thought or are ideas uncritically accepted?