The English Teacher Vol XXV October 1996





THE EFFECT OF TOPIC TYPE AND
DIFFICULTY ON ORAL PERFORMANCE

Janet Y. Yong and Maya Khemlani David
University of Malaya

 

ABSTRACT

This study describes and evaluates a typical oral-production test for first year Economics students taking an ESP Course (English for Economics and Business) in the Faculty of Economics, University Malaya, and provides suggestions and guidelines about how the test may be improved.

 

 

Introduction

In oral testing the test question or stimulus to which the speaker must respond orally is a key factor in determining the performance of the learner.

This study describes and evaluates a typical oral-production test for first year Economics students taking an ESP Course (English for Economics and Business) in the Faculty of Economics, University Malaya, and provides suggestions and guidelines about how the test may be improved.

The test questions used were analyzed. From the speakers' performance during the test we discovered that performance on the test depended greatly on the nature of the test question. The elicitation techniques, choice of format and the evaluation and scoring system were found to comply with the guidelines set by Preparatory Oral Skills for Management (POSM), University of Malaya Spoken English Project (1985). POSM aims to provide undergraduates with the spoken skills necessary for entering the professional fields of public administration, business and commerce. The skills tested involve information establishing and sharing, putting forward and evaluating positions and reaching decisions.

At times the nature of the test question was unsuitable due to a number of limitations. These limitations will be discussed in this article.

 

The Design and Construct of the Test

The test was designed to evaluate the ability of the students to engage in discussion of a professional nature. The student is required to make appropriate responses demonstrating communicative competence in business situations. By communicative competence we mean a level of organization, developed from habitual performance which represents a stage of preparedness of use. Halliday (1970) puts it this way: "Language serves for the expression of "content", that is, of the speakers experience of the real world, including the inner world of his own consciousness." The test is supposed to observe a number of characteristics in a communicational event: the use of language is interaction based, it is unpredictable, it takes place in a context, has a communicative purpose, it is authentic and is behaviour-based. (Morrow 1977).

Our oral production test of communicative and connected discourse utilizes wholly written elicitation cues. The examiner is not to introduce the topic or event with background information on what the examinees have to speak on or ask questions to determine their familiarity or experience with the topic. The examiner also does not interrupt the examinees. In other words, the examiner is removed from direct conversation with the examinees. The examiner is a silent observer.

The elicitation techniques used in the problem - solving task include role-play, oral presentation and structured dyad oral interaction. A group of four students are assessed together but graded individually, based on both the individual delivery of a given topic (linguistic ability) and the interactive ability or performance in the group (use of specific subskills eg. agreeing/disagreeing, giving suggestions/counter suggestions, deciding, etc.) and initiative in asking for and giving information.

A test stimulus is provided. In the first part of the test, each examinee is given a role to play - eg. as a business consultant or a committee member the examinee will have to present a talk on his assigned topic. (See Situations 1-5).

Each speaker is given five minutes to study her/his topic and talk about it for 5 minutes. The other three speakers in the group listen and take note of the main points which will be used later in the second part of the test, in the group discussion.

In the second part of the test, the four speakers are required to evaluate each others points and then come to a general consensus regarding the best solution to a specific problem.

 

1. Selection of Topics

As the claim of the oral test is to encourage students to speak freely on a topic familiar to them they are then assessed on the basis of that speech. It is, therefore, vital that the topics and tasks set should be within the (a) analytical/cognitive ability and (b) linguistic capability of the examinees concerned.

We found that the topics did not meet these criteria. The topics were either too specific or too vague.

(a) Analytical/Cognitive Ability

When a topic is too specialized in nature or scope, it can hinder the student insofar as she/he may have a limited world view of the issue or subject, even though the situation may be within the broad purview of her/his subject or discipline, and consequently she/he cannot talk sufficiently on the issue concerned.

Alternatively, if the topic is too broad or general, there is less control over the subject-matter and the student may not know the extent to which she/he should limit her/his talk in relation to the topic. Furthermore, being first year undergraduates these students found the subjects beyond their experience and understanding of the real world of business.

Below is an example of a situation which is 'broad' in scope and cognitively too difficult.

Situation 1

An entrepreneur with a capital of $ 100m has consulted you on investment opportunities in Malaysia. As an investment consultant, you have some proposals in mind.

Presentation

You propose that

The Enterpreneur invests in:

  1. Manufacturing
  2. The Motor Industry
  3. A Housing Development
  4. The Hotel Business
An example of a topic that is too specific is one that puts the student in a straitjacket insofar as the student is constrained to a discussion of a certain job or country of which she/he has minimal or no knowledge of.
Below is an example of a 'too specific' situation.

 

 

Situation 2

Your friend has recently completed his HSC. He intends to study for a degree in accountancy. He comes to you for advice.

Presentation

You advise him to do a degree in accountancy in:

  1. Britain
  2. Australia
  3. Singapore
  4. Malaysia
This immediately gives one student an unfair advantage over another who may be given a country which she/he is familiar with. We found that the students who talked about Singapore and Malaysia did significantly better than those who had to imagine what Britain and Australia are like. The options for the four students are therefore not of parallel difficulty. Hamp-Lyons and Prochnow (1991) talking of difficulties in writing assessment, state that the writing prompts must be of parallel difficulty. The same argument holds for the oral prompts. The above example illustrates this point and the test is therefore, not a valid and fair test of knowledge.

The test question must, therefore ensure that the various options presented to the four students must be comparable. The students should be deemed to have comparable world or prior knowledge of the issues involved in the four options, otherwise, one examinee may have an edge over the others.

 

(b) Linguistic ability

The language used in the questions was at times beyond the comprehension of the students. Thus, connected discourse, that flow of language generally felt to typify real communication, was not generated. The language used was found to be too difficult for Year 1 students. Examples of expressions used in some of the questions which required clarification are as follows:

'revamped', 'strategies', 'cease operation' and 'diversity'.
In addition to the lexical items, a task is more comprehensible if put in the active rather than the passive form. Language learners generally internalize the active forms before the passive. Moreover, instructions are best understood when phrased in the active form. Again, there was non-standardization with some questions appearing in the active form while others in the passive form as shown below:-
"You propose that the marketing strategies of the company be revamped".
The above sentence should read:-
"You propose to revise the marketing plan of the company".
The language in the test question was sometimes both vague and wordy. Too much information was provided in the question sheet. Many students spent more than ten minutes reading the question and had little time left for speaking.

 

2. Variations on a theme

Each student in the first part of the test has to discuss the advantages of a pre-assigned topic. Topics based on a similar theme should have been provided. However, we found that this was not always complied with.

Below is an example of a question that has not met this requirement.

 

 

Situation 3

Your friend recently graduated from the Faculty of Economics and Administration. She comes to you for advice about employment.

Presentation

You advise him to

  1. Apply for a Job in a Bank
  2. Become a Farmer
  3. Do a Postgraduate Degree
  4. Start a Food Business
Doing a postgraduate degree is not in tandem with the other three options. The common theme in the above situation could have been for instance, looking for jobs in different work situations eg. bank, university, library, departmental store.

 

3. Range of Topics - Level of Difficulty

Furthermore, the topics ranged from very easy to fairly difficult as is illustrated in the following examples:-

Fairly Easy Topic

 

Situation 4

Aljafri has to go to Singapore to attend to some important matters.

You recommend that -
  1. He goes by bus
  2. He goes by taxi
  3. He goes by train
  4. He goes by plane
Fairly Difficult Topic

 

Situation 5

You are a management consultant. A textile plant in your town has been making losses for the past one year.

Presentation

You propose that

  1. The marketing strategies of the Company be revamped
  2. The Plant be relocated
  3. The Company diversifies its manufacturing activities
  4. The Plant ceases its operation
The range of topics as given in the example above are not within the same level of linguistic and cognitive difficulty. The different sets of questions were inconsistent in so far as the level of difficulty was concerned.

 

4. Recommendations

The following points should be taken into account when writing oral test questions of this nature:-

  1. there should be a consistency of level of difficulty within a particular test
  2. the language used and the topics should be pitched at the level of the examinees concerned.
  3. the sets of topics should be of parallel difficulty ie. they should all carry the same theme or variations of the same theme, so a to ensure that the language input of examinees is similar, if not the same.
  4. the language should be simple, concise and to the point. Extraneous information would distract the speaker from his intended task and role in the test.

 

 

References

Halilday M.A.K. 1970. The Linguistic Science & Language Teaching. London. Longman.

Harris, D. 1969. Testing English as a Second Language. New York. Mcgraw Hill.

Preparatory Oral Skills for Management, 1985. Language Centre University of Malaya, Kuala Lumpur. University of Malaya Press.

Morrow, K. and Johnson, K. 1977. Techniques of Evaluation for National Syllabus. Reading. University of Reading.

Hamp-Lyons and Prochnow. 1991. Difficulties in Setting Writing Assessments. Language Reporter Volume 6.



© Copyright 2001 MELTA