The English Teacher Vol XXVIII October 1998

THE CONSTRUCT OF READING FOR TESTING

Yuji Nakamura
Tokyo Keizai University, Japan

ABSTRACT

The purpose of this study is three-fold. The first is to provide the ideas of components of reading in order to make a practical test to assess Japanese students' English reading ability. Information will be given from four viewpoints.

the nature of reading
the theoretical or linguistic underpinnings of reading
the test format of reading, and
classroom teachers' ideas based on their teaching experiences.
Another purpose is to investigate how much Japanese teachers and native English speaking teachers agree on the construct of reading. The third is to show how the new construct of reading is reflected in an actual reading test.
200 experienced English teachers were asked to answer a questionnaire consisting of 16 items. The teachers rated those items on a four-point scale according to the degree of importance of reading comprehension.
One result from the statistical analysis shows that ability to use schemata is more important to them than to use sentence level English. The results of cluster analysis will suggest more detailed and explainable components of reading both statistically and Iing-uistically. The comparative study between the native speakers' component and the Japanese teachers' component of the questionnaire results and the test format of an actual test will demonstrate (1) what is insufficient in the actual reading test and (2) what is causing the deviation between the idealized components and the actual reading tests.

I. Theoretical background and rationale

Reading is essential as one of the four language skills. It entails a variety of problems not only in teaching but also in assessment. Some teachers claim that reading assessment should include reading comprehension plus narrative summary. Others state that translation is inevitable in reading. Inevitably, the scores of a reading comprehension test deviate from the teachers' grades of students' reading ability in class even if they exclude non-language aspects such as students' attendance, submission of assignments, or making an effort from their grades. The problem is partly caused by the difference in the definition of reading ability used by existing tests and that used by teachers. Teachers first should come to an agreed construct of reading and then their construct of reading should be compared to the components of existing tests. Of the various methods used to arrive at a construct of reading three are introduced here. The first is to approach the construct of reading from the theoretical point of view. Another is to rely on ideas of experienced teachers and scholars. And the third is to explore the nature of reading.

II. Purpose of the research

The purpose of this study is three fold. The first purpose is to provide ideas on the components of reading in order to make a practical test to assess Japanese students' English reading ability. Information will be given from four viewpoints (1) the nature of reading, (2) the theoretical or linguistic underpinnings of reading, (3) the test format of reading, and (4) classroom teachers' ideas based on their teaching experiences. Another purpose is to investigate how much Japanese teachers and native English speaking teachers agree on the construct of reading. The third is to show how the construct of reading that is arrived at in the first step of this research is reflected in an actual reading test, such as the Society for Testing English Proficiency (STEP) test. Translation ability will be closely examined because of its unique status in reading according to Japanese teachers' understanding of reading compared to native speakers' understanding of reading.

III. Research design and methods

Seventy-four experienced English teachers (43 Japanese teachers of English and 31 Native English speaking teachers) out of 200 responded to a reading construct questionnaire consisting of 27 items (See Appendix). The teachers rated the items on a four-point scale according to their degree of importance to reading comprehension.

1. Establishment of Research Instrument

By reviewing the literature on the theory of reading (Clapham 1996; Riley and Lee 1996; Rost 1993; Allan 1992; Perkins 1992), interviewing colleagues and reflecting on my own teaching experience, I established the following three viewpoints on the construct of the components of reading:

the nature of reading (prototype reading)
the linguistic components of reading
the format of reading tests (including text types, response tasks and evaluation of the task performance)

2. Nature of Reading

The nature of reading can be discussed in relation to the five main points of reading. However, we must also consider that reading comprehension occurs within a student's mind and that the only way to test ability is to have students actually do something and then assess the thought process that produced the results. The five main points of reading are as follows:

the ability to understand what we read
From the testing viewpoint, we must take into account the type of text and the level of difficulty to decide what we want students to read.
the ability to connect what we read to what we know
This is related to schema theory. We use our background knowledge to understand what we read.
the ability to put new information into a script we are familiar with
This is similar to schema theory. We use our previous experience or knowledge of a similar situation to understand the situation we encounter through reading.
the ability to relate the information to what we already know
The point here is to try to relate the information to a previous similar experience.
the ability to use linguistic knowledge (grammar, vocabulary, discourse) to understand what we read at an intra-sentence level, inter-sentence level or large context level

N.B. Points 2 to 4 share more or less the same idea, which is the utilization of background knowledge (i.e. the script, the schema) to understand what we read.

3. Linguistic Components of Reading

When we think about how much we understand from what we read, the three linguistic elements of reading (grammar, vocabulary, discourse) will automatically arise because we are dealing with a language test. They are indispensable and they should be paid special attention during test construction.

4. Format (Text, Response Task, Evaluation of the Task Performance)

Format, in a broader sense, includes text types, response tasks and the evaluation of the task performance, and the three of these are closely interrelated. Text types which we have students read should be varied as follows:

textbooks
novels
newspapers
directions or instructions (recipes)

Response tasks through which students show their understanding of the material are as follows:

textbooks
translation
true/false
doze test

In dealing with text and response tasks, we must take into consideration the authenticity of the material and the performance of the students. To do this, we must take into account the crucial features of authenticity. The following are some possibilities:

English should be meaningful
English should be written in a natural tone
Context should be self-evident and as natural as possible. Test takers should be able to share the same experience.

5. Proposed Items for the Construct of Reading for the Present Research

Having considered all of the information above, I proposed 27 possible items for the construct of reading for the present research.

ability to grasp the context
ability to predict what the writer says next
ability to understand English at a discourse level
ability to understand English at a word level
ability to use vocabulary knowledge
ability to use grammar knowledge
ability to utilize schemata (background knowledge)
ability to understand sentence level English
ability to deal with academic material
ability to deal with authentic English
ability to summarize the material in written Japanese
ability to summarize the material in spoken Japanese
ability to summarize the material in written English
ability to summarize the material in spoken English
ability to read editorials in the newspaper
ability to read general articles
ability to read journals critically
ability to appreciate literature
ability to get the main idea of material
ability to understand idiomatic English expressions in the material
ability to read quickly
ability to translate the material into Japanese
ability to get the gist (outline) of the material (skimming ability)
ability to find specific information in the material (scanning ability)
ability to find the topic sentence of a paragraph
ability to draw conclusions from the material
ability to comprehend authentic writing

6. Analysis Procedure

Descriptive statistics were employed to show how much teachers agreed in their rating, a factor analysis and a cluster analysis was adopted to obtain the categorized components of the 27 items. Also a comparative analysis was conducted between the components agreed to by native speakers and Japanese teachers, and between the components of the new construct of reading and the test format of the STEP test.

IV. Results and Discussion

1. Overall Tendency of Rating

Table 1 demonstrates the overall tendency of teachers' rating for each item. Of the 27 items, seven items received scores below the median (2.5) on the 1-4 point scale. This means that all the teachers consider the seven items less important. The contents of these items are as follows:

item 9 -ability to deal with academic material
item 11 -ability to summarize the material in written Japanese
item 12 - ability to summarize the material in spoken Japanese
item 15 - ability to read editorials in the newspaper item 17 ability to read journals critically
item 18 - ability to appreciate literature
item 22 - ability to translate the material into Japanese

As a whole, testing reading ability through Japanese is thought to be less important. Also, critical reading and the appreciation of literature might not be considered an important element of reading ability where testing is concerned. This ability would probably be considered more important as a teaching element.

2. Comparison Between Japanese Teachers and Native English Speaking Teachers

Table 2 shows that with the t-test there are six cases that indicate a statistically significant difference in the mean scores between Japanese teachers and native English speaking teachers. The items are as follows:

item 2 - ability to predict what the writer says next
item 11 - ability to summarize the material in written Japanese
item 15 - ability to read editorials in the newspaper item 17 ability to read journals critically
item 18 - ability to appreciate literature
item 20 - ability to understand idiomatic English expressions in the material

It is interesting to note that in all of these six items the Japanese teachers' rating is higher than the native speakers' rating. It could be possible to say that Japanese teachers tend to pay more attention to the meticulous reading of specialized materials by focusing on idiomatic expressions. As was observed in Table 1 items 11, 15, 17 and 18 were given relatively low points. In Table 2 the statistically significant difference in the mean scores is derived from the fact that the Japanese teachers' scores were higher than native speakers' scores. This indicates that to native speakers reading newspapers critically or appreciating literature is less important as a necessary ability in a reading test. Another interesting point in Table 2 is that both groups of teachers give higher points (above 3.3 in the 1-4 point scale) to item 1, item 19, item 23 and item 26, while both groups give lower points (below 2.6 which is right below the median of the 1-4 point scale) to item 11 and item 22. It may be that both groups value reading abilities such as grasping the context, getting the main idea, getting the outline and drawing conclusions from the material whereas they consider the abifity to translate into Japanese less important.

3. Results of Factor Analysis and Cluster Analysis

Although there are some slight differences observed in the ttest between Japanese teachers and native English speaking teachers in their understanding of the six items, as a whole there are more similarities than differences between the two groups. Furthermore, considering the fact that in many institutions there is a possibility that Japanese teachers and native teachers cooperate with each other, we need to take a good balance between the two groups. When there is a distinctive feature in the idea of reading ability, we should look into the possibility that it is derived from the different ideas of the two groups. Accordingly, we should use the total score (combination of Japanese teachers' scores and native speakers' scores) to run a factor analysis. A six-factor structure seems to be reasonable to explain the components of reading ability.

1. Factor 1 Authentic-Material Reading Ability

item 2 - ability to predict what the writer says next item 9 ability to deal with academic material
item 10 - ability to deal with authentic English
item 15 - ability to read editorials in the newspaper item 16 ability to read general articles
item 17 - ability to read journals critically item 18 ability to appreciate literature
item 27 - ability to comprehend authentic writing

We will name this factor Authentic-Material Reading Ability. This ability deals with students' ability to handle authentic material. Although items 9, 15, 17 and 18 were considered less important in the descriptive statistics there is no harm in categorizing them under the ability to understand authentic material. This ability can be established as being part of reading ability.

2. Factor 2 Japanese Translation Ability

item 11 - ability to summarize the material in written Japanese
item 12 - ability to summarize the material in spoken Japanese
item 22 - ability to translate the material into Japanese

Factor 2 is named Japanese Translation Ability. This ability concerns students' translation ability from English into Japanese. Again, these items 11, 12 and 22 were given rather low points as shown in the descriptive statistics; however this ability can be established as one part of reading ability.

3. Factor 3 Prompt Context Grasping Ability

item 1 - ability to grasp the context
item 3 - ability to understand English at a discourse level
item 21 - ability to read quickly
item 25 - ability to find the topic sentence of a paragraph
item 26 - ability to draw conclusions from the material

We can call this factor Prompt Context Grasping Ability because students must understand the situation and the outline of the material rather cpiickly. This ability requires students to have speed reading ability or fast reading skills in order to understand the context and have an idea of the conclusion of the material.

4. Factor 4 Skimming and Scanning Ability

item 7 - ability to utilize schemata (background knowledge)
item 19 - ability to get the main idea of the material
item 23 - ability to get the gist (outline) of the material (skimming ability)
item 24 - ability to find specific information from the material (scanning ability)

Factor 4 is called Skimming and Scanning Ability. Students are required to use their skimming and scanning ability with their background knowledge to understand the material. This ability seems similar to Factor 3 (Prompt Context Grasping Ability). However, the difference is that in Prompt Context Ability students should read rather quickly to understand the material, while with the Skimming and Scanning Ability the point is whether students can use their script (schemata) when scanning or skimming. In other words, in Prompt Context Grasping Ability speed is essential, whereas in Skimming and Scanning Ability the students ability to adapt background knowledge for skimming and scanning is crucial.

5. Factor 5 Basic Intra-sentence Linguistic Ability (Grammar and Vocabulary Handling Ability)

item 4 - ability to understand English at word level
item 5 - ability to use vocabulary knowledge
item 6 - ability to use grammar knowledge
item 8 - ability to understand sentence level English

We call this factor Infra-sentence Lingusitic Ability. This ability mainly concerns the students' grammar and vocabulary handling ability which they use to understand the use of words and grammar in the material at the word or sentence level.

6. Factor 6 English Summarizing Ability

item 13 - ability to summarize the material in written English
item 14 - ability to summarize the material in spoken English
item 20 - ability to understand idiomatic English expressions in the material

Factor 6 is named English Summarizing Ability since students are supposed to give a summary of the material in English. It is difficult to distinguish between the ability for writing and the ability for speaking. However, English Summarizing Ability can be one element through which students show they have understood the material. In the process of reproducing the material in their own words or summarizing their ideas, they need to know the meaning of idiomatic expressions as well as words used in the material in order to reuse them in their sununary.

So far we have discussed the categorization of items and the content of factors. In other words we have dealt with intra-factor elements. Now we will look at the interfactor elements by examining the mutual relationship between them.

In order to further analyze the relationship among factors, we ran a cluster analysis. Table 4 show the results of cluster analysis.

The result in Table 4 shows one important phase of the translation ability. Items 11, 12 and 22 cluster at an early stage and go straight through to the final stage while other factors get mixed with each other in a complicated manner. This means that the translation ability is considered a quite separate reading ability in the reading test.

To analyze in more detail the content of the translation ability, we will look at two results of cluster analysis in Table 5 and Table 6. Table 5 shows the results of cluster analysis of Japanese teachers. Table 6 demonstrates the results of cluster analysis of native speakers.

The obvious difference concerning the translation ability between Table 5 and Table 6 is that in Table 5 items 11, 12 and 22 cluster at an early stage and go straight to the final stage as one cluster whereas in Table 6 those items, starting as one cluster, are combined with other items such as 2, 9, 15, 17 and 18 at some point before they come to the final stage. Another viewpoint for analyzing this data is that to Japanese teachers the translation ability is always one separate unit whether it is presented with easy or difficult material, while to native speakers the translation ability should go along with other elements such as type of reading materials, for example editorials, literature, journals where students need much critical and high analytic comprehension ability. Native speakers think that translation ability works when teachers want to examine students' deep understanding of journals, editorials and literature. Japanese teachers think that translation ability is quite separate from the other abilities. In other words, translation ability and other abilities are parallel. Judging from this phenomenon, although we have come to a six factor structure for the total scores of Japanese and native teachers, more attention should be paid to handling the translation ability in the real test situation. In other words, Japanese teachers and native speakers cooperatively discuss the effective, practical and reliable use of the translation test when they must make a decision on test construction.

V. Comparison Between a New Proposed Construct of
Reading and the Format of the STEP Reading Test

1. Summary of the Reading Section of the STEP test

There are seven grades in the STEP test and each has its own reading test section. The reading test is also included in the first written test to screen applicants to go to the second speaking test. The written test has two components (1) vocabulary test and (2) reading comprehension test.

1) Vocabulary Test

Students are required to fill in the blanks with words in short sentences.

2) Reading Comprehension Test

filling in the blanks with words in longer passages (1st, pre-ist, 2nd, pre-2nd grades)
filling in the blanks with sentences in short dialogues (3rd, 4th, 5th grades)
Multiple-Choice test in longer passages (all grades)
Japanese summarization (only 1st grade)
Japanese translation (only 1st grade)

2. Comparison between the new construct of reading and the STEP test

In the vocabulary test of the STEP test, applicants' basic vocabulary and grammar ability is assessed. This ability resembles Factor 5 (Basic Intra-sentence Linguistic Ability).
In the first (1) and the second (2) sections of the reading test in the STEP test applicants are supposed to demonstrate their context handling ability. This is similar to Factor 3 (Prompt Context Grasping Ability).
The third (3) section of the STEP test is the Multiple-Choice test which assesses applicants' general reading comprehension (skinuning, scanning, using background knowledge). This is close to Factor 4 (Skimming and Scanning Ability).
The fourth (4) and fifth (5) sections in the STEP test examine applicants' translation ability or summarizing ability in Japanese. This is the same as Japanese Translation Ability.

We have found many similarities between the reading test section in the STEP test and the proposed construct of reading ability in the present research. However, two things which could not be found in STEP should particularly be mentioned. One is that the Authentic Material Handling Ability in the present research did not have its counterpart in the STEP test. This is probably because in the real test situation it seem difficult to use purely authentic material. The point here is not whether it is authentic or not but how much the material is close to the real thing.

The second thing is that Japanese translation in the STEP test is only used for the first grade test. There may be several reasons for this. One is that translation ability can only be required for top level applicants because it is a professional skill. Another is that translation ability is necessary only for those top level applicants. Still another reason is that since the number of applicants for the first grade test is relatively small, it is easy to handle the subjective scoring of summary and translation tests.

All of these issues are related to what we have discussed in the factor-structure construction. The translation ability should be given more importance in classroom test construction and this can be decided variably in each institution.

VI. Conclusion

The following conclusions can be drawn:

1) Six factors were obtained through the factor analysis as follows:

Factor 1 Authentic-Material Reading Ability
Factor 2 Japanese Translation Ability
Factor 3 Prompt Context Grasping Ability
Factor 4 Skimming and Scanning Ability
Factor 5 Basic Intra-sentence Linguistic Abifity
Factor 6 English Summarizing Ability

However, we need to reconsider the relationship of each factor in practice depending on the situation of the individual institution.

There were more similarities than differences in understanding the constructing of items of reading ability between native speakers and Japanese teachers. One thing that should be kept in mind is that the translation ability needs to be given more thought.
Through the comparison between the proposed construct of the present research and the STEP test, four of the components were similar (Japanese Translation Ability, Prompt Context Grasping Ability, Skimming and Scanning ABility, and Basic Intra-sentence Linguistic Ability) whereas two abilities such as Authentic Material Handling Ability and English Summarizing Ability did not have similar components in the STEP text. One interesting finding was that the treatment of the translation ability in the STEP test is quite similar to the idea of the Japanese teachers' construct. Only the first grade of the test requires the translation. This is probably because translation demands special skills and ability in reading and only the top level students can manage it. Another possibility is that the number of applicants for the first grade test is small, so the subjective test is feasible even if it takes time to score the written summary in Japanese. Therefore, an ideal situation might be that if possible every grade test should have a translation test.
Teaching and learning points of reading in class should be reflected in the reading test. However, in testing reading ability the test method does not necessarily agree with the teaching method. When giving a test, we should be aware of what part of teaching is covered in our test and what is not covered. Also, we must clarify the meaning of the test components to the test takers and test users.

References

Alderson, J.C. & Urquhart, A.H. 1985. The effect of students' academic discipline on their performance on ESP reading tests. Language Testing 2:2, 192-204.
Allan, A. 1992. Development and validation of a scale to measure test-wiseness in EFL/ESL reading test takers. Language Testing, 9:2, 101-122.
Bachman, L.F. 1990. Fundamental Considerations in Language Testing. Oxford: Oxford University Press.
Brown, J.D. 1980. Relative merits of four methods for scoring doze tests. Modern Language Journal, 64: 3, 311- 317.
Clapham, C.C. 1998. The development of IELTS: A Study of the Effect of Background Knowledge on Reading Comprehension. Cambridge: Cambridge University Press.
Grabe, W. 1991. Current developments in second language reading research. TESOL Quarterly, 25: 3, 375-406.
Perkins, K. 1992. The effect of passage topical structure types on ESL reading comprehension difficulty. Language Testing, 9: 2, 163-172.
Riley, G.L. & Lee, J.F. 1996. A comparison of recall and summary protocols as measures of second language reading comprehension. Language Testing, 13: 2, 173- 189.
Rost, D.H. 1993. Assessing different components of re&ling comprehension: fact or fiction? Language Testing, 10, 1, 79-92.
Shohamy, E. 1984. Does the testing method make a difference? The case of reading comprehension. Language Testing, 1: 2, 147-170.
STEP (The Society for Testing English Proficiency) 1997. STEP News. No. 374 April 1 and STEP News N. 377 May 1.

APPENDIX

Questionnaire When you assess Japanese students' reading ability in class, how much weight do you put on each category below? Please circle one choice for each category. If you would like to add more items to this list, please feel free to do so in the blank space.