Primary Listening and Writing Tests

The design of the listening test for Primary Three and the listening and writing test for Primary Five was modelled on exemplar listening tasks developed by the TOC Assessment Section of the Curriculum Development Institute (GDI) and based on the Assessment Guidelines for Key Stages 1 and 2 (Education Department, 1998). The GDI exemplar tasks are developed by a moderation committee consisting of GDI staff, serving primary school teachers and academics from local tertiary institutions. The overall consultant is Professor Lyle Bachman of the University of California. After moderation the exemplar tasks are tried out in primary schools and modifications made according to difficulties encountered. By the time they are made available as exemplar tasks therefore they are reasonably well developed already.

Components of a number of these exemplar tasks were used to create a test which would serve as the basis for the two tests. The overall test design is the same for the two levels and is closely based on the exemplar tasks with the addition of a new overall context and one new task. The tests are thematic in content built around a situational context hypothesised to be within the range of experiences of primary pupils at the two levels. The component parts of the test are distinct and designed to assess diffd-ent skills within the overall construct of listening. A detailed breakdown of the skills assessed by the test is shown in the specification grid below:

40

Primary Listening Test; Tasks and Hypothesised Skills Assessed Parti

Part 2

PartS

Part 4

PartS

Part 6

Task

Completing a weather chart with numerical information

Selecting from six pictures, the one which represents a given weather forecast given in an extended monologue.

Labelling a diagram with the names of participants according to descriptions of they activities they are performing given in an extended monologue.

Labelling pictures according to descriptions given.

Colouring pictures according to information given.

Selecting from four pictures, the one which best matches a given description.

Hypothesised skill assessed

Intensive listening for specific (numerical) information.

Extensive listening retaining relevant pieces of information to form a coherent summary.

Matching graphical with aural information.

intensive listening for specific information relating to actions and activity involving identifying relevant lexis and matching it with graphical information.

Intensive listening for specific descriptive information involving identifying lexis and matching it with graphical information.

Listening to information given in an extended monologue and retaining relevant pieces of information to form a coherent summary. Matching graphical with aural information.

Intensive listening for specific descriptive information involving identifying lexis and matching it with graphical information.

The difficulty levels of the tests for the two levels of class were differentiated by means of differences in the tape script, in the task demands, and in the weighting accorded to different sections. The test for Primary Five included a writing component which was related to the thematic context of the listening tasks and asked pupils to describe related experiences they themselves had had.

(B) Piloting

Each individual component of the primary listening and writing assessments had been piloted as part of the process of development within the GDI, TOC Assessment Section. Placing the components together into a listening test for primary three and a listening and writing test for primary five necessitated additional piloting. The test was therefore piloted on two classes each of primary three and primary five pupils in two schools, one NET and one NNEX

Sections of the primary test were found to be rather easy and modifications were made in the graphics and the tape script to ensure greater discrimination between pupils.

(C) Reliability

(0 Reliability of the Primary Listening Test

The time constraints of the project meant that the reliability of the revised instalments could not be estimated except in terns of internal consistency. Moreover, the internal consistency of the refined test could not be estimated prior to actual implementation. The internal consistency of the refined Estraing tests was estimated on each administration using Cronbach's alpha yielding the results presented below.

Internal Consistency of the Primary Listening Tests 1* Cohort

1st Administration a =0.6406

1st Cohort 2nd Administration a -0.5781

2^nd Cohort 1st Administration a =0,6933

2™ Cohort 2^nd Administration a =0.5895

As shown in the table, the alpha scores range from 0.57-0.69, This suggests that the internal consistency of the test on various administrations range from marginally unacceptable to just acceptable. An alpha of less than 0.6 would be considered to indicate that the results produced by the various parts of the test are sufficiently unrelated to suggest that they are assessing a different construct.

An alternative interpretation might b e that the c onstruct of listening ability involves skills sufficiently distinct to mean that a test of the various skills would not display internal consistency. However, within the constraints of the evaluation study, there is not sufficient time to explore this interpretation. It was therefore decided to conduct further analysis of the Primary Listening test data in order to investigate the reasons for the low internal consistency of the test when administered on different occasions. The analysis will focus on the scoring of the test papers to identify ways of conducting a more fine-tuned scoring allowing for partial credit to be given for items which had been initially scored in a holistic manner i.e. Parts 2 and 6 in Table 1 above. The* results of this analysis will be discussed in the Final Report. At this stage it is sufficient to note that the preliminary results indicate that extreme caution needs to be exercised in interpreting the results of the primary listening tests.

(ii) Reliability of the Primary Writing Test

The writing test for primary five was a piece of continuous writing in which pupils were invited to describe their personal experience of the thematic content of the listening test they had just completed The writing was scored holistically according to a set of criteria adapted from the published criteria used for the Preliminary English Test (Cambridge, 1997) and from those provided in Millar (1995). In addition, the tests were scored for number of error free clauses, and number of words in error free clauses (Ishikawa, 1995). All scripts were double marked by a second marker. Reliability in the use of these various measures was ensured by means of a five-stage series of measures:

i. detailed training of markers in a briefing session where assessment criteria were explained and scripts were jointly analysed and scored according to the criteria;

ii. requiring all markers to score a sample of scripts acceptably prior to their final appointment as markers;

iiL encouraging markers to discuss ambiguous cases within peer groups and to raise queries with flie evaluation team;

iv. checking to identify examples of scoring which did not match up with the criteria and the causes of misunderstanding leadmg to these problems;

v, briefing of second markers on likely causes of misunderstanding leading to problems in scoring prior to their second marking of the scripts

Through this process we are confidant that the scoring of the writing test was consistent across markers and hence that reliable influences can be drawn from the data.

42