Stepping off the Treadmill for a Change

Stepping off the Treadmill for a Change –

A focused diagnostic analysis of literature exam scripts

John Goodall

Former teacher, currently an adviser in the DBE in PMB

Introduction

This article describes a method of analyzing Grade 12 English Home Language literature exam scripts with a view to pin-pointing a manageable number of academic skills that underperformers may need to master in order to perform at their level of competence.

This analysis focuses only on literary essays written by KwaZulu-Natal Grade 12 candidates for the November 2008 English Home Language literature exam paper. It focuses purely on performance rather than on the question paper itself. This selective approach aims to produce not only a manageable teaching agenda, but one that serves as a check-list against which one can note results.

The analysis could be described as an example of action research because it is conducted during the course of one’s normal schedule of duties. It displays what I believe to be the limitations of action research; namely, an absence of any literary review on the subject; limited resources; i.e., raw material available for analysis, and a breadth of analysis circumscribed by the amount of time available. In addition, results from this qualitative analysis of a small number of sample texts are not supported by evidence that is statistically significant.

On the other hand, I believe teachers will find both the method of this study and its results a productive resource against which to compare academic skills that under-achievers in their classes may need to learn.

The problem stated

Traditionally analyses of data from exam scripts reflect performances of learners across the performance curve. This means that data will coincide with performance descriptors across all ranges of a marking rubric. As a result, examiners’ reports tend to change very little year to year.

The following examples of examiners’ reports illustrate this:

Candidates need to be reminded that paraphrasing or quoting sections of text does not constitute critical analysis unless it is supported by critical comment. (Comment from 2003 English Home Language performance – on the Higher Grade. Reference: KwaZulu-Natal Department of Education Circular 32 of 2004.)
Most candidates had difficulty writing a critical response to poetry and paraphrased the poem instead. (Reporting on performance in the 2008 National Senior Certificate English Home Language (Paper 2 – literature) Exam. Reference: Minutes of KwaZulu-Natal English (home language) subject advisers’ meeting of 22 January 2009.)

Given the graph of performance in an exam, one would be surprise not to find such comments for any literature paper. One can conclude from this that a rubric serves as much as a marker’s guide to placing performance in ranges, as it serves to predict the content of an examiner’s report on any given literature exam.

So why is there no improvement in candidates’ performances? How do we tell if there is an improvement?

I contend in this article that, while the desire for improvement motivates writers of examiners’ reports, these aim rather to comment on the curve of the performance graph – a broad description – rather than to plan improvement. Examiners’ reports generally do not select particular areas where improvement can be expected from a particular range of candidates, from whom this improvement can be expected. Traditional examiners’ reports desire improvement but cannot be said to plan it.

The analysis in this article focuses on two performance ranges and aims to make comments that will assist in teaching “the next thing” that some of the learners in these ranges need to focus on in order to improve their performance. Where, for example, a teacher finds that top-performing learners tend to bunch within a range, say, for example, rating code six (symbol B), and that an unexpected low number achieve scores in the next range, this article may well assist such a teacher in identifying the feature that appears to prevent at least some of the candidates in this range from producing a rating code seven (symbols A or B) performance. This article pin-points academic skills, which if mastered, will probably enable these candidates to break through to the next level of achievement.

The approach used

Three stages summarise the way that the problem was dealt with:

Analyse responses to essay questions from candidates with rating code seven (symbols A & B) scores, and with rating code five (symbol C) scores. (Focusing on the literary essay allowed a speedier global analysis of responses from a greater variety of works than an analysis requiring reference to marking memoranda for contextual questions.)
Use the current rubric for marking essay questions for a novel and drama as the basis for comments on candidates’ responses. (This rubric came as part of the Department of Education’s (2009) Guidelines for the Setting of Grade 12Examinations in Languages.)
Determine a teaching and learning agenda for current matric candidates with ratings similar to those in the sample, and particularly for candidates who might be expected to push their performance to the next level of achievement.

It is noted that because of the limited time available to complete this study, and the number of scripts available, this analysis can at best be described as illustrative rather than comprehensive. However, the approach could serve as a template for teachers to conduct their own analysis.

Procedure

1. Select essays from the whole range of prescribed works from candidates with rating code seven (symbols A & B) scores, and with rating code five (symbol C) scores. (These rating levels are referred to as 7A, 7B, and 5C respectively, in the tables below.)

2. Read each selected essay and write notes on each candidate’s performance in from one to three lines. Use the rubrics as a prompt for these comments.

3. Compare essay scores with each candidate’s overall score for the literature exam paper.

4. Once all essays have been analysed and comments made, use these to articulate common tendencies, especially tendencies that could point to the next step in the candidate’s learning, and tendencies that are likely to result in an improved performance.

Results: Comments on Performance

The November 2008 literature paper asked essay questions on Shakespeare’s Hamlet, Tsitsi Dangarembga’s Nervous Conditions, and Marguerite Poland’s Shades. (Note that the ‘Notes’ in the table below are the comments on the markers.)

Comments for 7A and 7B candidates
	Total out of 80 & % for Paper	Questionselected	Total out of 25& %for essay	Notes
1	7898%	5.1Hamlet	2392%	Individual turn of phrase – fluent/ extensive, rich vocab – lots of detail – explains comments – one paragraphing error
2	7594%	3.1Nervous	2392%	Checks each point made; e.g., “but she was an exception” – looks at implications – lang. errors do not interfere with the message.
3	7594%	4.1Shades	2392%	Clarity of argument – detail for both sides of argument – lots of prep notes
4	7594%	5.1Hamlet	2080%	Not as fine a sense of continuity, sentence to sentence [as other 90% people]
5	7493%	5.1Hamlet	2392%	Exposition on the turning-point elaborated using text’s imagery [garden] – picks up fine points – extensive support from text.
6	7391%	5.1Hamlet	2392%	Complex argument – succinct – good par. links/ transitions – generalizes & articulates implications – presents alternative views – seemless
7	7391%	5.1Hamlet	2392%	consistently refers to elements of Q in essay – lots of examples [I think too many lang errors for 90%.] – [avenge – revenge]
8	7391%	5.1Hamlet	2080%	Links points made with Q set; viz., recursive
9	7088%	5.1Hamlet	2288%	Analysis of the dilemma – 80% less succinct – more likely to repeat statements – exploration with details
10	7088%	5.1Hamlet	2080%	Argument easy to follow – prep notes evident – has read over essay [amendments]
11	7088%	5.1Hamlet	2080%	Picks up on distinctions / fine points – picks out threads of an issue – clarity of presentation
	Immediate observations: 1 – 5: sprinkling of lang errors, or awkward wording probably result of writing under exam conditions. Why did some not get 100%? All do one essay only. [Should more be considered possible?]
	: 90s : not much that needs to be improved in this range, and considering a variation of 5% allowed in scoring, there is little difference between 95 and 100%. A seemless sense of continuity
	: 80s: evidence that concentration not as sustained as 90s. Explores by commenting on details, but less succinct (than 90s) & more likely to repeat statements
	Observation on rubric re: “understanding of genre”. How does one measure this? What signs demonstrate this? – [put these in rubric]

Comments for 5C candidates
	Total out of 80 & % for Paper	Questionselected	Total out of 25& %for essay	Notes
1	5468%	5.1Hamlet	1768%	Good points – clear packaging of ideas in paragraphs – loses focus half-way through, e.g., 5/13 par on Claudius. Tends not to link comments to Q; viz., not good at tracking ideas
2	5468%	4.1Shades	1456%	[I would not have given this more than code 3/E, 40+% – faulty paragraphing & prior planning]
3	5366%	5.1Hamlet	2288%	Concise – [I would not have given this 88%] – not get all points required – [I would say, top 60s] – tends to lose focus after half-way.
4	5366%	5.1Hamlet	2080%	Way too much time on essay Q – not succinct – [I would have given this 65%] – 2x lapses in logic, e.g., points juxtaposed rather than linked. – paraphrasing lapse – reliance on story line
5	5366%	3.1Nervous	1872	Too much prep detail – has clarity of argument & lots of detail.
6	5265%	4.1Shades	1560%	Misses large part of Q – not see an alternative view – marker’s comment, “Writes well when on topic.”
7	5164%	5.1Hamlet	1248%	Too short by half a page – discussion on-the-line – a 60-percenter who is poor at essays [odd!]
8	5063%	5.1Hamlet	1664%	Does order ideas using paragraphing – some fatuous comment (“Hamlet is a play with deep meaning”) that does not advance an argument + dubious/ incorrect/ unconvincing conclusions (“Hamlet’s religious beliefs began to change”) – tends to lose focus on Q + argument – good prep notes
9	4961%	5.1Hamlet	1456%	Useful rudimentary introduction – too much detail on prep – essay a bit short [ran out of time?] – Picks up on the theme and character details but not able to articulate these – tends to repeat unnecessarily
10	4860%	5.1Hamlet	1664%	Bit short maybe – not all the points [could plan & tick once done?] – tends to use rambling (rather than subordinated) sentence structures – lapses in logic.
11	4860%	5.1Hamlet	1352%	Pretty glaring lang & spelling errors (relegion for religion) – on-the-line discussion
	Immediate observations: candidates 1 – 7: some need for general spelling remediation. Comments with “tends to” or “lapses in” could indicate area for this grouping of candidates (especially under-performers in the range) to work on to move up to 70s. Planning = time on Q; too much detail in plan; not all required points responded to. Considerable ability, but performance compromised by fairly technical features – spelling, planning. The capacity for clarity could advance the score to the 70s even with faulty lang usage.

Results: Whole Paper and Essay Question Scores Compared

Columns on the left reflect percentage and raw scores for the whole of paper 2 (literature). Columns on the right contain the percentage and raw scores for the essay question.

Paper 2 (P) vs Essay Question (Q) scores: 7A & B Candidates
7A
P %	x/80	Q	Q %	x/25
98	78	5.1	92	23
94	75	3.1	92	23
94	75	4.1	92	23
94	75	5.1	80	20
93	74	5.1	92	23
91	73	5.1	92	23
91	73	5.1	92	23
91	73	5.1	80	20
7B
88	70	5.1	88	22
88	70	5.1	80	20
88	70	5.1	80	20

When one compares the relative performance (measured in percentages) in the essay and in paper 2 as a whole, one finds that five candidates’ scores for the two lie within a range of 5% of each other. (For these candidates there is little difference between the essay score and their performance in the literature paper as a whole.) The six remaining candidates achieved a percentage score lower on their essay than that achieved for the whole of paper 2.

Paper 2 (P) vs Essay Question (Q) scores: 5C Candidates
P %	x/80	Q	Q %	x/25
68	54	5.1	68	17
68	54	4.1	56	14
66	53	5.1	88	22
66	53	5.1	80	20
66	53	3.1	72	18
65	52	4.1	60	15
64	51	5.1	48	12
63	50	5.1	64	16
61	49	5.1	56	14
60	48	5.1	64	16
60	48	5.1	52	13

When one compares the relative performance (measured in percentages) in the essay and in paper 2 as a whole, one finds that six candidates’ scores for the two, lie within a range of 5% of each other. (For these candidates there is little difference between the essay score and their performance in the literature paper as a whole.) Of the five remaining candidates, three scored a higher percentage on the essay than for paper 2 as a whole, and three scored a lower percentage on the essay than for paper 2 as a whole. (This point is commented on later in the article.)

Summary of comments and conclusions for teaching: 90s and 80s ranges

As indicated earlier, expressions such as “tends to …” and “lapses in …” may well point to features for candidates to focus on in order to improve their performance. Secondly, the collection of the comments for the selected range of candidates provides a general profile of their performance. In the main this profile will form part of the rubric. A likely difference between rubric descriptors and a marker’s comments is that the latter will include common tendencies in addition to those described in a rubric.

Candidates in the 90s range invariably presented a seemless sense of continuity; possessed a clarity of argument and expression, and were likely to articulate more than one view of a situation. They were likely to see two sides of an argument. Preparatory notes appeared to be a feature of this range. (While some exam scripts showed evidence of preparatory notes, one can assume that some candidates used their question paper – rather than script – for preparatory notes.) Another feature of this group was evidence of reading over responses to questions. This evidence came in the form of corrections and changes to their essays.

The 80s group shared the capacity for detailed exploration of the subject with the 90s group. However, the former group tended to be less succinct and more likely to repeat statements. It seems as though this group is less able to sustain concentration.

Consequently, for an under-achieving learner in the 80s range, or for a learner at the top end of this range expecting to score in the 90s, the following may well be features to focus on in order to achieve this goal:

The candidate needs practice in writing succinctly (at both the initial writing and editing/ reading over stages).
This range of learners needs to seek out and correct any repetition of points while drafting and editing work. Practice in tracking the progress of their argument by ticking points in their preparatory notes may achieve the same goal.
After having driven home a point of view, or an argument, the candidate needs to consider using expressions such as “However, …..”, or “This having been said, …..”, because they will alert the candidate to an alternative view, or another side to an argument.

Summary of comments and conclusions for teaching: 60s range

This range comprises learners whose considerable ability is often undermined in the exam by technical inefficiencies. This study finds that most of these technicalities relate to planning. Academic skills to do with planning need to be “sold” – particularly to under-performing candidates who score between 65 and 69%, because they probably have the most to gain from using these skills. These candidates need both to set and adhere to time limits in tests and exams. A candidate who includes too much detail in preparatory notes for a response clearly signals the need for better planning skills. Some scripts reveal that the candidate spent far too much time on a particular question. This appeared to have left them insufficient time for completing other parts of the paper. Observations made here on candidates’ ability to focus and to concentrate may also point to the need for improved planning within an exam, or test. Candidates, for example, write preparatory notes, but then fail to use them in their essay. The marker’s comment, “Writes well when on topic” characterizes the dilemma common to this range of learners; namely, that the candidate’s ability to learn is often compromised by faulty performance. It is interesting to see that there are both positive and negative comments relating to the clarity of argument in this group, as well as positive and negative comments on the use of paragraphs. Difficulties in using paragraphs successfully, and in developing clarity of argument may well be connected to the ability to use preparatory notes effectively.

For under-achieving learners in the 60s range, or for learners at the top end of this range expecting to score in the 70s, the following agenda for learning may well assist in achieving this goal:

These candidates need practice in setting time limits for writing preparatory notes (both for assignments and in tests), and they need to practise jotting down these notes within this time limit. (Initial plans for an assignment could be written down within a time-limit appropriate for a test. Learners could then discuss the efficacy of a sample of these.)
Candidates need practice in writing out time limits for an entire (sample, or past) exam question paper. (It may be useful to have some form of discussion on ideas that are presented.)
Candidates would benefit from practice in analysing questions so that all aspects of the question are included in preparatory jottings.
Learners in this performance range should practise keeping to the plan, and practise checking that items have been included in the essay. (Tracking the use of items in a plan is difficult to teach. However, having learners design a post hoc flow chart, or concept map on a given essay, and then comparing notes, is one way of achieving this. Heather Meyer (2005) reports success in this method of teaching writing.

Conclusion and Further Questions about Performance in Exams

A small number of exam scripts (22 in all – 11 of the highest ranges and 11 for the 60 percent range) were analysed with a view to determining an agenda for best items to learn in order to improve scores. Three key items were suggested for learners in the top 80 percent range, and four key items were suggested for learners at the top of the second range. Clearly a teacher would need to confirm that the comments mentioned here apply to candidates scoring within these two ranges at his or her institution.

When one considers comments such as “concentration not as sustained as 90s” one wonders at which point a candidate scoring, for example 86%, is achieving optimally for his or her ability, and is not likely to be a candidate for improvement. While teachers often have an unerring sense of candidates achieving to their ability and also those who do not, how does one test this instinctive sense? A description of demonstrable characterizing features could be used to confirm a teacher’s instinctive opinion. Such a list could be as motivating for a learner capable of improving on performance as it would for those who perform optimally. One wonders whether a teacher’s knowledge of learners is solely responsible for determining these features, or whether indicators could be identified independently by viewing test, exam, or assignment scripts.

A final question. Given the generally rough correlation between essay scores and overall exam scores for the middle order (60s range), one wonders whether comments for this range of candidates provide an equally useful guide on their performance in contextual questions in a literature exam.

A final comment. While only the readers of this action research article will judge whether findings (expressed in diagnostic comment) are accurate or not, and whether conclusions for a teaching agenda prove successful or not, I am of the opinion that a focused approach to analysing performance in exam scripts is likely to produce a focused knock-on effect in teaching. In addition, readers may find that the model of analysis used here is easily replicable for other ranges of scores and consequently may open a door to progress for under-achievers in these other ranges. By seeking an alternative to the generic examiner’s report, one finds oneself inevitably stepping off the treadmill of current exam reporting practices. In doing so, one is probably more likely to find clearly articulated and manageable ways of directing the way learning can be achieved – a rewarding experience for any teacher.

References

Department of Education. Rubric for Marking the Essay Question for Novel and Drama. Guidelines for the Setting of Grade 12 Examinations in Languages. 2009. Page 21.

Heather Meyer. 2005. Analysing argumentative writing. English in Education vol.39 No.3 pages 78 – 92.

KwaZulu-Natal Department of Education Circular 32 of 2004. Examiners’ Reports: November 2003 Senior Certificate Examination. 23 March 2004.

Unpublished. Minutes of the English (Home Language) Provincial Advisory meeting held on 12 and 13 January 2009 in the Richard’s Hotel in Richards Bay.

Categories: Volume 3

Assisting teachers to teach effectively Tips on building your vocabulary