Teaching and Learning Forum 95 [ Contents ]

A critical incident: Reflecting on an evaluation report

Robert Fox
Teaching Learning Group
Curtin University
"Interpretation is important because we act according to what we think things mean" (Tripp, 1993). This paper concerns the questioning of reported events and issues which led to the production of an evaluation paper on teaching science at a distance. Rereading entries in a professional journal written during and after the research process, alongside revisiting data collected from over 80 individual interviews, has led to a new interpretation and has raised a number of questions regarding the influence of various stakeholders involved in the research project.

Introduction

A critical incident "tends to interrupt or bring into focus the taken for granted ways of thinking and doing and valuing, theorising and writing." (Brennon and Green, 1993). It's a moment that sparks perhaps a deconstruction effect - a moment that pulls apart that 'normal' or 'natural' approach to doing something. An incident can appear "to be 'typical' rather than 'critical' at first sight, but (is) rendered critical through analysis" (Tripp, 1993).

The critical incident described in this paper occurred following what Measor (1985) describes as an 'intrinsic ... mid career move'. Or, more accurately, a mid career opportunity which arose during an Outside Studies Program (OSP) at the Open University's Institute of Educational Technology (IET), UK.

The critical incident

Based on research conducted at the Open University, I wrote an evaluation report (Fox, 1994), on the home experiment kit (HEK) which offered practical and experimental activities to first year university students, studying science at a distance.

Rereading the evaluation report and raw data, several months after completing it, I came across a number of issues which at the time of doing the research and writing it up, I had either deliberately or subconsciously ignored. In part, I feel this was due to my attempt to come up with a clear, decisive report, which steered me to abandon various approaches to writing as well as disregarding many significant chunks of information that did not neatly fit into the report narrative I had ultimately chosen.

'The very language we use creates frames within which to realise knowledge.' (Walker 1992). In the introductory section of the report I had used the words 'researcher' to describe myself and 'informant' to describe those with whom I had discussions. In the context of my report, I now consider these terms were loaded with a meaning and a bias that did not accurately reflect the way the investigations were carried out.

These thoughts on the selection and use of words has lead me to question a number of other stances I took in the writing up stage of the report that at the time I saw as 'normal' or 'natural'.

First impressions

On the first day I arrived at the Open University, I was given a letter from the Deputy Director, IET, in which he proposed that I become involved in evaluating the role of the home experiment kit in an Open University course.

At first, I saw this request for involvement in an evaluation study as something that would get in the way of the main task I had set myself, in that it would restrict the time I had to investigate more broadly, Open University courses and courseware and whether the Open University materials and delivery strategies were used in other UK higher education institutions.

I felt too that the Open University was keen to get it's 'pound of flesh' for granting my attachment and I sensed that the IET staff were in general not overly enthusiastic about receiving visiting lecturers. An extract from my journal describes my initial perceptions:

...with tighter budgets and less time to chew the fat, visitors may be considered pests. They come here, soak up the hospitality, resources, staff time and do little or nothing in return. ....

....There's no real space anymore for visitors. They have to share existing staff offices, and this in itself can be an embarrassment. Yesterday, for example, Derek, whose office I'm sharing, had to turf me out in the afternoon, as he had a counselling session. I then had to find another office for the rest of the day. But there is much innovation going on here that I hope to tap in on. I've sent some, hopefully friendly sounding messages through E-mail as a soft introduction (hopefully intimating that I won't be expecting too much of their time) and intend to follow up early next week with a phone call. Phone calls are very hit and miss since in general academic staff aren't at their desks or aren't in at all: rather, they work from home. However, I hope people read their E-mail!

Sample E_mail reply re my request to meet people.
Thank you for your message. I am very sorry that over the next two months I have wall-to-wall visits and meetings, and no spare time for such luxuries as talking to an interesting and experienced visitor. That is the nightmare of current academic life. I do apologise. The best I can suggest is to look at the papers in the PLUM and CITE collections that are relevant...... If you are still around in September, I look forward to meeting you then.
D...

I felt on reflection that the HEK project could work to great advantage as it would give me access to people, both staff and students I would have no chance of meeting without such an Open University focused task. I therefore decided to accept the project.

Writing for a particular audience

I was conscious of wanting to produce a report that was, as I saw it, acceptable to the Open University academic community. I therefore modelled the report rhetoric on other Open University reports I had read.

I was also conscious of being an outsider, who was seen by certain key academic staff at the Open University, as someone who would be able to produce a concise, relatively unbiased evaluation report with a clear list of recommendations. I note in my journal:

......I was asked to include a summary page of recommendations and to keep the report short and clear. The report clients ....... wanted a report to help them make decisions ....... and .... that they could use as ammunition for supporting their arguments.
At the point of completing the investigation and starting to write up the report, I had no clear set of recommendations or statements about the role of the HEK. In fact I felt, as Hammersley put it, that the research study 'was a voyage of discovery' and 'much of the time was spent at sea' (1984). I felt I could only sensibly report on a variety of opinions from those I had spoken to and to include my own opinions too. In fact while I was involved in the investigations, I kept changing my opinions and this I've noted in my journal.
Changing directions and positions
I started out looking at the HEK to see if there was something that could be done to make the kit more efficient. Perhaps more condensed. There must be something in there that can be taken out, changed to some other technology or even supplemented by an alternative delivery. But I always envisaged that the kit would still be there: still be in existence. Now I'm not sure it can stay in any form. I'll have to see what Ruth .... thinks when I go down to Bristol tomorrow. Why can't ordinary household goods and everyday articles and phenomena replace the traditional chemistry kit? There's chemicals in the home and by using everyday goods, students would become less daunted by experimenting with familiar items. It would also bypass any health and safety rules and restrictions with taking the HEK across international borders and make the course perhaps more real, by using everyday things.

Doing interviews
I'm a bit like a ping-pong ball, going from one side to the other, depending on who I talk to. Having spoken to Ruth ....., Alan ..... and Phil ..... in Bristol, I'm no longer totally convinced that Tim's vision of the kit being entirely removed is the right choice. These are the people who teach the course to the students in the regions. They're not at Milton Keynes developing courses, researching, writing learned papers....... they .. teach the students. Is it really realistic to expect computers to completely replace the kit? I'll need to listen to the tape recording of the interviews again to get some detailed arguments from the other side. It seems it may be worth my while in arranging interviews with other key people -stakeholders, I've already spoken too. The topics I want to cover in the interviews are changing and I'd be interested in re-visiting some earlier discussed areas.

In the end, following the submission of a first draft report, it was recommended to me that I re-write the report, making it shorter, more decisive with an action plan of recommendations. I therefore reread my data findings and very selectively took out parts of recorded conversations and other data that did not reinforce this clearer more black and white report style I had been asked to adopt. In a journal entry, I start to question my approach by comparing what I did with the criticisms levelled at Mead.
An article about Margaret Mead's book, 'Coming of age' has given me further food for thought in this critical incident activity of revisiting the HEK evaluation study.

How honest were Mead's informants? Was Mead as Freeman suggests deliberately duped by Samoan adolescents into believing Samoans readily accepted the idea of free love? Was I, duped too in part, strongly influenced by senior staff at Milton Keynes? Was Mead's 9 month visit too short a period to really get to grips with Samoan society? And was my 3 months investigation into HEK equally too short to be very meaningful?

Did Mead impose ' her ideology on the evidence' and did I impose mine on the HEK evaluation? Were my interests in the use of alternative delivery strategies compared to the use of conventional ways of teaching leading me to favour reviewing whether individual activities within the existing HEK could be carried out through the use of other instructional strategies and media?

Was Mead too narrowly focused (too soon) in her topic/problem and was I? And was I like Mead even admits of her anthropological tutor, 'always tailoring a piece of research to the exigencies of theoretical priorities?'

Robin Fox, an anthropologist quoted Mead as having said of her tutor, 'He told us what to look for and we went and found it.' Having been told what to say by at least one key Open University figure that the new course had to have a PC incorporated ..... Could I have followed along the same path as Mead?

Mead wrote her study based on interviews with 68 girls (no males). Most of her information came from 25 girls, who came to visit her each day for questioning in her office. Was the sample too small to be representative? Did I have too many interviews of not enough depth to get significant differences out of individuals and was there insufficient grounds for me to create generalisations?

Freeman's strongest criticism is that Mead's 'falsifications of Samoan life were not accidents but the result of imposing her ideology on the evidence.'

Have I, in fact, imposed mine on the report? ....... So what's important is to make the nature of the activity and its ambiguities and contradictions as explicit as possible and to state my position as clearly as I can from where I stand at a particular point in time.

John Leo (1993) says 'anthropology is a subjective enterprise with the observer drawing unity out of confusion, by imposing patterns that are often arbitrary'. Have I done the same with my burning need to come up with a set of neat recommendations?

And finally, as Leo says 'some natives cheerfully tell fieldworkers whatever they want to hear'.

Were my interviewees telling me what I wanted to hear? Certainly, at times, I'm sure they were and I encouraged them to do so by: leading questions, nods of encouragement of statements I liked or wanted to hear. In fact it is seems that I adopted several strategies to acheive my unconscious aims. Maybe next time, it would be interesting to video record myself interviewing.

Own interests My interests in the use of alternative delivery strategies to established ways of teaching led me to favour reviewing whether individual activities within the existing HEK could be carried out through the use of other instructional strategies and media. This interest influenced doing the research and my recommendations to the Open University Science Faculty, Planning Committee: a key targeted client for this research project. In a journal entry, I start to discuss this point.
....... with some time lapsing after actually doing and writing up the research, it's interesting to reflect on what's happened or rather, my interpretations today of what's happened. I'm not sure how 'honest' I was in the reporting of the 'findings'. I had my own agenda, which was favouring finding suitable alternatives to the present way of offering practical and experimental work through the HEK. I don't think I was 'hygienic' in the sense that Ann Oakley (1986) describes in her chapter on interviewing women. If I was to write this report again, I know it would be quite different.....

......... Rob Walker's comment (1992) that writing about the past, one does with very much the issues of the present in mind. I'd agree with this totally. And it's not restricted to writing but to all life's events. David Hamilton (1992) went on to say .....'you can always go on writing history of the same period, because your questions are always changing.'

References

Brennon, M. & Green, B. (1993). Critical reflective writing. Geelong: Deakin University.

Fox, R. (1994). Teaching practical and experimental work: evaluating alternative delivery strategies. In Summers, L. (Ed.), The proceedings of the Teaching Learning Forum '94. Quality in teaching and learning. 'Making it happen'. (pp. 103-115). Perth, Western Australia: Edith Cowan University.

Hamilton, D. & Holly, M.L. (Speakers). (1992). Research methodology. (Cassette recording No. M1044). EdD901 Professional doctorate. Geelong, Victoria: Deakin University.

Hammersley, M. (1984). The Researcher exposed a natural history. In R. G. Burgess (Ed.), The Research Process In Educational Settings: Ten Case Studies. London: Farmer Press.

Leo, J. (1983). 'Bursting the South Sea Bubble. An anthropologist attacks Margaret Mead's research in Samoa'. Reported by Attinger, J.: Boston and Dunn, J.: Melbourne in Time, February 14, pp. 50-52.

Mearor, L. (1985). Critical incidents in the classroom: identities, choices and careers. In Ball, S.J. & Goodson, I.V. (Eds.). Issues in education and training series: 3. Teachers' lives and careers. London: Falmer Press.

Oakley, A (1986). Interviewing women: A contradiction in terms. In Roberts, H. (Ed.), Doing feminist research. New York: Routledge.

Richard Rorty (1980:203). Philosophy and the Mirror of nature. Princeton, New Jersey: Princeton University Press.

Tripp, D. (1993). Critical incidents in teaching. Developing professional judgement. London: Routledge.

Walker, R. (1992). Finding a silent voice for the researcher: using photographs in evaluation and research. In Schratz, M. (Ed.). Qualitative voices in educational research. London: Falmer Press.

Walker, R. (1992). (Speaker). (1992). Journal writing in research. (Cassette recording No. M1011). EdD901 Professional doctorate. Geelong, Victoria: Deakin University.

Please cite as: Fox, R. (1995). A critical incident: Reflecting on an evaluation report. In Summers, L. (Ed), A Focus on Learning, p84-88. Proceedings of the 4th Annual Teaching Learning Forum, Edith Cowan University, February 1995. Perth: Edith Cowan University. http://lsn.curtin.edu.au/tlf/tlf1995/fox1.html


[ TL Forum 1995 Proceedings Contents ] [ TL Forums Index ]
HTML: Roger Atkinson, Teaching and Learning Centre, Murdoch University [rjatkinson@bigpond.com]
This URL: http://lsn.curtin.edu.au/tlf/tlf1995/fox1.html
Last revision: 16 Apr 2002. Edith Cowan University
Previous URL 16 Feb 1997 to 15 Apr 2002 http://cleo.murdoch.edu.au/asu/pubs/tlf/tlf95/fox84.html