Illustrative case study two: the evaluation of the 1990-91 Grampian Primary Industry Project.

During he 1986-87 session, an ambitious project took place in Scotland's Grampian Region as part of its 'Industry Year' programme, a project that involved primary schools in all parts of the Region learning about industry through the medium of problem-based learning. During the 1990-91 session, this was followed up by an even more ambitious project - an attempt to give primary pupils an insight into the 'business' side of industry through the medium of simulation/gaming and by forming links with local industrial and commercial organisations. One of the authors (Henry Ellington) was heavily involved in the planning, organisation and implementation of this project, and also carried out an in-depth evaluation of the project on behalf of Grampian Region. Although it does not relate directly to tertiary-level teaching, this evaluation will   be examined in detail, since it illustrates many of the points made in this section and shows how the eight stages described above can be put into practice. It should therefore prove helpful as an illustrative case study on how evaluation should be carried out.

Stage 1 : Establishing the overall purpose of the evaluation. The 1990-91 Grampian Primary Industry Project involved 32 teachers in 26 different primary schools running business-related projects with their classes over the winter and spring of 1990-91. The teachers were given considerable leeway over the exact nature of their projects, but were told that they had to involve the use of games or simulations of some sort, and had to involve the establishment of links with at least one local industrial or commercial organisation.

The teachers were briefed on the project through a one-day workshop held in November, 1990, and spent the next six months planning and implementing their projects, under the guidance of staff from Grampian Region's School/Industry Liaison Service and The Robert Gordon University. The project reached a climax in May, 1991, when all 32 participating classes displayed their work at an exhibition mounted in one of Aberdeen's Education Centres. This also incorporated a conference for the participating teachers, a conference at which it was decided that a full-scale evaluation of the project should be carried out.

It was decided that this evaluation should concentrate on three specific aspects of the project - (i) its overall educational effectiveness, (ii) the use that had been made of games and simulations and how effective these had been, and (iii) the use that had been made of external industrial/commercial links and how effective these had been. The evaluation would also elicit detailed information about the work that had been done in the different schools (how many pupils had been involved, how long they had worked on the project, etc) and seek advice on whether the project should be used to promote more widespread work of a similar nature - and, if so, how this should be done. The evaluation would thus be largely summative in character, but would also be formative in the sense that it was to be used to inform the planning of future developments in Grampian Region's primary schools.

Stage 2 : Deciding on the general approach to the evaluation. At the teachers' conference described above, it was decided that the evaluation would be carried out by Henry Ellington, who had been heavily involved in all stages of the project (and thus knew all the people involved) and was also an experienced educational evaluator. An advisory team comprising Grampian's Education/Industry Liaison Officer (the Project Coordinator) and three of the participating teachers was also established to help him plan and implement the evaluation. The evaluator met with his advisory team shortly after the conference, when it was decided that a mixed scientific/illuminative approach should be adopted.

Stage 3 : Deciding on the overall strategy for the evaluation. At the above meeting with the advisory team, the overall strategy for the evaluation was also formulated. This would involve eliciting feedback from all the teachers who had participated in the project and also from their respective head teachers. The information would be obtained through two questionnaires - a fairly detailed one to be completed by the participating teachers, and a shorter one to be completed by the head teachers - and via follow-up interviews with selected teachers. The evaluation would also be informed by the extensive classroom observation that the evaluator had carried out during the many visits that he had made to participating schools during the six months of the project. A schedule for the evaluation, which had to be completed by the end of June (when the schools broke up for their summer holidays) was also agreed, as was a provisional list of the schools to be visited during the second phase of the evaluation. It was, however, acknowledged that this list might have to be modified in the light of the responses received via the questionnaires. It did, as we will see later.

Stage 4 : Designing the evaluation instruments. This work was carried out by the evaluator immediately after his meeting with the advisory group. It involved designing two questionnaires (one for head teachers; one for participating class teachers), drafting a covering letter to the head teachers that would be sent out along with these, and designing an interview schedule/recording pro-forma for use in the follow-up interviews. The questionnaire for the head teachers was produced on two sides of a single A4 sheet, and asked questions about such things as the heads' personal involvement in the project, their assessment of the overall value of the project to their schools, and how they thought the project should be followed up. The questionnaire for class teachers was produced as a folded four-page A4 leaflet, and was divided into four sections, dealing respectively with the general nature of their project, their use of games and simulations,their use of external industrial/commercial links, and their overall evaluation of the educational effectiveness of the project. The schedule/recording pro-forma for the follow-up interviews concentrated on the educational effectiveness of the games and simulations used in the project, since this was an area in which the evaluator had a long-standing professional interest, and he wished to take the opportunity to gather data for research purposes. All instruments were worded so that responses could be clustered and analysed for statistical significance.

Stage 5 : Preliminary testing of the instruments. Because of the shortage of time, this was limited to showing the instruments to one of the members of the advisory team (a highly-experienced class teacher). She made a number of extremely helpful suggestions, all of which were taken on board by the evaluator.

Stage 6 : Implementation of the evaluation. Once the evaluation instruments had been finalised, the two questionnaires were sent to the head teachers of all the schools that had been involved in the project along with the covering letter. Heads were asked to complete the first questionnaire themselves and to ensure that the second was completed by all class teachers who had been involved in the project. It was requested that the completed questionnaires be returned within 10 days, to allow time for the follow-up interview phase of the evaluation to take place before the end of term. By dint of a little chasing up on the part of the evaluator's secretary, a 100% return was achieved by the date requested.

The evaluator then carried out a preliminary analysis of the returns, and, on the basis of this, made a final decision as to which schools to visit to carry out follow-up interviews. It had previously been agreed with the advisory team that the evaluator should visit at least one school in each of Grampian's six Educational Areas, and that the interviews should cover the full range of projects carried out, both in terms of the age of the pupils involved and in terms of their content and methodology. The final decision was also based on the nature of the games and simulations that had been used in the projects, since the evaluator was anxious to cover all the main types. A total of nine schools were eventually visited, and a total of 11 class teachers interviewed. As a result, detailed information was gathered on 21 separate games and simulations (some teachers had used more than one).

Stage 7 : Analysis of responses and reporting on findings. Detailed analysis of the completed questionnaires and follow-up interview records and production of the required reports on the evaluation kept the evaluator more or less fully occupied for most of July and August. Two reports were in fact produced - a 68 page Full Report and an eight page Executive Summary. Drafts of these were circulated to the members of the advisory team by the end of July, and the final versions were submitted to Grampian Education Authority and other interested parties at the beginning of September.

Follow-up debriefing meetings for the participating teachers were subsequently held in all six of Grampian's Educational Areas. At these meetings, the teachers were shown a video that had been made of the project, and also explored ways in which the work of the project could be built upon. They agreed with the evaluator's conclusion that the project had been an outstanding success, and also supported the main recommendations made in the report. These were that steps should be taken to promote the wider use of games and simulations in Grampian's primary schools (eg through the development and general distribution of a 'do-it-yourself' kit for teachers), and that mechanisms should be set up to foster the development of links with industry and commerce (eg through the establishment of a network of link teachers together with a supportive infrastructure).

Stage 8 : Follow-up work. As a direct result of the recommendations made in the evaluation report and the feedback from the follow-up debriefing meetings, both of the above measures were subsequently implemented by Grampian Education Authority. One of the teachers who had participated in the project was seconded to work on the two areas, and, in collaboration with the evaluator, she developed a 'do-it-yourself' pack on the design and use of games and simulations in primary schools. This was distributed to all primary schools in Grampian Region, and has since proved extremely successful in promoting the wider use of such techniques. The seconded teacher also played a leading role in setting up a network of school/industry links throughout Grampian's primary sector, and in establishing a central infrastructure to provide support for these. The data from the evaluation also provided the evaluator with the material for several research papers, as had been hoped.

CLICK HERE FOR EXERCISE 1

Authorised and maintained by the Educational & Staff Development Section,

Centre for the Enhancement of Learning & Teaching

Main Index | EDSU | RGU Home