Parliament Workbench
Data: Official Report to 19 Nov 2026

Official Report · Committee

Exam Results

Wednesday 27 September 2000 Education, Culture and Sport Committee Committee business 351 contributions · 17 speakers

Speakers

Choose an MSP to read only their contributions

The record

351 contributions · showing 1–200
Mary Mulligan Lab The Convener · Linlithgow
link

Good morning and welcome to the Education, Culture and Sport Committee. I particularly welcome members of the Scottish Executive education department. Mr Elvidge will introduce his team.

John Elvidge Scottish Executive Education Department
link

On my left is Douglas Osler, the senior chief inspector of schools, whom the committee knows well. On my right is Eleanor Emberson, the head of the division that deals with policy on higher still and other matters.I will repeat what I said in my letter to the clerk about why I thought it would be helpful for Douglas Osler to be here today as well as in a couple of weeks' time, when you will see him separately. It is important to recognise that higher still is a corporate responsibility of the department and that Douglas and some of his staff, as well as members of Eleanor Emberson's division, are involved in the issues. Although I am not sure of the extent to which the committee will want to get into issues relating to higher still, I thought that it would be more convenient for you to have everyone who is responsible for the issue present in one place to facilitate the questioning.

Mary Mulligan Lab The Convener · Linlithgow
link

Thank you. I am sure that that will be helpful. We have your written submission in front of us. People have had an opportunity to read through it and we have a number of questions to ask. Specific questions will be put and I will try to bring in any members who have supplementary questions. We have approximately an hour for this section.

Ian Jenkins LD Tweeddale, Ettrick and Lauderdale
link

Before we get into the nitty-gritty of what went wrong in the past few months, I want to ask three questions. First, to what extent does the department now consider that the decision to merge the Scottish Examination Board and the Scottish Vocational Education Council was responsible for difficulties relating to corporate philosophy and structural amendments to the testing regime? Secondly, did the decision to go ahead with the implementation of higher still come a little early for everyone concerned? Thirdly, taking those two points together, does the department accept that the volume and complexity of the data that were subsequently mishandled were at the heart of the problem? The decision to merge the SEB and SCOTVEC was contentious in certain quarters. Some people thought that the organisations had different philosophies and that, although it was logical to create an umbrella organisation, the two did not fit together well.

John Elvidge Scottish Executive Education Department
link

Those questions go to the heart of the matter. As members will know, when the previous Government consulted on four options, the view was widely held in Scotland that the merger of the two organisations was the best solution. Given what we know now, was that judgment right?Inevitably, one is drawn into having an opinion about what precisely went wrong. The Enterprise and Lifelong Learning Committee heard some very useful evidence on that—I am sure that it is ground that this committee will explore in much more detail. Without jumping to the conclusion that the answers that were given yesterday will prove to be definitive, I will assume, for the purposes of answering this question, that they are substantially correct. If those answers are correct, the root of the problems was in a part of the Scottish Qualifications Authority that was not substantially affected by the cultural mix. It was a part of the SQA's operations that was run almost exclusively by people who brought with them the expertise of the former Scottish Examination Board. Although one can speculate that some of the cultural issues may have impacted on the quality of internal communications, the mystery of how the problems could exist without being widely known in the organisation—I think that that will come to preoccupy us all—did not have its root in cultural issues relating to the merger. As far as one can judge from what is known now, one would conclude that, whatever problems of merging cultures existed in the organisations, those problems were not at the root of what went wrong.

Ian Jenkins LD Tweeddale, Ettrick and Lauderdale
link

I will jump to my third question. A consequence of the SCOTVEC philosophy entering the exam system was modularisation and the use of performance indicators. That led to a volume and complexity of data of a different order to what had existed before. Of course, I know that there had already been a move to internal assessment, but I am thinking of the number and type of data and the way in which internal assessment was to be merged with exam performance. Even given what you say about the culture, do you think that the considerable change in the way in which things worked—in addition to the volume and complexity of data—was a problem?

John Elvidge Scottish Executive Education Department
link

I have spent some time thinking about that, as one of the natural questions to ask was whether there was a step change in the volume of data that clearly meant that the old ways of doing things were no longer adequate. It was helpful to get a feel for the total number of pieces of information that were being handled, which I understand to be in the region of 4 million, and what proportion of those were exam scripts—the pieces of information that would have been around under a system that had not added on the elements of internal assessment. I understand from the SQA that, in round terms, that second number would have been around 3 million pieces of information. In volume terms, the new system added something less than a third to the total volume of data. I say something less than a third because the old system was not devoid of pieces of internal information that had to be handled by the SQA. It does not seem to be an obvious conclusion from that that a step change in volume of data is the explanation for why the SQA failed to live up to its former high standards in handling those volumes of data.

Ian Jenkins LD Tweeddale, Ettrick and Lauderdale
link

I am rather surprised that only a third of the volume was added, but I am not in a position to argue with your statistics. Do you accept that some of the material that ended up on certificates was of no particular use and that a great deal of work had gone into producing something that was of questionable value but that must have added to the complexity of things? I am talking about such things as core skills and the long-term move to group awards. That sort of extra has made the whole business more complicated than it needed to be.

John Elvidge Scottish Executive Education Department
link

We should bear in mind the fact that, in relation to this summer's exams, we are dealing with a subset of the SQA's customers. I am not sure that anyone ever thought that for candidates coming from schools, particularly those candidates for whom the primary purpose is to obtain a passport to higher education, some of the data about core skills would be the most important part of the certificate. For the other client groups that the SQA serves, that information was considered extremely useful. I am thinking now about candidates whose primary concern is entering the world of work. We must remember that the whole foundation is built on the strong belief that existed in Scotland in the early 1990s, when the system was being designed, that integration of academic and vocational education was the one guiding principle that should shape our work. The inclusion on certificates of things that candidates in schools may not find particularly relevant is a natural consequence of that.

Ian Jenkins LD Tweeddale, Ettrick and Lauderdale
link

Everyone accepts that there was a gap in provision for the population that was coming into fifth and sixth year in schools and a need to change the system accordingly. Did the decision to go ahead with implementation come just a little bit too early for people?

John Elvidge Scottish Executive Education Department
link

That was a difficult decision. The fact that implementation had already been postponed twice obviously suggests that the question of when it was right to make the move was at the forefront of people's minds. We certainly believed that we had reached the stage at which implementing the change would be challenging but manageable. That is different from saying that anyone thought that implementing it in the year just passed would result in everything going smoothly. Implementing a change of that magnitude is never accompanied by the expectation that things will go absolutely smoothly in the first year. However, there was a belief that we, the SQA and schools were essentially ready to cope with the change. In the light of what we think we know about what went wrong, I would not be inclined to revise that opinion.

Ian Jenkins LD Tweeddale, Ettrick and Lauderdale
link

Why was implementation postponed twice? Why were things different the next time?

John Elvidge Scottish Executive Education Department
link

I think that there was a combination of two factors, although I may ask Douglas Osler to comment further in a moment, as he is better acquainted with the history. There were two essential preconditions: whether we were prepared for teaching higher still in schools and whether we had the necessary materials; and whether the SQA was ready to undertake its part in the process. In previous years we concluded that, on both fronts, a bit more time would be helpful in getting to the starting line in good shape, as both preconditions involved substantial undertakings and a lot of work on a broad front.

Douglas Osler HM Chief Inspector of Schools
link

There was a clear view that the previous system was not meeting the needs of all young people. In education, if there is a highly desirable change, one always wants to bring the benefits of that change to young people as quickly as possible. The first year proposed for the introduction of higher still was quite obviously unrealistic, so ministers took the view that more time should be given. That was the story of the subsequent postponement and the rephasing decisions that were taken more recently.I should add that our inspection evidence shows that the levels of achievement of young people in S5 and S6 are significantly higher in higher still courses than in courses that are not higher still. It is quite clear that the quality of learning, teaching and attainment has risen. Our evidence shows that schools had coped well up to the point at which examination scripts left the schools to go to the SQA. The 25,000 or so young people who will have intermediate certificates will have something that no previous generation had. There will always be questions about how quickly we wanted to bring that benefit, but our inspection evidence suggests that schools that did higher still had coped very well. S5 and S6 have always been of high quality in Scotland. The signs so far are that the higher still courses have improved on that.

Nicola Sturgeon SNP Glasgow
link

You said that the SQA provided information saying that there were 4 million pieces of data this year, compared with 3 million in previous years. Is that correct?

John Elvidge Scottish Executive Education Department
link

That is not precisely what I said; what I said was intended as an approximation of the volume of data.

Nicola Sturgeon SNP Glasgow
link

Even if it is an approximation, I want to challenge you on that figure, because there are three unit assessments in every subject for higher still. It strikes me that, where there had been one piece of data—the exam script—in previous years, there would have been four pieces of data under higher still. Do you believe the figures given by the SQA that you have quoted this morning?

John Elvidge Scottish Executive Education Department
link

What you have said would be true if highers were the only exams that were being handled in the system. However, highers form a relatively small proportion of the total number of exams being handled. The increase in data attached to the new higher has only a limited effect on the total volume of data handled by the SQA in the summer diet. I have been through the same process of questioning that you have just gone through.

Nicola Sturgeon SNP Glasgow
link

It would be useful to have a breakdown of the figures that you have provided. Would that be possible?

John Elvidge Scottish Executive Education Department
link

Certainly.

Nicola Sturgeon SNP Glasgow
link

My second question is about the speed of implementation of higher still. When standard grade was introduced, it was piloted in the first year in a limited range of subjects. Was piloting higher still ever discussed in your department? Was it something that the SQA ever suggested? If piloting was not discussed, why was it not discussed? It would seem to be a reasonable way to introduce change of that magnitude so that problems could be ironed out.

John Elvidge Scottish Executive Education Department
link

I will need Douglas Osler's help as we go further back. However, I understand that it was not felt necessary to pilot because we had had two successive delays and had been able to prepare adequately across a wider range of subjects. As Mr Osler said, the demand from the education community for the introduction of higher still, which was believed to be a better system, was a considerable factor in our way of handling the situation. Every year that we did not introduce higher still was a year when the candidates whom the new features were intended to benefit would lose something. As a result, there was a desire not to leave any young person in that position any longer than was necessary.

Nicola Sturgeon SNP Glasgow
link

The demand for speedy introduction of higher still from the education community is something that has passed me by.In the Scottish Executive's written evidence, there is a reference to the expression of concerns about higher still from members of the education community. Mr Osler, will you tell the committee how far back those concerns go, to whom they were communicated, what exactly the concerns were and how those concerns might have impacted on this year's problems? Furthermore, the submission says that the Executive considered the concerns expressed by stakeholders very seriously. What was done to respond to those concerns?

John Elvidge Scottish Executive Education Department
link

With respect, that is not a question purely for Mr Osler. The education community's opinions do not come in through any one channel. The department has a variety of ways of contacting the stakeholders. I will do my best to answer that question.

Nicola Sturgeon SNP Glasgow
link

I am not really bothered about who answers the question—I just want some answers.

John Elvidge Scottish Executive Education Department
link

The dialogue with the stakeholder community goes back a long way; a series of groupings in which stakeholders could be consulted have been carefully maintained in the process of thinking about what needed to happen and when it should happen. Clearly, the dialogue about when and how to introduce higher still stretches back a long way. From the evidence of the previous year in particular—we have supplied the committee with a lot of evidence of discussions in groups where stakeholders were present—there is no thread of feeling that the introduction of higher still this year was a mistake. Instead, there is a series of discussions about how to manage the process. Concerns that were expressed further back in time about whether it would be premature to introduce higher still—the concerns that led to the various postponements—had moved down a hierarchy of concern to narrower issues about how the implementation was made to work. This year, the department received remarkably few direct representations from any source about the process of implementation. Between October 1999 and June 2000, we received precisely six letters to ministers from any source about that set of issues.

Nicola Sturgeon SNP Glasgow
link

Are those letters from individuals or representative organisations? I am sure that you appreciate the difference.

John Elvidge Scottish Executive Education Department
link

There is a difference. The letters are all from individuals. There is a natural explanation for that, because the representative organisations were all in a structured dialogue with the SQA and us and, by and large, did not need to write letters in order to communicate their views. We frequently sat around the table together and they expressed their views. In the context of those discussions, it is interesting to note that those education authorities that decided to submit detailed views on their experience of implementation were a small minority. The impression that I get on reviewing the record of the dialogue with stakeholders is that they had moved into a constructive partnership with us and the SQA about how to manage the detail of the implementation process.

Brian Monteith Con Mid Scotland and Fife
link

I would like to pick up a similar point to the one that Nicola Sturgeon explored on the piloting of standard grade and the introduction of higher still. It would appear that teachers managed to cope with the introduction and that the two delays stemmed from concerns about the preparedness of teachers in schools and the SQA. However, it was then agreed that progress should be made and that higher still should be implemented. It appears, from media coverage and from communication that MSPs have received, that most of the concern that was expressed about higher still came from teaching staff about their preparedness.Nevertheless, the political decision to go ahead with implementation was taken. Only after that decision was taken did reports of concerns relating to information technology, exam markers and so on begin to emerge. However, by that time people were sitting exams and others were beginning to mark the papers. Some reports go back as far as October 1999, but the main reports came through in March and April 2000, by which time it was too late.There was great focus on whether schools and teachers were prepared. With hindsight, do you think that the department paid enough attention to the preparedness of the SQA, rather than that of teachers?

John Elvidge Scottish Executive Education Department
link

That is an extremely good question. Higher still appears to have worked in schools. It has not been perfect or absolutely smooth, but by and large, the schools have delivered. It seems fairly clear that the point when things went wrong was when the SQA received the outputs from the schools.I find it helpful to work through the succession of issues that arose during the year—which one can trace from the papers—and to ask whether they suggest that lack of preparation on the part of the SQA was a problem.There are four sets of issues that can be traced through the documents. The first set of issues surrounded the registration of candidates and entry for examinations. That was the dominant consideration in people's minds from around October last year to about March this year. It is clear that the process did not go as smoothly as it should have. It is equally clear that, by working together, the SQA and schools managed to resolve the practical implications of that difficulty.The next issue that comes to the forefront of everybody's minds is the exchange of assessment data between the centres and the SQA. The system appeared to work perfectly well for some centres, but not for others. However, as far as one could tell, the difficulties were not the centres' responsibility. By that I mean that it was not, by and large, something that the centres did wrong that accounted for the difficulties. Quite a lot of energy was expended on that issue, as it was thought that it might be indicative of a general problem with the IT systems, which might impact on the issue of results. The SQA spent a lot of time investigating case by case what was going wrong but found no consistent pattern, which is always frustrating. Life is a good deal easier if a problem has one head that can be cut off, rather than the problem appearing to have multiple causes.Broadly speaking, both we and the SQA satisfied ourselves that the IT systems were, essentially, working as intended. I would not argue that the IT systems and the relationships between centres and the SQA work as perfectly as one would wish. However, it was established that nothing had gone so wrong with the planning of the SQA's IT systems that the exam results could not be expected to run properly. From the evidence that the Enterprise and Lifelong Learning Committee received yesterday, that conclusion appears to be correct, bearing in mind my caveats about whether we are sure we know precisely what went wrong.There was a period when people were concerned about markers and it is clear that there was a difficulty in recruiting the last small cohort of markers. That problem was also resolved adequately, but later than one wished. I understand that the root of that difficulty was that the SQA was not as well placed to predict the number of markers that were required for each subject as it would have been in a more conventional year. Therefore, the estimate of what the SQA needed came later than was desirable. However, I am not sure that that is a planning fault—it might be one of the inevitable consequences of having one's first experience of a new practice.During the last phase of the problems, which began at the end of June in our view, attention moved to what proved to be the crucial issue at the end of the day—the SQA lost track of some information. For a long time during that period, we thought that that was somehow a manifestation of IT problems. People talked about the problem as if it were a data transmission or IT problem. If the evidence that was given to the Enterprise and Lifelong Learning Committee yesterday is correct—I have no reason to suppose that it is not—that problem was nothing to do with IT systems. It appears to have been because of the SQA's procedures for handling physical pieces of information—pieces of paper. During yesterday's meeting, people said that that was incredible. I am not sure that anyone who visited the big shed in Dalkeith, which is full of pieces of paper on racks, would find it quite so surprising that pieces of paper could be mislaid. However, it is surprising because handling pieces of paper is precisely what the Scottish Examination Board has always known how to do well. That does not lead one to the conclusion that a lack of pre-planning was eventually the crucial factor.

Mary Mulligan Lab The Convener · Linlithgow
link

We will return to questions on IT data management and marking arrangements. I am sorry to have interrupted you.

John Elvidge Scottish Executive Education Department
link

I am not saying that planning could not have been better. Planning is one of those things that could always be better. I am not saying that clearer project planning by the SQA would not have helped to deal with some of the difficulties that arose along the way; I am questioning whether better advance planning would have helped prevent what went wrong.

Brian Monteith Con Mid Scotland and Fife
link

Thank you for that clear and full answer to my question. Your answer suggests what I have suspected for some time—there was not one problem, but several, which added strains to existing strains, and that compounded the difficulty.Given that one can predict that there will be an effect further down the line if something goes wrong, would not it have been better to run several pilots? Almost every other educational policy that has been introduced by the Scottish Executive or the Scottish Office in recent times has been piloted.

John Elvidge Scottish Executive Education Department
link

That is an imponderable question. Who knows what would have happened if we had piloted higher still? The strains of a situation such as we have experienced arise only when one runs the system at full capacity. One can run the system at a fraction of its capacity and everything can seem fine. It is only when one tests the system in real life and at full capacity that one discovers whether it will work.We offered the SQA advice on IT issues in April and told it that we knew that it had carried out certain kinds of testing on its IT system. We also told the SQA that it had to test the system at volumes at which it would have to operate and that until it had done that, it would not know whether the system would stand up to the strain that it would face. The argument against piloting is similar.

Jamie Stone LD Caithness, Sutherland and Easter Ross
link

Piloting worked for standard grade. Before one tests a jet engine at full throttle, one tests it at half throttle. Do not you agree that, in retrospect and with all the facts that we now have, it was a mistake not to pilot the scheme?

John Elvidge Scottish Executive Education Department
link

I could not conclude that from the information that we received. The fact that we use piloting in many situations demonstrates that we believe that it can have advantages. However, from the information that is available to us, I do not conclude that the decision not to pilot made any difference in this case. I sense that Douglas Osler is itching to make a contribution.

Douglas Osler HM Chief Inspector of Schools
link

We are in danger of holding standard grade up as a model for introduction of a new examination system, but history would not bear that out. Higher still has received far more support from all the main stakeholders than the introduction of standard grade received, and higher still has been consulted on far more extensively than any previous development programme that I know of.I do not want to quibble over the wording, but technically, standard grade was not piloted. Part of the reason why standard grade took so long to introduce was that it was held up by a period of industrial action, which delayed it for about two years. Standard grade was phased in: groups of subjects were introduced in four separate phases.I understand that the exam boards that were responsible for that found it difficult, because several subjects ran in parallel. Rather than one subject having one course, subjects such as English were being run in two different forms. That was demanding of markers, IT systems and so on. It was also confusing for young people, parents and employers. Over a long period, people left school with standard grades and O-grades—the relationship between the two caused much confusion.When higher still was being introduced, all those issues were discussed. It was decided that the standard grade phasing had not been a good way to introduce a new scheme and that a big bang approach would be better for the system. However, that did not happen with the introduction of higher still—some higher still subjects ended up being partly phased in. That meant that there was a similar format of introduction as that which was used for standard grade, although it was not exactly the same.

Jamie Stone LD Caithness, Sutherland and Easter Ross
link

Nevertheless, there was no degradation of O-grades or standard grades—at least, not that I am aware of. Were ministers asked to make a decision on whether to pilot?

John Elvidge Scottish Executive Education Department
link

I am not sure whether we are allowed to tell you that. We are not talking about our present group of ministers; we are talking about the business of previous Governments.

Jamie Stone LD Caithness, Sutherland and Easter Ross
link

You might want to reflect on that answer.There is some circumstantial evidence that suggests that although the teachers at the chalk face used every endeavour to implement higher still—such is their professional ethos—there was some consternation. It has been suggested to me that HMI was aware of the situation and that representations were made to HMI by high schools and secondary schools throughout Scotland. In some way, that information was not relayed back to you, the civil servants in the Scottish Executive education department.

John Elvidge Scottish Executive Education Department
link

Regarding Douglas Osler and the other members of the inspectorate as if they were a different species is a misconception of the way in which the Scottish Executive education department operates—they are, for most purposes, officials of the department. We should be regarded as knowing what they know. We are not free from imperfections in internal communication—no organisation is.Throughout that crucial period, we had many opportunities to sit down with representatives of the teaching unions and the education authorities, as well as having continuing contact with individual schools. From the papers that the committee has received, it will be clear that the process was not without bumps, but in the majority of cases, the message that we received was that implementation was going ahead as planned. The excellent job that schools have done bears that out—they taught the courses and prepared their candidates for the exams. The system was not breaking down. There were irritations over whether materials were being made available as quickly as classroom teachers would have liked, and there were sometimes misunderstandings over the freedom for manoeuvre that classroom teachers had, which might have removed some of their frustrations. However, the message from our sources of contact about what was happening in the schools is not that the process was proving unworkable—or anything that closely approximated to unworkable—but that it was generating the kind of teething troubles that one would expect.

Jamie Stone LD Caithness, Sutherland and Easter Ross
link

Given what you know now, do you accept that you were not aware of certain signals and storm warnings that were coming from the chalk face?

John Elvidge Scottish Executive Education Department
link

We were aware of the warnings—that is what I have been trying to say. The evidence that has come to light includes communications from the SQA to schools. That communication plainly acknowledges the existence of anxieties in the schools and seeks to work with the schools to deal with their anxieties.I am constantly trying to refrain from leaping to conclusions about what happened, because the committee and others are in the middle of a process of trying to pin that down. However, although it is possible to improve the processes that govern how higher still functions in schools, the schools did an excellent job. What happened in the schools is not a contributory factor to the problems with the summer diet of exams.

Mary Mulligan Lab The Convener · Linlithgow
link

I am anxious to move on, but a couple of members still have supplementary questions. I shall invite them to speak, and we will then move to the next section.

Fiona McLeod SNP West of Scotland
link

Let us return to the evidence that was presented to you and the way in which you used it in the implementation of higher still. You talked about six letters that were sent to the Scottish Executive. I would like you to think about the consultation and representation that was made to you in terms of its content and source, rather its quantity. It would be interesting to know exactly what was said to you in those six letters.There seems to be a contradiction in what you have said. When you answered Nicola Sturgeon's question, you said that most of the representation that you received concerned the impact on the production of the exam results. However, in paragraph 4.4, on page 23 of your submission, you say: "Schools and colleges did raise concerns . . . but far more feedback was received on implementation issues relating to learning and teaching".I return to a question—that has been asked more than once—about the phasing and piloting of higher still. If the representation that you received was about learning and teaching, why was the decision made to go ahead with higher still this year?

John Elvidge Scottish Executive Education Department
link

I do not recollect saying to Nicola Sturgeon that the representations that were made to us were primarily about handling of the exam results. If I gave that impression, it was a false one. The letters that we received were about a combination of what was happening in the schools and people's anxieties about bits of the process. Typically, the letters focused on individuals' experiences, which led to worry that such experiences were widespread. The majority of the contacts that we had with people in schools naturally concerned learning and teaching aspects. It is difficult for the average teacher to see into the exchanges between their school and the SQA, and it is certainly near impossible for them to see into what is happening inside the SQA. One would expect teachers to talk to us about their first-hand experience—the process of teaching the courses.As I tried to say, nothing in those representations suggested that the delivery of higher still in the schools was in any way near to failure. There were irritations for individual teachers and, as we know, individual teachers hold differing views on higher still. Some teachers believe strongly that the process of internal assessment is simply wrong in principle, but that is not a majority view. Some teachers felt that materials that were produced centrally for their subject could have been made available sooner—and they were right. The materials for most subjects were produced on time, but some lagged behind a little.What one heard from the classroom seemed perfectly natural and was entirely consistent with the facts as we knew them. It did not lead one to the conclusion that something was going fundamentally wrong with the delivery of higher still in schools. The evidence suggests that nothing went fundamentally wrong with the delivery of higher still in schools. The problem is of the SQA's making, not the schools'.

Mary Mulligan Lab The Convener · Linlithgow
link

Nicola, has that answered your question as well?

Nicola Sturgeon SNP Glasgow
link

I have two further questions—we still have not received answers to some points that have been raised.Are you confident that the concerns that may or may not have been raised by teachers—for example, through the higher still development unit—were getting back to you and, by extension, to ministers? The picture that you paint does not accord in all respects with the reality of the past few months, during which it seems that there were real concerns in schools that things were not as they should be. An example of that—this is a delivery problem—is the difficulty that schools had in communicating information to the SQA and that the SQA had in processing that information. It amazes me that you have received only six letters this year about the problems that were associated with higher still. Perhaps that is because those concerns were being directed through the higher still development unit. If that is the case, can you guarantee that you were being made aware of those concerns and that those concerns were being acted on?My second question is one to which we have still not received an answer. Can you tell the committee—to the best of your knowledge—whether piloting of higher still was ever discussed in your department?

Mary Mulligan Lab The Convener · Linlithgow
link

I am not sure that we have not received an answer to that question. Please make your answer concise, as I am aware of the time.

John Elvidge Scottish Executive Education Department
link

I shall be as concise as possible, although these are complicated matters.It would be foolish of me to give an absolute guarantee that every representation that anybody made found its way through the system. A significant volume of comment came through the groups of stakeholders who we were consulting on the problems of transmitting information. From the evidence that the committee has, it should be clear that no one who was dealing with the issues was unaware that there was a problem on a significant scale. Regardless of whether every warning got through, there was certainly a substantial body of warnings and, as I said to Mr Monteith, those were dealt with systematically.In answer to Nicola Stugeon's second question, I have said that I am debarred from saying what discussions took place with the ministers of a previous Government.

John Elvidge Scottish Executive Education Department
link

I am also debarred from saying what advice we gave to the current Government. The committee needs ministers' agreement to release information about our advice to them, not my agreement to tell the committee what we might have said to ministers.

Mary Mulligan Lab The Convener · Linlithgow
link

I am sure that we will return to that at some stage, but I am anxious to move on. We will address a specific issue that has been raised and will then progress to the chronological order of the issues that have been discussed. Let us begin with discussion of the introduction of the IT system, data management and whatever followed that.

Ken Macintosh Lab Eastwood
link

The new IT system was obviously a major project. Can you explain the difference between the data communication problem and the IT system problem? Those seem to be two distinct difficulties.

John Elvidge Scottish Executive Education Department
link

Indeed. When I talk about the IT system, I am talking about the system that the SQA operates internally, which it uses to process the information that is available to the SQA. When I talk about the communication of data, I mean the electronic passing of data from computer systems in the centres to the SQA's computer system. For some centres, that link did not work as well as it was designed to, which led in some cases to paper being substituted for the electronic transmission of data.

Ken Macintosh Lab Eastwood
link

I would like clarification of the situation. Was the SQA computer system—which was different from the computer systems that are used in different local authorities—unable to read the data from the local authorities' systems as intended?

John Elvidge Scottish Executive Education Department
link

It is not the case that the SQA computer was incompatible in principle with any of the systems that are used in schools, because the communication worked fine for many centres. In many ways, it would be a relief if one could say that we know what the problem was—that one of those systems that are used in schools was not working, for example—but it is not as simple as that. By and large, the systems in schools seem to have done their job and the SQA's system seems to have done its job. However, for reasons that are not simple, that linkage did not work in a proportion of cases.Ron Tuck talked about that to the Enterprise and Lifelong Learning Committee yesterday, but I do not profess to have his depth of knowledge about what the SQA found when it checked the individual instances of what had gone wrong.

Ken Macintosh Lab Eastwood
link

Earlier, you said that the problem is continuing. Is the system still not working properly?

John Elvidge Scottish Executive Education Department
link

We are still not in a position to say that electronic communications between every centre and the SQA are guaranteed to work smoothly. One of the pieces of work that the SQA is carrying out at the moment, as part of its exercise to ensure that there is no repetition of this year's problems next year, involves simulated testing in an artificial environment of all the systems that are used in centres as well as the system that is used in the SQA centre. That is being done to check whether any of those systems misbehaves in a way that—although apparently random—can be tracked down to a feature of the system.

Ken Macintosh Lab Eastwood
link

According to reports, although the computers were supposed to talk to each other, all the data were eventually input manually. You returned, in effect, to a paper system and input the information into a computer. Is that true?

John Elvidge Scottish Executive Education Department
link

I cannot claim to know the breakdown of data that were entered electronically and manually.

Mary Mulligan Lab The Convener · Linlithgow
link

I am sure that we can ask about that elsewhere.

Ken Macintosh Lab Eastwood
link

Were you satisfied that the SQA had the IT knowledge to be able to introduce a brand new software system, make it work properly, pilot it and produce the results?

John Elvidge Scottish Executive Education Department
link

We were satisfied that the SQA had used reputable and established firms of advisers to design its software and that it was building the system on a widely used database system, so that there was nothing novel or risky in the foundations of the system. Although I cannot pretend to be able to judge the quality of another organisation's IT staff, our contacts with the SQA have not suggested that there is any reason to doubt the professional knowledge of its internal staff.

Ken Macintosh Lab Eastwood
link

Mr Tuck's submission says that you sent in an IT specialist at some point in March. I could find no reference to that in your submission. Is it referred to there?

Ken Macintosh Lab Eastwood
link

It is there, but I missed it. That is my fault. Sorry.

Cathy Jamieson Lab Carrick, Cumnock and Doon Valley
link

I want to examine the relationship between the education department and the SQA, to find out how some of the problems were brought to light.Reading the minutes of the liaison meetings between the Scottish Executive and the SQA, I was struck by the fact that an issue was raised in November 1999 in relation to previous standard grade exam appeals. A computer problem had been flagged up and reassurances were given that that could not happen again because a new awards processing system was being introduced. The error with the standard grade exams appeared to have occurred because the wrong program had been used. Did not that give some warning that all had not been well in the past and that, even with a new computer system, all might not be well still?

John Elvidge Scottish Executive Education Department
link

The only inference that I would draw from that is that mistakes happen, even when organisations have run the same processes many times. I am not led to conclude that there was any inherent reason to doubt the SQA's ability to handle its IT systems, although that possibility is the reason why we asked the questions.

Cathy Jamieson Lab Carrick, Cumnock and Doon Valley
link

What prompted the Scottish Executive to send an IT expert in and when was that decision made?

John Elvidge Scottish Executive Education Department
link

I shall try to be brief and sketch what I regard as three phases of our relationship with the SQA. The relationship changes demonstrably at points in this story.Until the beginning of March, our relationship was entirely normal. A series of pre-planned, structured meetings took place and whatever questions we had to ask were asked in the context of those meetings. We were going through what we regard as the normal process of ensuring that we were informed and that we were raising matters that worried us.From the beginning of March, we began to worry—largely because of questions that were being raised about data exchange—that reliance on that normal relationship was perhaps insufficient. We started to deal with the SQA outside the usual pattern of meetings. We did things that could by no stretch of the imagination be regarded as a part of our normal relationship with a non-departmental public body, such as suggesting that it take advice from our deputy director of information technology. We took that decision between early March—when discussion of data communication problems began to be widespread—and April, when the deputy director of IT visited the SQA. After we had spent some time trying to understand what was going wrong, we concluded that it would be helpful for the most experienced member of our IT staff to visit the SQA and offer it a peer review of its processes.

Cathy Jamieson Lab Carrick, Cumnock and Doon Valley
link

Who did that person report to? Will a copy of that report or its key findings be made available to us?

John Elvidge Scottish Executive Education Department
link

The key findings are summarised in one of the documents that you have been given.

John Elvidge Scottish Executive Education Department
link

From memory, I think that the key findings of that visit are summarised in a letter from Alastair Wallace to David Elliot.The report by our deputy director of IT was widely shared among people dealing with the SQA.

Cathy Jamieson Lab Carrick, Cumnock and Doon Valley
link

I will press you on that, as I do not think that the letter gives much information. To whom was the report passed at that stage?

John Elvidge Scottish Executive Education Department
link

I cannot from memory give you a full list of the people who saw it. It was not a report as such—it did not have a front cover and it did not set out the deputy director's investigations and findings. It was a statement of a number of suggestions of actions that the SQA should ensure that it took, largely as a matter of prudent contingency planning. The report did not consist of much more than a list of what those actions were, and that is largely what you will find in the letter from Alastair Wallace.

Cathy Jamieson Lab Carrick, Cumnock and Doon Valley
link

Are you content that the SQA took on board the report's recommendations?

John Elvidge Scottish Executive Education Department
link

In discussions over subsequent weeks, we asked the SQA what it was doing. We clearly identified—or the SQA clearly told us, and I have no reason to doubt it—that it had followed up many of the recommendations. If you are asking me whether I can say, hand on heart, that the SQA carried out all the actions in precisely the way in which we suggested that it should, I cannot say that I know that.

Cathy Jamieson Lab Carrick, Cumnock and Doon Valley
link

There are some other points to which I will return, but I am aware that other members wish to ask questions.

Cathy Peattie Lab Falkirk East
link

I recall reading somewhere in this tome of evidence that the SQA said that the IT information and advice that it had been given was not relevant and that it did not use it.

John Elvidge Scottish Executive Education Department
link

I think that you are referring to Ron Tuck's evidence to the Enterprise and Lifelong Learning Committee—I have not seen his evidence to this committee—which I would paraphrase as saying that the advice did not tell the SQA anything that it did not already know that it should be doing.

Mary Mulligan Lab The Convener · Linlithgow
link

It is maybe unfair to ask you to paraphrase what someone else said. I am sure that we can raise that point elsewhere.

Cathy Peattie Lab Falkirk East
link

I would paraphrase Ron Tuck's statement as saying thanks, but no thanks. What was your response to that, given that you had sent someone in to respond to concerns coming from many sources about IT and data management?

John Elvidge Scottish Executive Education Department
link

It is not my impression that that is how we perceived what was happening. Our perception was that, whether as a result of our suggestions or of its own thinking, the SQA was doing most of the things that our deputy director of IT had suggested would be sensible for it to do.

Johann Lamont Lab Glasgow Pollok
link

The evidence that we have heard was that the IT expert had no particular advice to offer. If there is a report that identifies things that ought to be done, logic dictates that your department would have a series of tick boxes in which it would subsequently be confirmed that those things had been done. The SQA was saying that there was no particular advice, but you then pressed the SQA, so I presume that it then realised that there had been advice. Did you know whether it was doing something? Was someone specifically responsible for checking that? Is there a report that identifies what ought to have been done?

John Elvidge Scottish Executive Education Department
link

There is not a report as such. Our deputy director of IT had a long discussion with David Elliot at the SQA. He did not come back and write a report, because the purpose of his visit was to offer helpful advice, not to write a report on the state of the IT systems. However, there was an identifiable list of suggestions that he made, so I do not agree that we had no particular advice to offer.

Johann Lamont Lab Glasgow Pollok
link

Did your department have a means of checking whether that advice was taken?

John Elvidge Scottish Executive Education Department
link

Yes. Well—

Johann Lamont Lab Glasgow Pollok
link

If you were concerned about the way in which the system was working, you would give specific advice, and then either ensure that that advice was taken or find out why it was not taken. You would not just hope that it was taken.

John Elvidge Scottish Executive Education Department
link

We raised all those issues in the course of subsequent discussions.

Johann Lamont Lab Glasgow Pollok
link

What would the consequences have been if advice was not taken? What was the next step?

John Elvidge Scottish Executive Education Department
link

I am aware of no instance where we felt that the SQA had not taken our advice, in some way or other.

Johann Lamont Lab Glasgow Pollok
link

Yet the SQA has said that no particular advice was offered. You think that it was taking advice; it does not think that there was any advice.

John Elvidge Scottish Executive Education Department
link

Members have in front of them a letter from us to the SQA, which identifies a number of pieces of advice that we offered. Members are as well placed as I am to decide whether that constitutes no advice.

Johann Lamont Lab Glasgow Pollok
link

But the issue is whether that was pursued, to see whether the advice was taken.

John Elvidge Scottish Executive Education Department
link

We did pursue it.

Mary Mulligan Lab The Convener · Linlithgow
link

We have information in front of us, and there seems to be an inconsistency. That is why people are pressing you on this, to get your view.

John Elvidge Scottish Executive Education Department
link

Of course.

Brian Monteith Con Mid Scotland and Fife
link

We hear about IT and we hear about data processing. I would ask that questions and answers be specific as to what those terms mean. Yesterday, in the Enterprise and Lifelong Learning Committee, we had the same problem. IT can cover a wide variety of things. We need to talk about both hardware and software. Some advice was specifically about hardware, and some was specifically about software. It is important that we differentiate between them.

Mary Mulligan Lab The Convener · Linlithgow
link

Members and witnesses will take that on board. We should be clear about what we are referring to.

Michael Russell SNP South of Scotland
link

I would like to ask about the letter from Alastair Campbell that has been referred to us—sorry, from Alastair Wallace, although it is spun almost as well as Alastair Campbell would have spun it. Looking through the letter, I can see no specific advice whatsoever. Indeed, the letter says:"we had no reason to doubt the thoroughness of SQA's analysis or the robustness of its planning."It goes on to say:"we recognise that SQA's basic analysis and project planning is sound."That does not sound to me like specific advice on things to do; it sounds, frankly, like a commendation of what was taking place.

John Elvidge Scottish Executive Education Department
link

It may help me if I look at the letter. My recollection is that that letter also refers to a number of things that we suggested would be helpful for the SQA to do. It is true to say that the impression was formed that there was no fundamental problem with the IT systems. Our suggestions were about prudent checking of various things and about contingency planning.

Michael Russell SNP South of Scotland
link

The letter comments on various things that are happening, but that is pretty far from being an assessment of the SQA's systems and where the problems lay. I know that you do not want to mislead the committee, but many of us thought that you were talking about an assessment of the situation, of where the computer system was going and of any problems. We did not think that you were expressing some satisfaction with the way that things were going and were simply making some suggestions.

John Elvidge Scottish Executive Education Department
link

I apologise if I have given the impression that what we did was other than it was. There was an extended discussion, the broad conclusion of which was that there was no identifiable major flaw in the SQA's systems. However, we thought that there were some checks that it would be sensible for the SQA to run and listed those. In following up, we asked the SQA whether it had carried out the checks.

Michael Russell SNP South of Scotland
link

The broad conclusion of the letter is:"we recognise that SQA's basic analysis and project planning is sound".

John Elvidge Scottish Executive Education Department
link

Yes.

John Elvidge Scottish Executive Education Department
link

Yes, and it is my position now.

John Elvidge Scottish Executive Education Department
link

I would be interested to hear what evidence you think runs contrary to that.

Michael Russell SNP South of Scotland
link

The distress caused to pupils and others is the evidence, Mr Elvidge.

Ken Macintosh Lab Eastwood
link

As far as I can work out, the letter makes three specific suggestions. I am interested in finding out about your relationship with the SQA. You say:"we would be grateful for information on your plans for volume testing at, say, 25% above the highest expected volume of output."Did that testing happen? You go on:"It would be helpful to have a note of what priorities SQA have in mind if problems arise, (for example, I understand Music takes a particularly large amount of processing time)."You then offer the use of PCs and of your integration centre"to simulate data interchange between centres and SQA, if this would be helpful."The key thing that we need to know is whether those suggestions were followed up. What was your relationship with the SQA at this point? Did it take advantage of your offer? We will speak to Mr Tuck later, but his response appears slightly dismissive.

John Elvidge Scottish Executive Education Department
link

I will need to deal with the suggestions individually. The basic answer is that we followed up the letter. As I have already said, the SQA told us that it had undertaken volume testing of its system. As far as I know, that is an accurate statement. We had subsequent discussions with the SQA that demonstrated that it was thinking through the prioritisation issues and that it was going through the process of analysis that one would expect.On the offer of our integration centre, my recollection is that the SQA said that it would prefer to do the testing itself in another way. The SQA was worried about having first to resolve some issues of commercial confidentiality attaching to each of the proprietary systems that it would be testing. As I understand it, the SQA was anxious that a number of software providers had designed different systems that were intended to incorporate the best features that they could incorporate. The SQA did not want to leave itself open to accusations of having contributed to some form of industrial piracy by allowing the intellectual property of one company to be transferred to the system of another. I understand that it was working on finding a way round that anxiety, so that it could undertake the kind of testing that we suggested. The SQA did not disagree that that kind of testing has value.

Ken Macintosh Lab Eastwood
link

This may be internal documentation, but I could not find the report or memo of the IT expert who visited the SQA. It would be helpful to have that. It would also be helpful to have some reference to how these issues were followed up, not necessarily in written form.

John Elvidge Scottish Executive Education Department
link

You should be able to find the evidence of the follow-up in the documents. By and large, it took place in meetings that we had with the SQA, especially in the meeting on 28 April. We will try to make that more accessible.

Johann Lamont Lab Glasgow Pollok
link

You said that you asked the SQA to say what its priorities were when there were difficulties, and that you were confident that there was evidence of the SQA thinking about those. In hindsight, was it sufficient that the education department had evidence that the SQA was thinking about priorities, or would it have been reasonable to expect it to say what its priorities were, given that that was identified as a problem?

John Elvidge Scottish Executive Education Department
link

The evidence to show that the SQA was thinking the matter through consisted of it telling us its priorities.

Johann Lamont Lab Glasgow Pollok
link

So the SQA was saying to you, "We have thought this through, and these are our conclusions"?

John Elvidge Scottish Executive Education Department
link

Yes.

Johann Lamont Lab Glasgow Pollok
link

The education department's role was then to monitor the situation, by asking for specific responses and checking that those responses were received.

John Elvidge Scottish Executive Education Department
link

Yes. It is a constant of our relationship with the SQA that we cannot pretend that we are the people who know how to run the processing system of examinations. The SQA has the people with the professional expertise. Our role is to say, "Show us that you have thought about this and explain to us the conclusions that you have reached." Substituting our judgment for theirs would be extremely difficult.

Johann Lamont Lab Glasgow Pollok
link

What would your role have been if you had been unhappy with the SQA's response? Given that you cannot substitute your expertise for the SQA's, at what stage would you say, "Wait a minute. We do not think that your thinking through is rigorous enough", or "This is creating anxieties for us"? How would you intervene, given the distinction that you have drawn?

John Elvidge Scottish Executive Education Department
link

In the framework of regular meetings, we are not shy creatures. We have no difficulty in saying when we are not given an adequate explanation and asking for further reassurance. If we had thought that the SQA's conclusions were manifestly illogical or wrong, we would have continued to press it towards a conclusion that seemed more satisfactory to us.

Johann Lamont Lab Glasgow Pollok
link

So throughout the whole process, you were reassured by what the SQA was doing?

John Elvidge Scottish Executive Education Department
link

Yes, that is true. I spent quite a lot of time thinking about the issue of reassurance, as I am conscious that part of the currency of this debate is the question of how we could have accepted reassurances when things went wrong. I have also been thinking about the general question of how one knows when someone else is not giving one a satisfactory explanation of something on which they are an expert.There are two basic things that one can do: when that person purports to provide factual information, one can try to verify that against some other source, and one can look for the internal consistency and logic of what they are saying. If the reassurances pass those two tests, it is difficult to get beyond them, and the reassurances that we were given passed those tests. Despite the fact that Michael Russell thinks that that is inconsistent with what happened, it is consistent with the view that it was not the functioning of the IT system that led to what happened. We still have no reason to believe that the kinds of issues that we were raising with the SQA were the root of the problem.

Johann Lamont Lab Glasgow Pollok
link

It may have been the general process of monitoring that created the difficulty. You received reassurances but, because of the barriers that you have identified, you did not have the capacity to establish that that was really what was happening.

John Elvidge Scottish Executive Education Department
link

Short of having a member of staff standing behind every member of staff of the SQA, it is not possible to obtain absolute certainty that everything that one is told about what that organisation is doing is 100 per cent true. We subjected the SQA's reassurances to the tests to which it was possible to subject them.

Marilyn Livingstone Lab Kirkcaldy
link

We discussed this issue yesterday in the Enterprise and Lifelong Learning Committee. The evidence that we received from Ron Tuck about the APS development is crucial. He said:"At all our regular liaison meetings with Scottish Executive officials, progress with Higher Still implementation was discussed in great detail."The APS development was discussed and"was running somewhat behind schedule, but not critically".Ron Tuck assured us that the issue was not critical. I asked him about the implementation of the software. He said that, at that stage, the problem was data management and that the "crucial flaw which led to the problems with August certification lay in the management of data. Other problems such as software"and other issues"did not. . . . impact directly on the accuracy of certification, although these factors did exacerbate the data management problems."It became clear through all our discussions that the problem was data management. That was my impression. Cathy Peattie might want to comment on that too.

Mary Mulligan Lab The Convener · Linlithgow
link

I am anxious not to repeat what happened yesterday. I am sure that that is useful information that we can pick up on.

Marilyn Livingstone Lab Kirkcaldy
link

I am just saying that we were told clearly yesterday that the APS was not the issue, so maybe we should not go back to exploring that.

Mary Mulligan Lab The Convener · Linlithgow
link

Fiona, do you still have a question?

Fiona McLeod SNP West of Scotland
link

It is really a statement. I wonder what it takes to set alarm bells ringing in the Scottish Executive. Throughout the questioning, we have heard that you never thoroughly checked out the reassurances that you were given. What does it take to make the Scottish Executive say, "Let's investigate a bit further and ensure that we are getting the right answers"?

John Elvidge Scottish Executive Education Department
link

That is a bit of a cheap shot. I am not sure what your definition of thoroughly checking something out is.

John Elvidge Scottish Executive Education Department
link

I have given you at some length my definition of the processes that it is possible to follow to try to check that something that one has been told is true. A judgment must be made about whether individual questions are central to the issues. Throughout the process, we discussed with the SQA a wide range of emerging issues that might have been a source of anxiety. We pursued each issue in a way that was proportionate to the risk that it appeared to pose to the outcome. We followed a proportionate process of checking with the SQA.

Mary Mulligan Lab The Convener · Linlithgow
link

Can we move on, if there are no specific questions?

Nicola Sturgeon SNP Glasgow
link

On the same general theme, I am interested to know what further information you can furnish us with to help us resolve some of the questions and form a judgment about the rigour of your relationship with the SQA. I read the memorandum that you sent, which notes concerns about data management and data processing and about delivery of the examination results. Sometimes, vague assurances from the SQA that things were all right were accepted at face value. The SQA once reportedly told you that "we remain optimistic that the diet will be delivered successfully."That was hardly a hard and fast assurance that things were okay. What information can you give us to allow us to form a picture of the questions that you asked, the steps that you took to investigate the vague assurances and the efforts that you made to find out what was really going on in the SQA?

John Elvidge Scottish Executive Education Department
link

We are happy to give you a fuller account. Notes of meetings necessarily constitute a summary of what may have been long discussions. We are happy to try to give the committee a better flavour of the issues than it can get from the documents available.

Nicola Sturgeon SNP Glasgow
link

That would be useful, because those issues go to the heart of our inquiry.The memorandum refers to other forms of assistance that you offered the SQA. Can you go into some detail on what those forms of assistance were? How did you respond when, as this memorandum suggests, the assistance that you offered was rejected out of hand?

John Elvidge Scottish Executive Education Department
link

We were not providing itemised lists of things that we thought might help the SQA. Help comes in two forms: people and money. We were saying that if the SQA could find any way of using either of those things, we would try to help it. The SQA's response, generally, was that it thought it had the resources that it needed. It had employed significant numbers of extra staff to help deal with the problems and it did not need our money to do that.Another constant theme of the SQA's response to us was that, when one is in the critical phase of a major processing operation of this kind, introducing untrained staff who have to be trained by the staff who are getting on with the job would create more disruption than it could add value. That is a managerial judgment. It is not an implausible argument, so we listened to it and accepted the SQA's judgment. In my experience, when people say that they do not want help when one offers it, their decision is usually founded on a sound judgment. We normally face the opposite situation, with people asking for more resources.

Nicola Sturgeon SNP Glasgow
link

That in itself might have set alarm bells ringing.This is something of a theoretical question, but I want to ask you about the legal relationship between the Scottish Executive and the SQA. To what extent do you think it would have been open to you, through ministers or HMI, to intervene if you thought that help that you were offering and that was being turned down was required and that advice that you were giving to the SQA was not being followed? Could you have intervened to ensure that certain things happened?

John Elvidge Scottish Executive Education Department
link

There are two parts to your question. I would describe what we did, particularly between the end of June and the critical date, as intervention of a sort, in that it bore no relation to the normal relationship between a non-departmental public body and the department. We were offering advice and putting our minds to the problems faced by the SQA on a completely abnormal scale. To go beyond that, we would have had to find a set of circumstances in which one could identify a particular thing that we knew would solve the problems and that for some reason the SQA was refusing to do. In this set of circumstances, I cannot identify such a thing.

Nicola Sturgeon SNP Glasgow
link

Are you saying that you could have intervened? If that is what you are saying, was Sam Galbraith wrong to say in Parliament that he had no power over the SQA?

John Elvidge Scottish Executive Education Department
link

I do not think that he said that in Parliament. He gave a rather lengthy explanation of the nature of his powers and of the way in which they are hedged around. I am saying that I cannot see a set of circumstances in which those powers, whatever they may be, could have been brought to bear in this case.

John Elvidge Scottish Executive Education Department
link

It is a matter of record in the legislation that the powers exist. Matching them up to this set of circumstances is an entirely different thing.

Mary Mulligan Lab The Convener · Linlithgow
link

Unless anybody is desperate to ask a question, I would like to move on.

Johann Lamont Lab Glasgow Pollok
link

The fact that you cannot think of circumstances in which you would intervene may explain why there was not any intervention, because at each stage the reassurances given were sufficient. I would like you to reflect on how you manage the rest of your department and the way in which tasks are allocated to staff. What are the obvious things that you as a manager were unable to do that you would have done if it had been in-house?

John Elvidge Scottish Executive Education Department
link

That is a big question. If this had been an entirely in-house operation, all the relationships would have been different. I am hesitant about saying that if it had been in-house, the problems could have been solved. I go back to Ron Tuck's evidence to the Enterprise and Lifelong Learning Committee yesterday. He was unaware, as chief executive, of the things that eventually went wrong. I cannot put my hand on my heart and say that if the operation had been in-house we would necessarily—simply by virtue of that—have been able to spot something going wrong that the management of the SQA was not able to spot. I am not saying that nothing would have gone wrong. I cannot say with certainty that because we were running an operation of this kind, we would have been infallible and would have spotted a problem that seems to have occurred a long way down the organisation.

Johann Lamont Lab Glasgow Pollok
link

But did you have a different process for ensuring that your advice was pursued?

John Elvidge Scottish Executive Education Department
link

Indeed. We do not have to give anybody any advice if we are running something—we just do what we think is right.

Mary Mulligan Lab The Convener · Linlithgow
link

That is clear. I am conscious that others want to ask questions. I will bring them in at the end if we have time. We move to Mike Russell's questions.

Michael Russell SNP South of Scotland
link

You talk about your reluctance to intervene. Eleanor Emberson might want to pass you a letter from her dated 17 July 2000. Reluctance to intervene does not appear to be part of it. Committee members should comment on it, but the letter appears to be a series of instructions from you to the SQA. The letter says:"By 21 July, you will have to consider the plan for data processing". The following paragraph says:"When the main data file is being closed, you will identify".The paragraph after that says:"The possibility of moving the 10 August publication date will have to be considered".The next paragraph says:"It might . . . be sensible to go through some of the arrangements".I welcome your comments or Eleanor's on whether the letter was helpful or whether it was a set of instructions, which would have meant that you were by that stage running the SQA in all but name.

John Elvidge Scottish Executive Education Department
link

I do not dispute that we were intervening or that writing a letter like that is a long way outside the normal conduct of the sponsorship relationship, but that does not constitute running the SQA. We were leaning heavily and producing something that looks much like a checklist of actions. You are right to say that that letter is evidence of a relationship that has travelled a long way towards intervention.

Michael Russell SNP South of Scotland
link

We have already discussed the letter of 20 April from Alastair Wallace, which is not just about IT but about all the arrangements within the SQA. It says:"This . . . enabled us to give reassurance to ministers about the true position"——and you call it the true position—"including that we had no reason to doubt the thoroughness of SQA's analysis or the robustness of its planning."You also say:"we recognise that SQA's basic analysis and project planning is sound."You have said that you will provide us with notes of meetings that the minister has indicated took place on 28 April; 27 June; 7, 14, 21 and 28 July; and 2, 4 and 9 August, but in the papers that you gave us there is nothing from your side to indicate that there is any great problem until this extraordinary letter dated 17 July. What happened in those two and a half months to mean that you changed from saying that everything was fine to your giving instructions—as you admit—to the SQA about how to do its business?

John Elvidge Scottish Executive Education Department
link

That is a good question. A discernible change in the nature of the relationship can be located quite precisely at the end of June. In answer to an earlier question, I sketched out what I saw as the first change in our relationship with the SQA, which took place in early March. I would characterise the relationship between early March and the end of June as one of trying to ensure that the SQA had in its sights what seemed to us to be the appropriate issues. However, we were taking it as given that, once issues were identified, the SQA would be capable of dealing with them. As you will see from the evidence, written assurances were given internally to the SQA board and others about the position in the week ending 23 June. At the beginning of the following week, on 26 June, Eleanor Emberson made a telephone call to the SQA in which information emerged that seemed to us to contradict the impressions that had been given as recently as the previous week.

John Elvidge Scottish Executive Education Department
link

With Ron Tuck. Through that telephone call, we came into possession of information either about the number of scripts that were still unmarked or about the number of markers still being sought. For the first time, reference was made to the issue of the missing data. Knowledge of those things seemed to us to place us in a different position and made us feel less secure in the belief that we could operate in the same mode as we had been operating in. From that point on, we moved into a relationship of greater intensity of meetings and greater intrusion on our part into things that we would normally regard as the management business of an NDPB. The letter that you quoted is symptomatic of something that was happening through a period that stretches from that week to the end of the process.

Michael Russell SNP South of Scotland
link

Your submission to the committee says that you had "been aware for some time that recruitment of sufficient markers might be an issue . . . and there had been some discussion of possible problems at the meeting on 28 April on the Awards Processing System".Two months before that momentous phone call, you had an indication that things might be wrong. What did you do in those two months?

John Elvidge Scottish Executive Education Department
link

One can know that something might be a problem, but that is not the same as concluding that it has become a serious problem. We were in discussion with the SQA about how those things were going, and we eventually received entirely adequate reassurances that it had the number of markers necessary to do the job. On the APS, we come back to the key distinction that members of the committee have been anxious to make. The issue of missing data was not about the functioning of the APS. It was not a matter of what was happening inside a computer system, but something that was happening in the assembly of the information that needed to be available to allow that computer system to operate—a quite distinct issue.

Michael Russell SNP South of Scotland
link

Let me take you from the end of June through July to the beginning of August. The letter from Eleanor Emberson indicates that in all but name—we may disagree about this—you were issuing instructions to the SQA on what it should do. One of the key issues became whether it should put the exam results out. There is debate about that, and I think that we need a lot more correspondence. There are six letters and five e-mails here, but there must have been a great deal more correspondence and I ask you to consider supplying that.In the midst of all that discussion, there was an e-mail on 1 August—eight days before the results were due—from Alastair Wallace, which was copied to a number of people, including Eleanor Emberson. It said: "There is also the more general problem of how to manage public reactions . . . We would expect Ministers to issue a supportive statement making these points".We would expect ministers to issue a supportive statement on those points, but there was no decision on whether to postpone. The minister has said that that was an SQA decision. Given that you were telling the SQA all sorts of things that it had to do, why did you not tell it not to issue the results on that day if, as you feared, many were going wrong?

John Elvidge Scottish Executive Education Department
link

It was felt that that was a judgment that the SQA was best placed to make. You will see from the papers that its estimate of how many candidates might be affected by incomplete data was moving around—if not on a daily basis, over a wider margin than one would have expected over a short period of time. The final judgment had to rest on the balance between candidates who would be disadvantaged by the late issue of results and candidates who would be in a better position as a result of delaying the issue of certificates. At the end of the day, that came down to a practical management question: if there were a certain number of extra days, how many candidates who would otherwise receive incomplete information would receive complete information? We could not possibly make that judgment—only the people at the coal face of the system could. Having agreed with the SQA what the criteria were, Mr Galbraith decided that it was right to leave the decision to it in the light of that operational knowledge.

Michael Russell SNP South of Scotland
link

I will mention Mr Galbraith in a minute.On the evidence of 17 July, it seems that Eleanor Emberson was involved in all the detail of all the decisions. There is an item about the postal service and another about whether there were enough telephone lines—she was involved at an extraordinarily detailed level—yet from that day on your department does not recommend that there be a postponement. Would it have been possible for you or the minister to remove the chairman and the board?

John Elvidge Scottish Executive Education Department
link

If there were reasonable grounds, yes.

Michael Russell SNP South of Scotland
link

Given what you have told us about the phone call of 26 June, there might have been reasonable grounds, yet that was not done. Despite the fact that you were running the organisation and had those options before you, you did nothing to stop the disaster taking place on 9 August. Do you regret that?

John Elvidge Scottish Executive Education Department
link

I do not think that any of those actions would have affected the outcome at that stage. Before I forget, let me go back and correct myself on one point. I do not think that it is within our power to remove the chief executive.

Michael Russell SNP South of Scotland
link

I did not ask you that; I asked whether it was in your power to remove the board.

John Elvidge Scottish Executive Education Department
link

I thought that you did—in that case I have not misled you. It is within the minister's power to remove the chairman or members of the board if there are grounds to do so and if the unfitness of those individuals to hold their roles can be demonstrated.

Michael Russell SNP South of Scotland
link

It is clear from the letter of 17 July that you had realised that the organisation was in chaos. The detail of this letter is extraordinary. You knew that the issuing of the results was of vital importance to hundreds of thousands of young people. Do you regret not having done something more in the period leading up to 9 August? Do you regret not having advised the minister to do something more? Did you so advise him?

John Elvidge Scottish Executive Education Department
link

If I were able to identify an action that I believed would have changed the outcome, I would certainly regret that we had not taken it. However, we must recognise that, at that time, we were at one minute to midnight. I cannot think of any intervention that we could have made at that stage that would have changed the outcome.

Michael Russell SNP South of Scotland
link

Did you advise the minister to take any action at that stage?

John Elvidge Scottish Executive Education Department
link

You are tempting me, but you know that I have already said that I cannot reveal what advice, if any, I might have given the minister. You can infer from the fact that I cannot think of an action that could have been taken that I would have had difficulty communicating such an action to anybody.

Jamie Stone LD Caithness, Sutherland and Easter Ross
link

For much of the time, you were communicating only with the SQA, and I regret that. There were signals for other organisations. As Mike Russell has pointed out, your approach was then hands-on, and you weighed right in and started to try to sort things. Do you now regret that you did not go in further and grab the organisation, as it were, to ensure that it ran properly?

John Elvidge Scottish Executive Education Department
link

I do not believe that that is within our power. The clear intention of the legislation is for the operation of the examination system to be separated from political control. It is not my understanding that the powers could have been used effectively to take control of the organisation. The SQA remained—and remains—free to thank us for our advice, but to choose to do things differently. In fact, we were working closely together. Eleanor Emberson's letter may look like a list of instructions, but it may more accurately be regarded as a record of the steps that had already been agreed with the SQA as sensible.

Jamie Stone LD Caithness, Sutherland and Easter Ross
link

Yet you had the ability to offer IT support, which was not taken up. You could have gone further than you did.

John Elvidge Scottish Executive Education Department
link

We could have offered the SQA people or money: we could have offered to second people to it; we could have offered to supply it access to contractors; we could have offered it the money to employ other people. In effect, we did offer it those things. I do not think that it is within our power—I am conscious that we are in the sort of territory about which a legal textbook could be written—to say to people at the SQA, "You as an employer, irrespective of your own judgment, will employ X or Y."

Jamie Stone LD Caithness, Sutherland and Easter Ross
link

But the storm signals were there. You could see the shambles that was developing. You could see the conflicting figures that were appearing day by day, yet you are telling us that you felt it best for the SQA to run with it and with the date for the exams.

John Elvidge Scottish Executive Education Department
link

I am saying that—

Jamie Stone LD Caithness, Sutherland and Easter Ross
link

Do you admit that that was a mistake?

John Elvidge Scottish Executive Education Department
link

I am saying that it is a huge decision to take the running of a very complex system out of the hands of the people who know how to run it into the hands of people who do not have experience of it.

Jamie Stone LD Caithness, Sutherland and Easter Ross
link

But there was an extreme situation in which thousands of pupils stood to have their qualifications compromised, possibly blighting them for life. Was such action not demanded by that situation?

John Elvidge Scottish Executive Education Department
link

There is a danger of my sounding complacent or excusatory in saying this, but we have to consider the numbers.

Jamie Stone LD Caithness, Sutherland and Easter Ross
link

Thousands were affected.

John Elvidge Scottish Executive Education Department
link

They were thousands out of more than 100,000. As far as we could tell, the system would work correctly for 95 per cent or more of cases and might not work as well as it should in a proportion of cases. At times, including towards the end of the process, it was suggested that that proportion would be less than 1 per cent of cases. Removing the running of a complex system from the hands of the people responsible because one believed that one knew some other group of people who could achieve a 100 per cent result over the few weeks at their disposal would have been a huge decision. I would certainly not suggest to members that if such a decision had been taken the result would have been better. The most probable result of such a course of action would have been that many more thousands of young people would have experienced problems.

Jamie Stone LD Caithness, Sutherland and Easter Ross
link

So you are saying that you acted with the best intention at the time, but you now agree, with hindsight, that you made a mistake.

John Elvidge Scottish Executive Education Department
link

No, I would not agree with that. It is not like substituting a football player: there is no second SQA waiting to be brought on to the pitch. There is no organisation that could obviously have been substituted for the SQA.

Brian Monteith Con Mid Scotland and Fife
link

We keep hearing about IT. I want to be quite specific about this. According to the evidence before us, there was a meeting with your head professional adviser in the IT sphere to consider what stage things had reached with regard to software—I am referring to the APS. It seems that, following that meeting, you were mainly offering help with hardware. Is that the case? Were you in a position to help the SQA with software?

John Elvidge Scottish Executive Education Department
link

I do not think that we could have helped the SQA with either hardware or software. There was no indication that either the hardware or the software was the problem. Even if it had been, the SQA was dealing with perfectly reputable companies that were responsible for supplying them both hardware and software. We offered, in one form or another, people—foot-soldiers if you like. If the number of people trying to do the work was the problem, we were offering to try to help with that very simple difficulty.

Brian Monteith Con Mid Scotland and Fife
link

I wish to clarify that. Our evidence was that the Executive's integration suite was offered. That is hardware—or does that include people working there?

John Elvidge Scottish Executive Education Department
link

The integration suite is a physical facility in which one can simulate the interaction of two computer systems. We did offer its use. It is not like some sophisticated science lab of which there are only two in the UK; it offers a relatively well-understood testing approach. We simply happen to have a dedicated place where it is easy to do that, and we were offering the use of that physical facility.

Brian Monteith Con Mid Scotland and Fife
link

What would the people who were offered have done?

John Elvidge Scottish Executive Education Department
link

We were not specifying that; we were saying to the SQA, "If your judgment, as managers, is that more bodies will help you, tell us, and we will try to get you those bodies."

Brian Monteith Con Mid Scotland and Fife
link

Given the evidence that we have heard, particularly in the past 15 minutes or so, about the number of meetings, the escalation of your instructions, the change in the nature of the relationship between you and the SQA and the decision to let the SQA decide that it should proceed with the issuing of certificates on 9 August, for delivery on 10 August, was any advice given to the minister by you or by your department that a public relations disaster was about to, or could, happen? Was it suggested to him that he might be required to be available? Irrespective of what advice you gave the minister, did you not think it surprising that the minister was not making himself available, given all the preceding, escalating discussions?

John Elvidge Scottish Executive Education Department
link

I will try to stand on firm ground in approaching those questions. Once we had entered the phase in which it seemed certain that some candidates, even if a small number, were going to experience a problem, there was a lot of discussion about the public impact of that. Our primary concern was how to manage information flows in such a way as to minimise the distress caused to candidates. We believed, in the run-up to 9 August, that a set of arrangements to do that was in place. What happened on 9 and 10 August did not bear any relationship to the set of arrangements that we believed would be in place to avoid that distress to candidates. Had we known what was going to go wrong, we might well have thought that there would be a serious public relations issue that would have a political dimension. As it was, we believed that a sensible set of plans for differentiating the substantial majority of candidates who would not have a problem from the small minority who would have one, and for offering targeted help to the small minority, was in place. We were wrong.

Brian Monteith Con Mid Scotland and Fife
link

I follow your train of thought and your rationale, but given that higher still has been seen as a flagship policy of this Executive, if not previous Administrations, and given that the date on which exam papers are to be sent out and delivered is known well in advance, would you not have advised the minister that this would be an appropriate time for him to be available to champion the delivery of the policy?

John Elvidge Scottish Executive Education Department
link

Generally, I do not take the view that ministers need advice on political matters from me. We were not thinking of this situation as a set of issues that linked to wider debates that had a political profile; we were thinking of it as a concerning practical problem that we wanted to ensure was managed as effectively as possible and whose impact on candidates we wanted to minimise. The time to worry about the splash is when one has run out of time to control the size of the thing that is going to make it.

  1. Scottish Parliament Education, Culture and Sport Committee Wednesday 27 September 2000 (Morning) Committee business · 4
  2. Exam Results Committee business · 351

The type of each item (debate, statement, questions, members’ business and so on) and the times shown are worked out by this site from headings, the chair’s opening words and the Official Report’s running timestamps. They are not the Parliament’s own classification. Text: the Scottish Parliament Official Report.