• Home
• About
• Students
• HE staff
• FE staff
• AVAs

Learning from AVA review 1999-2004

QAA 047 06/2004

 Principle 5: Development and approval of programmes

 

Rationale

87 The approval of individual Access programmes is at the heart of the Recognition Scheme and is one of its key quality assurance processes. This principle acknowledges the importance of thorough and transparent development and recognition processes, and requires that the act of recognition is made with the full authority of the AVA as the holder of the licence.

Major issues from AVA review

88 From the evidence of the review reports, the process of programme development and recognition is of high quality in most AVAs. Many AVAs undertake these processes in a broadly similar fashion. It is thus common to read of extensive pre-recognition development supported by good paperwork and AVA officer involvement; properly constituted recognition panels undertaking their work in a purposeful and expert manner; and formal approval granted by a responsible committee within the AVA. A characteristic example is provided by an AVA where the review team noted that:

In the initial stages, a Development Officer works with tutors to ensure that submission documents are accurate and comprehensive in preparation for a more formal recognition or validation event. Detailed pro forma are issued by the OCN to cover individual unit specifications as well as more general programme and provider information. Separate documents are employed where a broader Credit Framework is offered for recognition together with more streamlined submission for additions to such Frameworks which are already recognised. In each case, [the OCN] also issues helpful notes for guidance.

Acting on behalf of the Quality Assurance Committee, it is the responsibility of the assigned Development Officer to approve the readiness of a proposal to progress to a formal recognition panel. In respect of proposed Access programmes, this panel's membership includes representation from further and higher education providers. The OCN provides general guidance to recognition panels and, specifically, to those which are undertaking the validation of Access programmes. Such guidance is detailed and includes a set of principles to which a proposed programme must adhere, together with advice on specifications for the award of the kitemark, notes on the approval of individual programmes of study, and arrangements for the accreditation of prior learning.

A formal recognition panel report records the decisions of the panel, including any recommendations or conditions...Examples of recognition panel reports considered by the review team demonstrated the detailed and careful consideration given to submissions...

In summary, and on the evidence of recognition panel reports and discussions with both officers and tutors from providing organisations, the review team was able to confirm the thorough and robust nature of the programme recognition process operated by [the OCN].

89 It is a measure of the routine high quality of programme development and recognition processes that this example is one of many which could have been cited. It is likely that this widespread good practice is related to the familiarity of the peer process for validation and recognition which has been operating for some time in both further and higher education. The faith in this process has worked to the advantage of AVAs in providing a basis for consistency in the recognition of Access programmes.

90 Review teams did raise a small number of issues in relation to recognition processes which might contribute to the further enhancement of already successful practice. The licensing criteria are clear that the recognition of an Access to HE programme should be made with 'the full authority and approval of the AVA'. In most cases, a recognition panel makes a recommendation to the quality committee or an equivalent body in the AVA whose remit indicates its authority to approve programmes. Review teams were critical of a number of AVAs where the power to approve a programme appeared to rest with the recognition panel rather than with a more senior committee within the AVA, for example:

The review team noted with some concern the absence of a clear, single point of responsibility within the AVA for the final recognition of Access programmes. Although the Q[uality] A[ssurance] C[ommittee] is notified of all programmes which complete the validation process, it is not clear that the Committee holds any responsibilities in relation to the approval of Access programmes, and there is no documentary evidence to suggest that this responsibility goes beyond individual validation panels. While the evidence presented for the review indicated that panels were properly conducted, the team considered it essential that the process should include verification of consistency by a body with some continuity of membership and holding formal responsibility for the recognition of Access programmes.

91 In the face of some feeling that, following a recognition event, such committee approval was routine and amounted to 'rubber-stamping', this captures the two main reasons why review teams raised this issue on each appropriate occasion:

• the need for responsibility for the final approval of Access to HE programmes to be specifically located within the AVA's governance structures to ensure that the decisions of recognitions panels are subject to accountability;
• the view, as one report commented, that 'validation panels are transient bodies selected for a specific purpose and, except in the case of fortuitous continuity of membership, are unlikely to be able to provide a structured consistency in overseeing the granting of the 'kitemark' to the variety of Access programmes which are approved by [the AVA]'.

92 In some cases, the purely formal nature of a committee's approval of the recommendation of a recognition panel might be a further example of the issue raised in paragraphs 45 and 46 above, that is, the practice of assigning a responsibility to a committee which is then not able properly to fulfil it. The act of final approval of an Access programme does not have to occupy one of the extreme positions occasionally cited in review reports, that is, either a nominal confirmation or a further complete and detailed scrutiny of the proposal. There are a number of methods by which AVAs can discharge their responsibility to exercise full authority in the approval of Access programmes through their deliberative structures. In essence, the role of the deliberative structure is not to conduct a duplicate recognition event but to ensure that the recognition panel has reached its recommendation through the proper exercise of the AVA's stated procedures.

93 The issue of consistency and comparability in the development of Access programmes was a feature of some reports. The setting of common credit targets (see paragraphs 111-113 below) is periodically discussed as a foundation for comparability of programmes. In some particular AVAs, a measure of standardisation is achieved by default through the adoption of curriculum structures that make use of banks of common units. Elsewhere, subject-based 'standing panels' are employed in some curriculum areas to encourage the development of units of a consistent and appropriate standard. There has also been the encouraging development of inter-AVA work to plan GCSE-equivalent units for mathematics, science and English. All of these developments should be encouraged as valuable contributions to a move towards consistency and comparability which will doubtless be given further impetus by the recent Access to Higher Education Development Project, the recommendations of which include the development of both qualification and subject-level benchmarks for Access provision.

94 Although it may appear to be a relatively minor detail, the issue of the eligibility to chair recognition panels was raised in a number of reports. The catalyst for this was the view of review teams that the independence of the chair should not be in doubt and should respond appropriately to the requirement in the criteria for 'externality, objectivity and consistency of process'. AVAs have different methods for identifying the chairs of their recognition panels: they may be drawn from the membership of the AVA (in many cases, more specifically from the membership of the quality committee), or from the officers of the AVA. Given the difficulty which can be experienced in locating chairs for panel events, some AVAs now use their own officers exclusively in this role (in some cases, this might include the chief officer). Review teams were critical of AVAs where an officer who might have been closely involved in the development stages of a programme was also appointed as the chair of the recognition panel for that programme; they also commented on other systems where the chairing of panels might be shared between a relatively small number of the same AVA officers. In essence, the importance of this issue is not the detail of whether AVA officers should be in the chair, but the way in which the choice of a chair indicates the essential transparency and independence of the recognition event as a whole. The ideal chair for such an event, irrespective of provenance, is a person who is competent and has no real or apparent interest in the outcome.

Commended practice

95 At the time of the relevant AVA review, the following examples of commended practice were noted:

• providing opportunities which allow students to enter higher education feeling confident that they have been adequately prepared for higher education [OCNW];
• the effectiveness of the support it offers to learners in their progression to higher education [OCNSW];
• its thorough and supportive processes for the development and validation of new programmes [OCNSW];
• the thorough and robust nature of its programme recognition process [NEMOCN];
• the support offered to providers in the process of unitisation of programmes [OCNNWM];
• the care taken in the development of Access programmes and the thorough and robust nature of the validation process [SEWOCN];
• the work of its subject standing panels, which ensure comparability of standards across Access programmes, between Access programmes and other equivalent qualifications, and across other OCNs in Wales [NWOCN];
• the work of its Curriculum Working Parties and the consistency of standards that their work brings to Access provision [HAVA];
• the clarity of the 'kitemark' specification produced as part of the validation process [SEWOCN];
• the thoroughness of its procedures for the evaluation of the programme recognition process, and the use made of evaluation for the enhancement of this process [OCNWNY].

Areas for further development

96 Some of the areas which might be developed further by AVAs in respect of this aspect of the Recognition Scheme are:

a ensuring that the recognition of an Access to HE programme is made with the full authority and approval of the AVA, and the need for responsibility for the final approval of Access to HE programmes to be specifically located within the AVA's governance structures to ensure that the decisions of recognitions panels are consistent and subject to accountability;

b continuing to work towards methods, both regionally and nationally, which will enhance consistency and comparability in the development and recognition of Access programmes.

Principle 6: Standards of achievement

 

Rationale

99 Given that it deals with both the quality of Access programmes and the standards which students must achieve to be awarded an Access to HE certificate, this is a central principle of the Recognition Scheme. The focus is on the ways that an AVA establishes and operates its systems for ensuring that the consistency and sufficiency of academic standards is maintained, predominantly through external moderation. This principle also incorporates the requirement for regulated and secure processes for the award and issue of the Access to HE certificate.

Major issues from AVA review

100 In contrast to their consideration of AVA practice in programme development and recognition, review reports delivered relatively few commendations and a relatively large number of conditions and recommendations in respect of the systems of moderation which are central to the principle covering standards of achievement. In considering the evidence of the reports, it is also noticeable that, in comparison with the standard patterns of procedure for programme recognition, AVAs have found a variety of different ways of undertaking what might seem to be fundamentally the same process of moderation.

General management of moderation

101 Despite this general context, there is evidence of good practice in the general administration of moderation including the selection and training of moderators. Where issues have been raised, they have tended to congregate around the responsibilities of the AVA as a validating body as expressed in the conduct of moderation. Some examples of these issues are:

• control of the choice and appointment of moderators. In many cases, moderators for Access programmes are drawn from HEIs although there are relatively few instances of an AVA selecting its moderator cohort solely from higher education as deliberate policy. AVAs frequently use their own member institutions as a pool for moderator selection. This poses few problems for large organisations but has given cause for comment with smaller AVAs, especially those with a limited number of HEIs in membership. In these cases, AVAs have sought to expand their cohort by appointing moderators from outside the AVA membership. As with other quality assurance processes, it is not unusual for AVAs to assign the formal approval of a moderator to its quality committee but for the process to be conducted by an officer. The committee thus exercises this responsibility with a light touch, or no touch at all. AVAs may wish to consider that the standards of achievement on the programmes which they validate - and therefore, their reputation as a validating body - rests on the quality of the moderator cohort which they appoint. In this context, it is not unreasonable to anticipate that the committee in the AVA's structure with overall responsibility for quality and standards should have a significant involvement in the appointment of moderators;
• the periods of office of moderators. AVAs have established different practices in this respect: some renew moderator contracts on an annual basis; some make appointments on a fixed basis (for example, three or five years); and some do not appear to have any formal limit on how long a moderator may serve. From the evidence of the reports, review teams have been critical of AVAs which do not appear to monitor the periods of office of moderators and, in extreme cases, have identified cases where the same moderator has been attached to an Access programme for many years. It is a principle in most academic quality assurance systems that moderators and similar external examiners should not, through time, become over-familiar with a particular programme. Following the best practice of a number of AVAs, the establishment of an appropriate period of office for moderators, and the subsequent monitoring of appointments to ensure that they are not exceeded, is one of the more straightforward messages from AVA review;
• the training of moderators. Review teams reported much good practice in the initial training of moderators and the provision of later updating opportunities. One report provides a typical summary: 'Moderators are required to undertake initial training and also periodic updating. [The AVA] issues a helpful and supportive Moderator Handbook and those moderators met by the review team endorsed the quality of the training and support they received'. The reference to a handbook for moderators is supported by many other references in reports to helpful guides and information packs produced by AVAs. Induction courses are also a common feature of practice, although there were some AVAs where induction and training was undertaken solely through paper-based guidance. The review reports present an interesting difference in the general approach made by AVAs to moderator training. In some AVAs, initial training was compulsory (in some cases, periodic renewal of training was also mandatory); in other AVAs, all training appeared to be discretionary. This latter practice may be a function of the most significant problem encountered by AVAs, that is, the securing of attendance at such training. However, given the importance of the role in relation to standards, review teams have been quicker to commend and encourage those AVAs which do not permit a moderator to serve without appropriate training. Other AVAs may wish to consider this practice as being fundamental to an organisation with awarding powers. It was also noticeable that the presence of an AVA officer or officers dedicated to the moderation process was valuable not just in the administration of the moderation system, but also because the staff involved tended to accumulate specific expertise and experience which could be put to good use in training events.

Models of moderation

102 In respect of the basic framework for moderation, AVAs presented a variety of models, with an occasionally bewildering range of nomenclature. Except in a small number of cases, however, they involved systems of moderation at both discipline and programme level. This is an appropriate response to the requirement of the licensing criteria that the 'sufficiency of standards of student achievement' is assured. A typical model therefore comprised subject-based moderators, who might be working at unit level in large modular frameworks or at programme level if an Access programme is discipline-based, and lead moderators whose task is to ensure the consistency of assessment across the range of a student's programme.

103 The most significant departures from this basic model are those where moderators are not appointed as subject specialists but, as one report describes them, 'arbiters of process':

In this model, it is necessary to have only one (or occasionally two) moderators for a programme, even where the programme includes pathways or subjects in which the moderator has no specialist knowledge or background...

Although there could be concerns that, with such a system, the level of subject-specific skills and knowledge could receive insufficient attention, moderators advised the review team that external moderation is supported by interaction with colleges' internal verification systems which consider the detail of subject-specific standards of achievement. The Account acknowledges that this model of moderation 'relies heavily on effective systems of internal moderation/verification', but also recognises that this position has not yet been achieved. Indeed, the OCN's current guidance for moderators suggests a developing model, stating that 'Increasingly, an organisation's internal moderation (or verification) system will become a key quality indicator and the moderator's role ... will become less of a curriculum specialist and more of a generic function'. Discussion with various participants in review meetings confirmed that the two systems are not yet working to complement each other in all cases...

...there is not yet evidence to suggest that all providers have rigorous systems of internal verification in place...[and]...the review team concluded that, although there was clear evidence that the OCN had begun to address this matter, the stage had not yet been reached where it could be assumed that moderation was supported by reliable and fully operational systems of internal moderation or verification.

104 The interaction between internal moderation or verification (commonly referred to as IV) and the moderation systems put in place by an AVA has been a developing theme in the review cycle. The example quoted above, which describes such interaction in its early stages, is from a review conducted in November 1999. In the same month, another review notes that:

One key external development influencing models of moderation has been the introduction and increasingly widespread use of internal verification (IV) within further education. The use of IV within Access programmes is still evolving and the review team noted some concern expressed by moderators as to the effectiveness and consistency of application of IV practices for Access. OCN guidelines have been produced with regard to IV but, with continuing developments in this area, these will need careful monitoring and there may be a need to strengthen the support and guidance available to providers and moderators, in order to ensure consistency of moderation practice in Access provision in relation to the use of IV.

105 This sense of evolution continued to be a theme in review reports and was seen as particularly crucial for AVAs which were making a transition towards the role of moderator as a quality auditor, focusing on standardisation and verification, and therefore becoming more dependent on internal verification to assure standards of achievement at subject level. In June 2000, in one such model:

Some providers had internal systems in place and some did not. Where these did exist, there were considerable differences and inconsistencies between them. Moderators had little guidance as to how they were expected to interact with internal systems, and practice was inconsistent. While a model of internal verification had been identified in one member institution as excellent practice which might be promoted and adopted, there was currently no clear process by which this might take place nor was there any certainty that it would be acceptable to all member institutions. The review team was unable to identify any minimum requirements made by [the OCN], either in relation to the quality standards for systems and processes of internal moderation/verification, or in relation to their articulation with [its] own external moderation processes.

and, in June 2001, in another AVA with:

...a model of moderation which appears to be in transition between a subject specialist model and one where the moderator becomes, to a great extent, an arbiter of process, heavily reliant on the successful implementation of an internal verification system. While both programme managers and moderators who met the review team expressed broad confidence in the system, the team noted that, in the composite moderation report...for 1999-2000, there were a number of comments on specific moderation reports which indicated that some aspects of internal verification continued to cause concern, and that moderators may be asked to take on trust that the college system is working, with minimal evidence provided to support a tutor's assertion.

106 By February 2003, review reports were indicating significant progress in this area, for example, an AVA where 'internal verification within provider institutions is integrated within the moderation process and used as a quality check by moderators' and, towards the end of the review cycle in May 2003, it is clear that integrated systems of verification and moderation were being considered with more confidence:

The AVA, through the Q[uality] A[ssurance] G[roup], is considering the appropriateness of the current system of moderation with two visits per year from all module moderators and also from the overall programme moderators. It is thought that, in view of a considerable reduction in the number of curriculum-related issues over the past few years and the increasing rigour of providers' internal moderation systems, it may be possible to reduce the level of review by external moderators of learners' work, relying more on internal moderation processes, thus possibly reducing the number of visits necessary by external moderators. Currently, both module and overall programme moderators review internal moderation processes to confirm their effectiveness rather than to utilise their outcomes.

107 The growth of internal verification has offered a series of interesting challenges to AVAs. From the evidence of the review reports, they have begun to meet those challenges by developing their approaches to moderation and seeking ways in which internal and external systems can be integrated. From the perspective of the Recognition Scheme, the licensing criteria do not legislate for any one particular model of moderation as long as there is an assurance of 'quality, comparability and fitness for purpose of Access to HE programmes', and of 'the consistency and sufficiency of standards of student achievement', that is, as long as the system of moderation guarantees standards at both programme and subject level. Models of moderation which use internal verification as the main source of evidence for subject-level standards can fulfil the criteria although ultimate authority should still rest with those appointed by, and reporting to, the AVA itself. It may also be a message from AVA review that those AVAs which do incorporate internal verification into their moderation systems should have in place appropriately rigorous processes for considering and approving the IV system which is proposed. In this context, scrutiny of the IV system of a provider would clearly become a vital part of recognition and review procedures.

Consistency of standards

108 Review reports have also commented on the methods employed by AVAs to ensure the consistency of standards achieved across the range of Access programmes which they validate. A limited number of AVAs have experienced some difficulties in the basic processes which contribute to consistency of judgement, for example, inconsistent practice in the operation of final assessment or examination boards. The use of standard guidelines for the conduct of such boards, and standard agendas has helped in this context.

109 In managing other aspects of consistency, it is not uncommon for AVAs to manipulate their moderation systems to ensure that each moderator works with more than one programme and at more than one provider. There is evidence from the review reports that such a method can contribute to a greater uniformity (although they are understandably more difficult to operate logistically). Often, AVAs have established particular groups to enable moderators to meet and discuss practice and standards. These moderation forums or task groups have the potential to contribute significantly to consistency of practice. In some cases, more ambitious initiatives have been launched to address the issue of standardisation. An example would be an AVA which has:

...organised standardisation events to review assessment practice. These focus on curriculum by looking at specific units and allow for the review of assessment practice and further confirm the standardisation of the award of level and of credit...

In its Account, the AVA states that 'it has made considerable efforts to engage Access practitioners and moderators in standardisation events and will be introducing an element of compulsion to ensure that participation in these activities is prioritised in the future'...The Account also notes, in relation to regional standardisation, that all colleges will be 'required to attend and provide samples'...the team considered the AVA's approach to engaging Access practitioners and moderators in standardisation events to be a positive and realistic response to a difficult situation.

110 Review reports commend all of these efforts while noting that many of them are undermined by poor or erratic attendance. A message from AVA review would be that any initiative aimed at enhancing the consistency of standards is to be encouraged. In order to lend greater support to AVAs, however, there is a growing feeling that, as an issue, standardisation should have a national dimension. This is certainly one of the significant issues covered by the recent Access to Higher Education Project Report.

Credit targets

111 In 1995-96, at the point at which the first cycle of AVA reviews had been completed by HEQC, it had been the intention to produce a report, Learning from Access Reviews: A digest of issues and recommendations from the reviews of authorised validating agencies 1994-1996. The report was begun but never completed, largely as a result of a combination of staff illness and the imminent transfer of regulatory responsibilities from HEQC to QAA. However, a draft of the report was presented to the then managing committee, HEQC's Access Course Recognition Subgroup. It is instructive to note what the draft had to say on the issue of credit targets:

For a 'kitemark plus' offer from HE, the Access qualification may come with 16 credits, but the student may be required to achieve say 24 credits for his/her chosen University place. Clearly this threatens to undermine the significance of the kitemark itself as the standard matriculation qualification. On the other hand, it is obvious and inescapable that strong competition does exist for places in HE, especially the more popular subjects/courses and the prestigious institutions, and HEIs have to find a basis for selection and conditional offers. Selection through individual in-depth interview (and tests) continues, but this is diminishing because it is so costly, and many HEIs distrust references alone as a basis for unconditional offers.

112 Almost 10 years on, the Access to Higher Education Development Project Report comments on:

the much-reported variation in credit requirements for the award of the Access to HE certificate, from the national minimum for AVAs which are Open College Networks, of 16 credits with 12 at level 3, to 22 credits required by one AVA using the same credit framework. Other work in this area suggests the upper limit of requirements for the award of the Access certificate is higher still for some courses...

This inconsistency in the requirements for the award of the Access certificate was the major cause of concern for HE admissions staff, who commented on the implications for consistency of offers when the variability between programmes' requirements was so great...[and]...were experiencing difficulties in applying fair admissions procedures in a situation of extremely diverse requirements and regulations...They suggested that the current position could create barriers for students who were not local to the HEI to which they were applying, and that it undermined the status of the Access certificate as a nationally recognised qualification.

Furthermore, admissions staff were concerned that these differences in descriptions of requirements for the award of the Access certificate could indicate a more fundamental variation in the actual required standard of achievement for the award of the Access to HE certificate, and that this produced a reluctance, in some areas of HE, to consider Access candidates. Discussions with national bodies also noted a concern about 'the relatively small proportion of Access entrants to pre-92 universities', which led to the suggestion that this may indicate a 'lack of recognition or confidence in the qualification'. The acceptance of the Access to HE certificate as a robust nationally recognised qualification is a matter of priority if those students who choose this route into higher education are to be able to apply successfully to all Universities, whether pre- or post-1992.

113 This appears to require little further comment. Review reports are a source of evidence of the variation in credit targets to which the Access to Higher Education Development Project Report refers, and which does not appear to have improved in almost ten years. Reports note, for example, 'a degree of uncertainty over the minimum credits required for an Access certificate', or 'that beyond the statement of the national credit minimum, the AVA did not have a developed policy in relation to credit targets on Access programmes'. Where there has been ambiguity in this area, review teams have normally requested that the AVA clarifies its position and works towards a greater degree of consistency amongst its providers. Echoing the Access to Higher Education Development Project Report, and ten years on from an initial identification through AVA review, a key message from this cycle of AVA reviews is that the issue of consistent credit targets must be resolved if the Access to HE certificate is to be confirmed as a national award.

The use of moderator reports

114 Most AVAs have in place a system which is designed to provide a functional response to moderators' reports and to resolve any issues which they raise. All such systems appear to have been satisfactory with the best of them providing models for the efficient processing of moderators' reports and an appropriate level of involvement by AVA officers and committees with responsibilities for quality:

The Moderation Officer is responsible for receiving and logging moderation reports, which are written to a prescribed format, facilitating comparison and consistency. The Moderation Officer, normally in conjunction with the Development Officer responsible for the institution to which the report relates, reads and evaluates all moderation reports. Any matters of concern are signalled to the institution and are reported to the relevant A[ccess] V[alidating] C[ommittee], although the Moderation Officer will take early action where appropriate. The Moderation Officer is also responsible for compiling an annual evaluation report for consideration by AVCs and the Q[uality] A[ssurance] C[ommittee]. The report will highlight a range of issues, make recommendations for improvement, and identify any issues of concern. Moderation reports and evaluations are available to the autumn meeting of the AVC for sampling. All reports which raise issues in respect of the quality of programme delivery or the validity of the award of credit are separately identified to the AVC for action. All actions are followed up by the Moderation Officer and reported back to the AVC on a regular basis. The AVCs report any issues of on-going concern to the QAC. Evidence presented to the reviewers confirmed that this system is working well and that difficult issues had been picked up and followed through to appropriate outcomes.

The control and issue of certificates

115 In keeping with the generally positive comments made in review reports regarding AVA administrative structures, teams found satisfactory procedures in place for the secure control and issue of certificates in almost all AVAs. In the isolated cases where some improvement in procedures was required, the issue was usually to ensure that the necessary procedures were totally in the control of the AVA and that they included a verification process which involved the appropriate moderator.

Commended practice

116 At the time of the relevant AVA review, the following examples of commended practice were noted:

• the potential of the system of moderation for developing comparability and consistency across partner institutions as well as in its own operations [OCNW];
• its arrangements for the training of moderators and the quality of the supporting documentation [OCNSW];
• the attention given to the induction of new moderators through briefing sessions [NEMOCN];
• the establishment of a Moderators' Task Group [SEWOCN];
• the introduction of a certificated programme of training for moderators [LOCN];
• the introduction of activities to monitor consistency of assessment [SEOCN];
• its recognition of the central importance of assessment to the quality of Access programmes, and the design of unit templates to assist the rigour of the assessment process and consistency of results [TRAC];
• in collaboration with the Lead Moderator, having developed and implemented the model of lead moderation in a robust and supportive manner [OCNSEM].

Areas for further development

117 Some of the areas which might be developed further by AVAs in respect of this aspect of the Recognition Scheme are:

a the further 'professionalisation' of their approach to moderation by establishing and monitoring fixed periods of office for moderators, and by ensuring - through mandatory programmes of initial and continuing training - the readiness of all moderators to undertake their prescribed duties;

b ensuring the involvement of key committees with responsibilities for quality and standards at appropriate points in the overall framework for moderation including the appointment of moderators and the receipt of, and response to, moderators' reports;

c the continuing integration of internal verification systems within overall moderation processes;

d initiatives aimed at enhancing the consistency of standards, and a readiness to become involved in any national projects which have the same purpose;

e the introduction of consistent credit targets across the range of Access provision within an AVA and a readiness to become involved in any national initiative which addresses the issue of credit targets for Access programmes.

Principle 7: Review and evaluation

 

Rationale

120 This principle expresses those aspects of the Recognition Scheme which require AVAs to establish mechanisms by which both their own operations and the Access programmes which they validate are subject to continuing review and enhancement. The intention of the principle is to encourage the development of a culture of quality and self-assessment within an AVA.

Major issues from AVA review

121 Given that in many AVAs the processes used for the periodic revalidation of programmes are based closely on those used for initial recognition, it is unsurprising that review reports are generally complimentary about prevailing practice. A standard pattern emerges from reports of a three to five-year period of approval before revalidation, an appropriate level of support from AVA officers for those providers undergoing revalidation, the use of moderators' reports and annual reviews from intervening years to inform the process, and similar levels of efficiency and expertise in the establishment and conduct of revalidation panels as had been applied for the initial recognition event. As an example, in one AVA:

...revalidation takes place when a course has run through the normal three years of approval granted for a new course. The procedures for revalidation, designed 'to encourage the planned development of Access courses and based on the operation of the course to date', are similar to those for validation with one significant exception. Incorporated into the proposal document must be a 'thorough analytical and evaluative review of the course, assessing the extent to which the aims and objectives of the course have been achieved and detailing all the changes already made or proposed'. If successfully revalidated, a further four years of approval will normally be given to a course. In looking at the evidence the review team was able to concur with the conclusion reached by [the AVA] in its account, namely that the AVA 'has developed vigorous and effective procedures for the process of revalidation.'

122 In respect of the regular scrutiny of Access provision, AVAs commonly maintain a system of annual programme review. The nature of the review may differ: some AVAs require specially produced documents; others make greater use of reports which have been generated as part of a provider institution's own internal quality assurance procedures. In the latter case, there is evidence that the consideration of annual reviews is facilitated by a uniformity of structure and that some problems have been experienced through the variety of provider-generated reviews. Good practice in annual review is not difficult to locate in review reports. In one example:

The OCN requires an annual Evaluation and Review report for each of the programmes which it recognises (a further summary report is also required where a provider operates programmes through a broader credit framework). A common format for such reports is encouraged which, in particular, requires the tabulation of major issues, the action proposed, responsibility for such action, and the timescale by which it will be achieved. Review reports are considered by officers of [the OCN] and, where necessary, issues are raised with providers or, where relevant, with moderators for the programme in question. In discussion with Access programme leaders, who generally confirmed the value of the annual review process, the review team heard that such feedback is usually provided promptly.

The Quality Assurance Committee is informed in the case of late submission of reports and considers the reasons for delay. Minutes of the Committee demonstrated that this responsibility was taken seriously, with some programmes being subject to additional monitoring in the subsequent year. Providers are also given notice that the withdrawal of recognition is a possible consequence of not fulfilling the requirement to submit reports on time...

The review team was able to confirm that the process of annual review was conscientious and carefully managed, with the OCN being active in enhancing its systems through evaluation of the outcomes of each round.

123 There are some examples of more complex systems of monitoring and review although, in general, review teams report that they do not always operate with appropriate efficiency. As an example, one AVA:

...has developed a five-year cycle of evaluation and reporting for Access programmes. This constitutes a requirement on providers for a full evaluation report in years one and four, and a short, summary report in year five, in addition to regular action plans. The evaluation cycle is designed to complement the external moderation process...In practice, the evaluation cycle does not appear to be operating with any consistency and the boundary between evaluation by providers and monitoring by external moderators is unclear. A review of programme files revealed inconsistency in the provision of evaluation reports...Those reports that are received by the OCN are not currently dealt with in any systematic way and it was acknowledged that this was an area of the AVA's procedures that needed development.

124 In some AVAs there is no separate system of annual review of programmes. In such cases, annual monitoring is usually undertaken through the use of moderators' reports. There is no compulsion in the licensing criteria for a separate process of annual programme review (or, if such a system is in place, that it should be conducted on an annual basis). However, the criteria, and their accompanying indicative evidence statements, do imply the receipt and consideration by AVAs of material beyond that normally included in moderators' reports. In many cases, AVAs appear to have responded positively to this position: there are examples where annual programme review has been made compulsory rather than voluntary; has been introduced for the first time; or has been re-introduced 'to reinforce the ability of the AVA to evaluate and monitor its recognised provision'. The overall message from AVA review in this respect is that specific annual programme review is considered as best practice in order to ensure the successful 'regular review and development...of Access to HE provision with reference to quality and comparability of programmes and consistency of student outcomes'.

125 One possible area for improvement in relation to annual programme reviews is their use in any overall process of self-review implemented by the AVA. There are few examples of the issues which are generated in annual review being used to inform the enhancement of the AVA's overall activities. The ways in which AVAs have, in general, met the licensing criterion which refers to 'the regular review and development of AVA activities and procedures', that is, the quality enhancement of the organisation as a whole, is difficult to judge from the evidence of review reports. Prior to the publication of the specific principles and licensing criteria in 2001, this area was not a feature of reports and, prior to 2000, was probably not a feature of AVA practice in any methodical or systematic fashion.

126 The point in the review cycle at which more regular mention is made of organisational self-review coincides with the introduction of the licensing process by NOCN and the subsequent first round of NOCN Quality Audits which were aimed at ensuring compliance with the licence requirements. At the heart of these licence requirements is the notion of quality development through systematic self-assessment. Hence by November 2000, an OCN/AVA is reported as operating 'a detailed quality development plan which has been introduced recently. The overall purpose of the plan would appear to be the demonstration of compliance and continuous improvement as driven by licensing requirements from NOCN' and, later in March 2001, a report notes that an OCN/AVA 'has begun to develop systematic self-assessment over the past year in order to ensure it is operating effectively against quality standards and the requirements of the regulatory bodies'. By June 2001, assumptions are being made regarding the impact of these quality standards: 'Compliance with recent NOCN licensing requirements, such as the process of self-assessment, formal production of an annual business plan and record-keeping requirements within a specific format, are likely over time to improve the effectiveness and consistency of [the OCN's] operations'. By 2002, it is clear that self-assessment has been developed further and more integrated as in the OCN/AVA where:

A 'continuous improvement model' was adopted in August 2002 in which strengths and weaknesses are identified in relation to four 'key product/service areas': [OCN] policy; regulatory requirements; the award of credit, NOCN qualifications and Access to HE qualifications; and services to external organisations and internal customers. This process results in the production of a self-assessment report and a continuous development plan. The Quality Committee is responsible for overseeing the self-assessment process and the Access Committee monitors progress on those items within the development plan which are relevant to the network's AVA activities. The potential effectiveness of this new procedure was demonstrated by the clarity and comprehensiveness of the documentation that it has generated and by the careful manner in which the documentation was considered at November 2002 meeting of the Quality Committee.

127 Given the partial nature of the evidence, and the fact that NOCN-driven self-assessment systems will only pertain in OCN/AVAs, it is difficult to draw any equitable or robust conclusions on the methods by which AVAs undertake the process of self-review for organisational enhancement. It is likely, however, that this area will be a much more significant part of the next review cycle and those AVAs which are not also OCNs - and therefore might not yet necessarily have introduced systematic self-assessment - may wish to consider the example of an internal quality assurance system commended in the later part of the review cycle.

Commended practice

128 At the time of the relevant AVA review, the following examples of commended practice were noted:

• the self-critical approach demonstrated through the Inspectorate system and the action taken in response to the recommendations of Inspectors' Reports [OCNW];
• having raised [its] profile in colleges through the Inspectorate system and improving quality by ensuring compliance with [its] systems and quality standards [OCNW];
• the maintenance of an overview of Access activities and trends in recruitment, achievement and progression since 1992 [OCNCE];
• the conscientious and careful management of the process of annual review [NEMOCN];
• cooperation with other OCNs on work on GCSE equivalent programmes [OCNNWM];
• its use and response to interim reports to ensure that early action is taken to rectify problems and deal with difficulties about programmes which are raised during the year [OCNKM];
• the comprehensiveness of the annual review of Access programmes [OxOCN];
• the steer provided to ensure that students have a voice in the processes for the review of Access programmes [OxOCN];
• the potential value to its quality assurance arrangements of the work of the Quality Audit Unit [AQA];
• its undertaking of a comparative study of Access to HE courses [AQA];
• its tracking of students in higher education [AQA];
• its internal quality assurance systems [OCNWM].

Areas for further development

129 Some of the areas which might be further developed by AVAs in respect of this aspect of the Recognition Scheme are:

a the introduction, where it does not already exist, of specific annual programme review;

b the establishment by non-OCN/AVAs of systematic self-assessment for the purposes of the enhancement of organisational quality.

Conclusions

130 In addition to the areas for further development listed at the end of each of the sections above, a number of general conclusions might be drawn from a consideration of this cycle of AVA reviews.

131 There are some examples quoted in this report of progressive improvements, for example, in areas such as the integration of internal verification systems and advances in self-assessment but, in general, these tend to be the result of external forces or imperatives. It is difficult to demonstrate that the overall AVA review cycle per se was used as a tool for quality enhancement by AVAs while it was being undertaken. A reader is therefore likely to find remarkably similar conditions and recommendations in the earliest and the latest reports in the review cycle. Using only the review reports themselves, it is clearly impossible to make any authoritative claim, but there is certainly no evidence that they were considered routinely within AVAs, either tactically - to note which aspects of practice were receiving praise or criticism - or more generally as a ready-made source of discussion of quality issues facing all AVAs. If accurate, this is disappointing since it may indicate a lack of awareness of the value of external information and intelligence in the general consideration of quality assurance matters. It may also say something about the current and future nature and composition of those AVA committees with responsibilities for quality and standards. In addition to their functional responsibilities in areas such as recognition, review, and moderation, it is likely that under self-assessment systems they will also have broader responsibilities for the principles and values of quality in the organisation as a whole. In which case, consideration of sources of information such as the reports generated over a five-year period by AVA review should become a key feature of their work.

132 There is of necessity a healthy tension between conformity and diversity in the interpretation of the Recognition Scheme and the licensing criteria. If this were not the case, then a somewhat dull regime of compliance would be likely to restrict innovation by AVAs and limit the possibility of necessary regional differences. However, a consideration of areas of broad similarity, and areas of distinct variety, is of interest. In key quality assurance processes, for example, it is clear that recognition practices are very much the same across the range of AVAs; it is equally clear that moderation systems can be very different. On the evidence of the review reports, there is more good practice to commend in the former, and more practice which attracted conditions and recommendations in the latter. There is a strong temptation to draw a causal link here. In one sense, AVAs themselves appear almost to have established an invisible code of practice for programme recognition. Would there be merit in some similar agreement on the process of moderation? And should this be encouraged by some appropriate amendments to the Recognition Scheme? Given the issue about the balance between diversity and adherence to some common code, it might be helpful if - in the case of moderation and its key role in maintaining standards - there were not multiple methods adopted for doing fundamentally the same thing.

133 Particularly in the mid-1990s, a common theme in the first round of AVA reviews was the way in which the transition between AVA to OCN/AVA was being managed and how the AVA function, role and presence was being protected (it was indeed to have been a focus of the unfinished Learning from Access Reviews: A digest of issues and recommendations from the reviews of authorised validating agencies 1994-1996). That period of transition is now presumably almost complete but, in this second cycle of reviews, there have still been residual concerns about the AVA function in OCN/AVAs. These are often expressed in issues related to governance and strategic planning where review teams have sought to identify the precise role and identity of Access in an OCN/AVA where it may, in truth, be a minor presence. However, the issue now is not the management of transition but the ways in which OCN/AVAs manage the fact that they have (at least) two regulatory bodies. A first acknowledgement is that OCNs as organisations currently exist through their licence from NOCN and that this will naturally dictate their sense of first loyalty. This should not mean, however, that Access and the OCN's role as an AVA needs to be wholly subsumed. In some areas OCN and NOCN methods and practices are not always appropriate for the requirements of the Recognition Scheme and OCN/AVAs need to recognise this and react appropriately. The evidence of the review reports tends to suggest that there is value in maintaining a distinct body within the OCN/AVA's deliberative structure as a focus for its AVA work (although not necessarily with responsibilities for quality assurance).

134 The issue of the size or the critical mass of AVAs has been raised on a number of occasions in this report. The issue is a relatively simple one: evidence of personal competence is to be found throughout the review reports and at no time do the reports (or the licensing criteria) make any general assumptions about a minimum size for an AVA. However, it is difficult to ignore the fact that the larger, well-resourced AVAs have decided advantages. Their breadth of membership allows a greater choice and opportunity for staff of member organisations in areas such as appointment to panels, working groups and committees; the same breadth also offers more opportunities for the AVA itself to secure the expertise it needs to operate some of its key processes. A larger staffing base also offers the prospect of more development support and enhancement opportunities for providers, especially if specific posts are created for Access provision. The demands of external regulatory bodies such as QAA and NOCN have led to greater pressure on the resources of AVAs and it appears, from the evidence of the review reports, this has been felt more keenly in the smaller AVAs: it is unlikely that these demands will ease in the foreseeable future.

135 Given the existence of the Access to Higher Education Development Project and its report, what might be learnt from AVA review cannot stand on its own. The messages from the Project, and the experience gained from the review cycle, should be invaluable partners. As an example, the fulfilment of one aim of the Project - to confirm the national recognition of the Access to HE award - has much resonance with learning from AVA review. Some of the key messages of this report are crucial for the national status of the award, including the unquestioned independence of the AVA as a validating body, the ways in which the AVA accepts full responsibility for its actions as a validating body and the need for clarity in the credit specification of the award.

136 On the evidence of the review reports, and on any other evidence base, AVAs have been extraordinarily successful in the development of Access. As the name implies, their first responsibility has been for the validation of Access provision although, of course, they have been involved in a much greater range of Access-related activities. However, it is possible that the recommendations of the Access to Higher Education Development Project Report may represent a watershed for AVAs. Certainly, some of the recommendations indicate a deliberate development of their role and anticipates them being substantially involved in the development, enhancement and promotion of Access and other learning opportunities for adults at a regional level, and active contributors to broader regional and national strategies for access and widening participation. AVAs have never been simple passive validators of Access programmes; in the future, however, more may well be asked of them in accepting responsibility for reaching out and proactively creating opportunities for adult learners on a broader regional basis.

Learning about AVA review

The review process

137 At the point of the transfer of responsibilities for Access regulation from HEQC to QAA in 1997, it was decided that the initial review cycle would be six years, a period determined, in part, by a plan for mid-term 'minor' reviews, a plan which was later overtaken by the introduction of a process of annual reporting by AVAs instead. The review process itself was initially developed by the QAA Assistant Director with responsibility for Access, with reference to other QAA review methods, and discussed and agreed in early meetings of the Access Recognition Advisory Committee and the Access Recognition and Licensing Committee (ARLC).

138 The review process typically extends over nine months from the point where an AVA is informed of its review dates to the publication of the final report. In broad terms, the process follows the model which is used by QAA for other of its review and audit activities:

• the organisation under review produces a document analysing and evaluating its activities (the Analytical Account);
• the document is considered by a team of peer reviewers supported by a QAA Assistant Director and a visit schedule is planned;
• a visit takes place during which the review team gathers evidence to test the organisation's description of itself in the self-assessment document; the evidence sources comprise relevant papers and minutes, and discussions with key staff and others connected with the organisation;
• a report is written, and a recommendation is made, on the basis of the evidence gathered.

Prior to the review visit

139 The review process begins with an initial information meeting between the AVA and the QAA Assistant Director, during which the scope of the review is described and any queries on the part of the AVA are clarified. The dates for the review are also provisionally set. The AVA then submits the evaluation of its activities, the Analytical Account, together with other supporting material which describes its quality assurance procedures and standard operational processes. The Analytical Account is prepared with reference to guidelines issued by QAA.

140 Some six weeks prior to the review, the selected review team attend a planning meeting, the principal purpose of which is to enable the team to decide on its main lines of enquiry and to establish a schedule and draft programme for the review visit. The team may also request further documentation from the AVA if it believes that this will assist in its full understanding of the AVA's structures, processes and procedures. The review team also meets on the day immediately preceding the visit to discuss any additional information which has been received.

During the visit

141 The review visit normally extends over two days. The common pattern is for the first day to be spent in discussion with groups involved in, and affected by, the AVA's activities. For each meeting, the reviewers work to a previously agreed agenda. The QAA Assistant Director makes notes of the discussion at all meetings. The review team will also allocate time to read and consider other written sources of evidence during the visit: in most cases, these comprise AVA records related to a selected number of individual Access programmes. On the afternoon of the second day, the team considers its key findings in relation to the licensing criteria and the areas it wishes to highlight in its report.

142 The recommendation to be made to the ARLC is also decided before the end of the visit. The outcomes following a review are:

• unconditional confirmation of renewal of licence for a specified period;
• conditional confirmation of licence with conditions to be met by a specified date or dates;
• provisional confirmation of licence with conditions to be met and a further review visit by a specified date;
• withdrawal of licence for operation as an AVA;
• temporary renewal of licence with request for further information by specified date (decision suspended).

After the visit

143 According to a division of responsibilities agreed at the planning meeting, the team compiles its report in the two weeks following the review visit. Since the formal approval of the principles and accompanying criteria in 2001, reports have been written to a template which reflects those principles. The QAA Assistant Director is responsible for the compilation and editing of the final report using the contributions from the two reviewers. The confirmed report is presented to the next scheduled meeting of the ARLC by a member of the review team who answers any queries which the ARLC may have about the report or the team's recommendation. On the basis of the draft report and its own discussions, the ARLC reaches a decision on the renewal of the AVA's licence.

144 At this point, the agreed report and the decision of the ARLC is forwarded to the AVA. The AVA is asked to consider the report and to highlight any perceived factual inaccuracies or judgements which it believes are based on a misunderstanding of the facts. The response of the AVA is circulated to the review team who may then agree amendments to the text of the report. In cases where the AVA's response indicates that the security of the final judgement may be in doubt, the report and response is referred back for reconsideration by the ARLC.

145 In cases where conditional or provisional judgements are reached, the AVA supplies the requested additional material by a specified date or dates. The reviewers are asked to consider this material and indicate whether they believe that the conditions have been appropriately met. All reviewers' responses are considered by the the ARLC prior to informing the AVA of the committee's decision about the renewal of the AVA's licence.

Evaluating the review process

146 Following a licence review, QAA invites each AVA to submit an evaluation of the review process. Guidelines are offered which suggest structuring the evaluation under the three headings of: the preparation for the review, the review visit, and the review report and follow-up. By no means all AVAs took advantage of this invitation during the review cycle but where evaluations were received, they were considered by ARLC. Members of review teams are also asked to complete a Feedback Report following each review which included a self-assessment section to enable them to reflect on their own performance during the review. Evidence from AVA evaluations and reviewer feedback has been used in this section, although in neither case has it been ascribed.

The planning stages

147 It is clear that both AVAs and reviewers appreciated the advice and assistance of the QAA Assistant Director who managed the complete cycle of reviews from 1999 to 2004. AVA evaluations commented in particular on the usefulness of the initial planning meeting and the opportunity to understand fully the scope and logistics of the review process. These comments were supported by others which noted the readiness with which advice and support was offered during the whole of the review process. An appreciation of the work of the Assistant Director was also a common theme in reviewers' feedback particularly the support given to new reviewers and the general expertise provided in Access matters. There seems little doubt that this cycle of reviews benefited significantly from the consistent presence of an expert officer: it helped to ensure the overall stability of the review process and the reliability of judgements delivered by review teams.

148 In the early part of the review cycle, AVA evaluations commented on the brevity of the period between the planning meeting and the review visit. AVAs reported difficulties in preparing for the visit with what they considered to be the late receipt of a detailed programme. As the review cycle progressed, a longer period of time between the planning meeting and the review visit was built into the process. In addition, as the process developed and programmes began to share common characteristics, it became easier to provide guidance to AVAs about the likely pattern of, and participants in, the review programme. Although AVAs would wish for as long a period as possible in which to prepare, it is likely that this period has now been extended as far as is possible without creating such a large gap between the two events that the initial preparatory work of the review team is invalidated.

Review teams

149 The basic principles in the selection of review teams are that they are knowledgeable about the Recognition Scheme and Access in general, and are able to examine an AVA independently and impartially. Many reviewers commented on the importance of achieving a balance in review teams reflecting complementary interests and expertise. They had also found the training offered by QAA to be appropriate in most respects - particularly the inclusion of observation at a 'live' review - although some felt that further emphasis on report writing would have been welcome.

150 Building on one aspect of practice inherited from the first HEQC round of reviews, virtually all review teams contained at least one member of the ARLC. The functional reason for this was to facilitate the presentation of reports to the committee. However, it also reflected a belief that, in order to undertake its responsibilities properly, the ARLC needed not only to be an expert committee but also to be in constant touch with the major process which it employed for its licensing activities. In a very practical sense, the ability of the committee to undertake the licensing of AVAs, and the expert nature of its discussions, are aided considerably when all members have direct experience of the review process. A balance needed to be struck, however, between such direct experience and the consistency it adds to the process, and the possible limiting of the cohort of reviewers (in all 25 individuals were involved in AVA review of whom 12 undertook more than one review). With a relatively small number of reviews being undertaken in any one year, there were limited opportunities for drawing in new reviewers, developing the experience of others and ensuring that all review teams included at least one experienced member. That there were few negative comments made by AVAs on the composition of review teams tends to suggest that the personnel, mix and balance was generally appropriate.

Analytical Accounts

151 As might be expected across a large number of AVAs, the nature of Analytical Accounts varied greatly. The feedback from review teams frequently contained references to Accounts which were particularly helpful or unhelpful. In the former category were those Accounts which set out the major activities of the AVA in a clear and comprehensible manner; included useful guides to major changes since the previous review visit; provided evaluations of the AVA's work in key areas often indicating those where some improvements were needed or were underway; and included well-organised and informative appendices with appropriate cross-references to them in the text. Less helpful Accounts shared few or none of these qualities.

152 A good Analytical Account is valuable for a number of different reasons including its primary purpose of allowing the AVA to reflect upon its own activities as a preface to its licence review. Whatever the future development of AVA review, it is difficult to imagine that self-assessment of some kind will not continue to form an integral part of the process. One message from this cycle of AVA review is that further advice and guidance from QAA on the compilation of Accounts and further sharing of practice between AVAs would be worthwhile exercises.

The review process and review visits

153 The essence of the review process is the statement and evaluation by the AVA of how it considers it is meeting the requirements of the Recognition Scheme, and the testing of this by the review team. A further essential feature of the process is that review teams cannot cover every aspect of the operation of an AVA during a licence review and therefore must employ sampling techniques to decide on the effectiveness of the AVA's work, its quality assurance arrangements, and its overall operation of the licence. The review process seeks to ensure the robustness of its findings through the proper construction of sampling, the careful consideration of the evidence provided, consensus between team members, and the advice of the Assistant Director who ensures a level of consistency of practice between reviews.

154 One of the key sampling techniques, the range of meetings conducted during a review, appeared satisfactory to both AVAs and review teams themselves. Although there was some variability at the beginning of the cycle, on the evidence of the review reports, the nature and number of meetings settled into a reasonably predictable pattern. People invariably seen included AVA officers and staff, members of key committees including those specifically for quality and Access matters, moderators, Access programme coordinators, HE admissions tutors, and students. The only issue raised by review teams was that, not infrequently, personnel who had been invited to attend a meeting as part of the agreed schedule were then not able to be present at the review itself. The absence of key personnel, or the general overall reduction in the size of groups, may in some cases have had an effect on the conduct of the review, for example, where a review team had asked to meet eight or 10 Access programme leaders but where only two or three were eventually able to attend. In such cases, the evidence gathered from the meeting needs to be treated with particular care since it is much less likely to constitute a reasonable sample from which to draw conclusions. There is every sympathy for AVAs in this context: in all reviews, the majority of people seen were not employed by AVAs who therefore had little absolute control over their commitments and eventual presence. However, if such meetings are to continue to be used in the process as an important source of evidence, then some resolution to this issue needs to be found.

155 Other than in this area, there was much praise from review teams on the good administration of review visits by AVAs. Additional evidence for sampling, such as the papers related to a range of individual Access programmes validated by the AVA, were generally made very accessible, were well laid out and referenced, and made this part of the review team's work significantly easier.

156 There was little in the feedback from AVAs to suggest that the review visit had not been conducted satisfactorily in the majority of cases. There were some occasional comments that teams may have been overly adversarial in their tone and approach, but in many more cases teams were noted for the professional and thorough approach which they took.

Decision-making process

157 On the part of AVAs, the most common criticism of the whole review process was the lack of any immediate feedback at the conclusion of the review visit itself and the provision of some sense of closure to an exercise which, at that point, would already have extended over several months. It has been the position of QAA throughout that it is the ARLC and not the review team which has the authority to make decisions on the renewal of a licence. For this reason, QAA has considered that the review team should not share their recommendations with the AVA at the conclusion of the review visit, since those recommendations may be subject to amendment by the committee.

158 This would not be a sustainable position if the ARLC's approval of a review recommendation was a cursory action. However, as suggested in paragraph 143 above, the consideration by the ARLC of the report and recommendation of a review team is not perfunctory and may involve a good deal of detailed discussion. These discussions can lead to changes to review reports and their conditions. Part of the rationale for this method of operation is to ensure that decisions on licence renewal are as equitable and consistent as possible. Although some feedback to AVAs at the end of a review visit might be perceived as desirable, it would remain an inappropriate exercise for as long as the ARLC undertakes its responsibilities for licence renewal by the current method.

Future options

159 The purpose of this concluding section is to reflect on the nature and conduct of AVA review and to discuss the possible options and developments for the next cycle. An initial question, of course, is whether there should be any changes at all. With some necessary amendments, can the basic review process used between 1999 and 2004 be employed again? The issue is perhaps not the durability of the review process, but the nature of what is being reviewed. There have been substantial developments and significantly new challenges for AVAs in the period since 1999. Some of these, for example, the growth of internal verification methods, the introduction of internal audit, the changing requirements of regulatory and funding bodies, have been touched upon in this report. There would seem little doubt that the process of licence review must also change in order to ensure that it is appropriate for the next stage in the development of AVAs and Access provision.

160 It might be argued that there are three factors which should influence changes to the nature of AVA review:

• the development of the relationship between periodic licence review and the annual reviews which QAA requires from each AVA;
• the report of the Access to Higher Education Development Project;
• the impact of the 1999-2004 cycle of reviews.

161 The annual review process applied to AVAs has been progressively developed by the ARLC since its piloting in 1999. ARLC devotes a substantial amount of time to the consideration of these reviews and, increasingly, uses them as a significant indicator of the continuing health of an AVA. The logic of this development is that the use of annual reviews must be allowed to influence the conduct and application of periodic review. In a context where annual review is well established, and where the ARLC is increasingly confident of the evidence which it provides, should an AVA which consistently demonstrates its effectiveness and suitability to hold and operate its licence, be subjected to a different form of periodic review? If the two processes of annual and periodic review are to be seen as coherent and mutually supportive, then this option would appear to be an appropriate one to explore in more detail. In so doing, the nature and content of the annual review submitted by an AVA would need to be the subject of further consideration to ensure that the report is capable of supplying the necessary and consistent level of assurance for the ARLC.

162 The Access to Higher Education Development Project has already been mentioned several times in this report. If the recommendations of the Project report are accepted and implemented, they will constitute a substantial development of Access and will perhaps lead to a different concept of the AVA. Some new or expanded responsibilities would be assigned to AVAs leading to changes to the licensing criteria and, consequently, changes to the nature of AVA review.

163 The impact of the 1999-2004 cycle of reviews should not be ignored, particularly where good and consistent practice has been identified, or where relatively straightforward operational features have been checked. Inevitably, part of the cycle of reviews has been devoted to straightforward monitoring against the criteria (a process which has been made significantly easier for review teams since the formal articulation of the principles and licensing criteria). If the 1999-2004 cycle of AVA reviews has been effective, then the next cycle should not need to monitor in detail, for example, the process of recognition (a clear point of consistent and good practice) but might wish to focus on how AVAs assist in targeting the development of new programmes; equally, there should no longer be the need simply to check the existence of an equal opportunities policy in an AVA, but more emphasis on how such a policy, and others like it, is implemented across the activities of the AVA.

164 Although the nature of what is reviewed may be changing, and given that any process can always be enhanced and improved, there is little from AVA review to suggest that the fundamentals of the current method need to be radically amended. A peer process, the catalyst for which is self-assessment, involving a visit and a variety of sampling techniques, might still provide the basic platform for review. Considering the three factors above, however, the themes which might dominate the next cycle and the philosophy of the review method might change. It is likely, for example, that such themes might include:

• self-assessment and a greater use of, and focus on, an AVA's internal quality audit or assessment practices as an indicator of its maturity as an organisation;
• the academic standards of Access programmes and the ways in which AVAs use the internal moderation practices of their providers and their own external moderation in order to ensure that the Access award is confirmed as a national qualification indicating clearly a readiness for higher education;
• the regional role of an AVA, its involvement in broader policies and strategies for adult learning, the purposeful targeting of disadvantaged groups, and the generation and analysis of data in order to achieve this;
• the conception of the AVA as a 'training' body as well as a validating body, being equipped and willing to provide a broad programme of initial and updating opportunities for the development of all those involved in Access provision;
• if the recommendations of the Project are implemented, the ways in which AVAs adopt and implement the qualification and subject benchmarks that are envisaged.

165 As for the 'philosophy' of an amended review process, QAA has within its own operational experience what might appear to be a helpful model. The first complete round of academic audits of HEIs from 1991 to 1997 was characterised in part by its relatively detailed scrutiny of quality assurance processes and their practical operation. QAA's reflection on the nature of the second round of audits led it to develop and then implement the process called 'continuation audit'. In essence, the difference between the two processes was one of focus; where first-round audit teams checked for themselves whether quality assurance processes were operating effectively, continuation audit teams were more interested in scrutinising an HEI's own methods for providing such assurance. Key questions, therefore, were how does an institution know that it is discharging effectively its corporate responsibility for the standard of each award granted in its name? And how does it assure the sufficiency, validity and reliability of the evidence that it is relying on for this purpose?

166 AVA review between 1999 and 2004 had already adopted some of the key characteristics of continuation audit, particularly the reliance on the AVA's own account of its activities as a basis for the conduct of the review. It is an option for the future for AVA review also to adopt that approach of continuation audit which scrutinises not the detail of processes, but the ways in which an AVA itself undertakes the assurance of the quality of those processes.

 

Appendix A

Authorised Validating Agencies

Authorised Validating Agency	Acronym (as used in this report)	Further information	Notes
Anglia South Open College Network	ASOCN	www.asocn.co.uk
Assessment and Qualifications Alliance	AQA	www.aqa.org.uk
Cambridge Access Validating Agency	CAVA	www.cava.ac.uk
Chiltern Region Open College Network and Access Consortium	CROCNAC	www.crocn.org.uk
Greater Manchester Open College Network	GMOCN	www.gmocn.ac.uk
Hampshire Authorised Validating Agency	HAVA
London Open College Network	LOCN	www.locn.org.uk
Merseyside Open College Network	MOCN	www.mocn.co.uk
North East Midlands Open College Network	NEMOCN	www.nemocn.org.uk
North Wales Open College Network	NWOCN	www.nwocn.org.uk	Known as North Wales Access and Credit Consortium (NWACC) a tthe time of its review in April 2000.
Open College Network (North and East London and Hertfordshire)	OCN (NELH)	www.ocnetwork.co.uk
Open College Network (SouthYorkshire and Humber Region)	OCNSYH	www.shu.ac.uk/ocn
Open College Network (TROCN)	TROCN	www.trocn.co.uk
Open College Network for Central England	OCNCE	www.ocnce.warwick.ac.uk
Open College Network Kent and Medway	OCNKM	www.ocnkm.ac.uk
Open College Network North West Midlands	OCNNWM	ocnnwm@staffs.ac.uk
Open College Network of the South West	OCNSW	www.ocnsw.org.uk
Open College Network of the West Midlands	OCNWM	ocnwm.wlv.ac.uk
Open College Network South East Midlands	OCNSEM	ocnsem@ocnsem.com
Open College Network West and North Yorkshire	OCNWNY	www.wnyocn.org.uk	Known as West and NorthYorkshire Open College Network at the time of its review in November 2001.
Open College of the North West	OCNW	www.ocnw.com
Oxfordshire Open College Network	OxOCN	ocn@oxocn.org.uk
South East Wales Open College Network	SEWOCN	croeso@sewocn.co.uk
South of England Open College Network	SEOCN	www.brighton.ac.uk/seocn/
Thames Region Accrediting Consortium Open College Network	TRACOCN	www.tracocn.co.uk
Western Region Open College Network	WROCN	info@wrocn.co.uk	Known as Open College Network West at the time of its review in March 2002

---

Appendix B

Details of AVA reviews and licence applications, 1999-2004

The titles of AVAs are those in use at the time of the review or licence application

AVA	Review date	Review team	Outcome
1998-99 (academic year)
Open College of the North West	26-27 April	Professor Beverly Sand (University of Derby) Dr Clive Linnett (Bradford & IlkleyCommunity College)	Provisional Renewal
Open College Network of the South West	20-21 May	Dr Peter Easy (Cheltenham & Gloucester College of HE) Ms Miriam Griffiths (Open University in Wales)	Conditional Renewal
Open College Network for Central England	16-17 June	Professor Steve Bristow (Shrewsbury College of Arts& Technology) Ms Margaret Davidson (Charles Keene College)	Unconditional Renewal
Bedfordshire Access Consortium*	24-25 June	Dr Pete Johnston (University of Essex) Ms Carole Stott (LOCN)	Provisional Renewal
1999-2000
North East Midlands Open College Network	9-10 November	Dr Peter Easy (Cheltenham & Gloucester College of HE) Ms Sue Georgious (OCNC)	Conditional Renewal
Open College Network NorthWest Midlands	17-18 November	Dr Pete Johnston (University of Essex) Mr Alan Smith (NWACC)	Conditional Renewal
University of Ulster Authorised Validating Agency*	8-9 December	Professor Beverly Sand (University of Derby) Ms Miriam Griffiths (Open University in Wales)	Conditional Renewal
Open College Network (North and East London and Hertfordshire)	10 February	Dr Pete Johnston (University of Essex) Professor Steve Bristow (Shrewsbury College of Arts & Technology)	Licence Awarded
South East Wales Open College Network	1-2 March	Dr Peter Easy (Cheltenham & Gloucester College of HE) Mr Anthony McClaran (UCAS)	Provisional Renewal
South West Wales Open College Network*	15-16 March	Dr Geoffrey Copland (University of Westminster) Mr Steve Babbidge (SEOCN)	Provisional Renewal
North Wales Access and Credit Consortium	6-7 April	Professor Steve Bristow (Shrewsbury College of Arts & Technology) Ms Christine Davies (Nottingham Trent University)	Conditional Renewal
London Open College Network	1-2 June	Professor Beverly Sand (University of Derby) Mr Mike Farmer (Cheltenham & Gloucester College of HE)	Conditional Renewal
Tyne and Border Counties Access Partnership*	14-15 June	Professor Janet Finch (University ofKeele) Mr Malcolm Barry (Goldsmiths College, University of London)	Licence Withdrawn
2000-01
Open College Network Kentand Medway	8-9 November	Dr Pete Johnston (University of Essex) Mr Malcolm Barry (Goldsmiths College, University of London)	Provisional Renewal
Oxfordshire Open College Network	22-23 November	Mr Steve Babbidge (SEOCN) Ms Priscilla McGuire (OCNWNY)	Provisional Renewal
South of England Open College Network	22-23 March	Mr Anthony McClaran (UCAS) Ms Sue Georgious (OCNCE)	Conditional Renewal
Assessment and Qualifications Alliance	27-28 February	Dr Peter Easy (Cheltenham & Gloucester College of HE) Mr Alan Smith (NWOCN)	Provisional Renewal
Thames Region Accrediting Consortium Open College Network	7-8 June	Professor Beverly Sand (University of Derby) Ms Gill Evans (OCNNWM)	Conditional Renewal
Chiltern Region Open College Network and Access Consortium	13 June	Dr Pete Johnston (Independent) Professor Tony Chapman (University of Wales Institute Cardiff)	Licence Awarded
2001-02
West and North Yorkshire Open College Network	20-21 November	Dr Pete Johnston (Independent) Mr Anthony McClaran (UCAS)	Conditional Renewal
Open College Network West	5-6 March	Mr Steve Babbidge (SEOCN) Mr Anthony McClaran (UCAS)	Provisional Renewal
Hampshire Authorised Validating Agency	15-16 May	Dr Pete Johnston (Independent) Dr Philip Bentley (Shrewsbury College of Arts & Technology)	Conditional Renewal
Cambridge Access Validating Agency	28-29 May	Dr Peter Easy (University of Gloucestershire) Dr Rob Allen (University of Greenwich)	Provisional Renewal
2002-03
South West Wales Open College Network*	5 November	Dr Pete Johnston (Independent) Dr Andy Thompson (NESCOT)	Application Referred
Open College Network South East Midlands	19-20 November	Mr Steve Babbidge (SEOCN) Professor Colin Raban (Edge Hill College of HE)	Conditional Renewal
Greater Manchester Open College Network	11-12 February	Ms Sue Georgious (OCNCE) Mr David Burtenshaw (HAVA)	Conditional Renewal
Open College Network (TROCN)	18-19 February	Dr Pete Johnston (Independent) Ms Jill Ward (University of Derby)	Conditional Renewal
Open College Network of the West Midlands	13-14 May	Dr Peter Easy (University of Gloucestershire) Mr Alan Smith (NWOCN)	Conditional Renewal
Merseyside Open College Network	7-8 May	Mr Mike Farmer (University of Gloucestershire) Dr Philip Bentley (Shrewsbury College of Arts & Technology)	Provisional Renewal
2003-04
Open College Network (South Yorkshire and Humber Region)	12-13 November	Mr Steve Babbidge (SEOCN) Ms Jill Ward (Wedgwood Memorial College)	Provisional Renewal
Anglia South Open College Network	19-20 January	Dr Peter Easy (University of Gloucestershire) Mr Keith Fletcher (OCNSW)	Conditional Renewal

Normally, a review team included one member of the ARLC in order to involve the committee directly in the process of licence review (and to facilitate the presentation of the draft report to the committee). ARLC members are identified by the use of italics. In cases where no ARLC representative was present, the team included at least one member with previous experience of AVA review, who was invited to report on the review outcomes to the ARLC. In the case of applications for new licences, both team members are drawn from the ARLC.

* = no longer operating as an AVA.

ISBN 1 84482 083 1

| Share your success | Contact | Privacy statement | © QAA 2012