AVA annual self-evaluation report

Guidance notes, 2007-08

(report due at QAA: 1 December 2008)

Part 1 – AVA annual self-evaluation report

Introduction
Preparation and presentation
Report length
Submission
AVA self-evaluation report checklist
Sections for inclusion in self-evaluation reports
Special appendix 2008

PART 1: Annual self-evaluation report – guidance to AVAs (2007-08)

Introduction

Purpose

The consideration of AVA annual self-evaluation reports plays a key role in the monitoring of AVAs by the Quality Assurance Agency for Higher Education (QAA). Together with the system of AVA review, AVAs’ self-evaluation reports provide the means by which QAA can satisfy itself and assure others that an AVA is fulfilling the terms of its licence and continues to be a fit body for the development and validation of Access to HE programmes and the award of Access to HE qualifications. The requirement for a self-evaluation report echoes the expectation of the AVA licensing criteria that there should be a ‘culture of quality and self-assessment within the AVA’.

Annual self-evaluation reports make a key contribution to the AVA review process, providing the primary documentary evidence base for reviews. Review teams may also be asked to follow up queries raised by self-evaluation reports, and AVAs are asked to provide a response to recommendations resulting from review findings in their subsequent self-evaluation report.

AVA self-evaluation

The preparation of a self-evaluation report requires each AVA to assess the effectiveness of its performance against QAA’s published AVA licensing criteria, which are grouped in four main areas: governance; management; quality assurance enhancement and development; and academic standards. The need for comment in all four areas requires the AVA to report on both its evaluation of its own performance and of the provision for which it holds awarding body responsibilities. The self-evaluation report also includes an analysis of the AVA’s statistical data.

QAA makes no specific requirements about the means by which an AVA undertakes its self-evaluation, but the self-evaluation report should bring together the outcomes of routine and explicit procedures, undertaken with reference to the particular requirements of the AVA licensing criteria. The resulting written report which is shared with QAA provides the AVA with a formal record of the outcomes of its self-evaluation and a tool by which the AVA can monitor its own progress and continuing quality improvement.

Self-evaluation reports are expected to be evidence-based documents. The report’s standard appendices include copies of formal AVA documents which should provide the evidential base to support the judgements in the main text of the self-evaluation report.

Scrutiny and outcomes

Self-evaluation reports are submitted to QAA’s Access Recognition and Licensing Committee (ARLC) which is charged with considering whether QAA can, on the basis of the report provided, be confident that each AVA is fulfilling the terms of its AVA licence.
Following the ARLC’s process for the scrutiny of self-evaluation reports, AVAs receive individual feedback on their self-evaluation reports and are notified of whether their reports have been

approved
accepted (with a request for further clarification or information)
not approved (in which case further action is taken).

If the ARLC’s scrutiny of the self-evaluation report and any follow-up enquiries suggests that there is some doubt about whether the AVA is meeting the terms of its licence, there may be a request for further information or a special meeting or review may be arranged. If initial concerns are confirmed by this further investigation, the ARLC may specify particular actions which the AVA is required to take as a condition of continuation of the licence.

Preparation and presentation

In considering the approach to be adopted, AVAs should note that

an objective evaluation of the AVA’s progress and developments in the previous year, based on the outcomes of the AVA’s own processes for evaluation and self-evaluation - rather than a merely descriptive or celebratory account of activities - should be presented
the AVA self-evaluation report reflects on the AVA’s performance and Access to HE-related activities during the year which is the subject of the self-evaluation, from the perspective of its position at the start of the following year
the self-evaluation report highlights any problems or difficulties affecting the AVA’s work during the report year, and how the AVA has responded to these and its plans for further action, as well as noting successes and positive outcomes of its work
the self-evaluation report includes evidence that the AVA has monitored its position against the AVA licensing criteria and identified areas in which work may be needed for the AVA to continue to meet the expectations of the licensing criteria in full.

In preparing the self-evaluation report for submission, AVAs should ensure that

full account is taken of feedback provided by QAA to the AVA in previous years
material is organised according to the sections given in this guidance (see Sections for inclusion in AVA annual self-evaluation reports,page 6 onwards), withthe headings listed here used as section headings for the main text of self-evaluation report
a tone and style which is appropriate to the report’s formal status is used throughout
acronyms, local terminology and references to local circumstances are explained, to allow readers unfamiliar with the AVA or its regional context to understand its work
use is made of sub-headings, lists, tables, charts and process diagrams, if it helps to provide information more succinctly and directly: lengthy, detailed narrative is avoided.

In preparing the self-evaluation report’s appendices, AVAs should ensure that

only one copy of any individual document is included in the appendices
appendices are used to support and verify statements in the main text, and material is not included in appendices without reference to its significance in the main text of the report
appendices are clearly and accurately cross-referenced to the main text of the report
appendices appear together at the end of, and separated from, the self-evaluation report
individual appendices are grouped, numbered and titled as indicated in this guidance
where revised documents are included, the date of publication and (where relevant) version number is included on documents
where appendices are grouped (eg committee minutes, provided in Appendix 5), these are separately identified by use of sub-classifications (eg 5(a), 5(b), 5(c) etc)
appendices/groups of appendices are clearly marked and/or separated from each other
double-sided copying is considered for lengthy documents included in appendices, in order to reduce the overall bulk of self-evaluation reports.

Before submitting the self-evaluation report, AVAs should check each copy and ensure that

the report has been proof-read and any corrections have been made to each copy
all specified material has been included, including the required appendices (see the Annual self-evaluation report checklist, page 5)
pages have been numbered in sequence throughout and numbers are consistent with those given in the contents page
pages have been properly collated
reports are securely bound, and each copy is presented as a single document under one set of durable covers which will withstand several readings and journeys.

Report length

Main text

The main text of the self-evaluation report (excluding appendices) should be 5,000-7,000 words. Any self-evaluation report that exceeds this guidance by more than 10 per cent (ie where the main text is more than 7,700 words) will be returned to the AVA for editing. An AVA would be expected to edit a report that was returned to it within five working days in order for the report to be considered for approval at the ARLC’s report scrutiny meeting at the beginning of January.

Appendices

There are 20 standard appendices (or groups of appendices) required for self-evaluation reports. Most of the required appendices (see Annual self-evaluation report checklist, page 5) will be documents which have been produced prior to the preparation of the self-evaluation report for the AVA’s internal purposes.

If additional information is needed in order to provide the evidence of the AVA’s self-evaluation outcomes, this is included, wherever possible, as a clearly labelled sub-section within the standard appendices. Any further appendices are included at the end of the run of standard appendices and numbered in sequence. The total number of appendices (or groups of appendices) will not exceed 25.

Submission

AVAs should submit one electronic copy and five hard copies of the annual self-evaluation report, to arrive at QAA’s offices no later than Monday 1 December 2008.

Electronic copies of self-evaluation reports should be sent by email attachment to access@qaa.ac.uk. If any appendix is not available in electronic format, the contents page should state that the appendix is available in hard copy only.

AVAs should also send a separate email to the same address, stating that the self-evaluation report has been sent. This will alert QAA to any difficulties that might arise with the electronic delivery of the self-evaluation report.

Printed copies of self-evaluation reports should be sent to:

Ann-Marie Karadia
Project Officer (Access to HE)
Quality Assurance Agency for Higher Education
Southgate House
Southgate Street
Gloucester GL1 1UB

AVA self-evaluation report checklist

Main text	Appendices
Executive summary
Contents page (including sections of main text and appendices)
Statement of formal approval (signed & dated)	annual self-evaluation report preparation (including the process of formal approval). This may be shown in a diagram.
Section 1 Governance	list of AVA stakeholders (marked to show higher education institutions and Access to HE providers), with additions and deletions since the previous year diagram of the AVA’s committee structure, with reporting lines list of the AVA committees (include membership/representation and names/affiliation of members during the report year) dates of committee meetings during the report year (any cancelled? dates changed?) minutes of AVA board/governing body minutes of committees (including finance committee)
Section 2 Management	strategic plan and targets for report year strategic plan and targets for the coming year staff structure (eg organisational diagram) and list of current staffing (including job titles; fractional and temporary posts; dates of new appointments) Access to HE financial information externally verified accounts
Section 3 Quality assurance, enhancement and development	samples of new quality assurance documentation AVA evaluations of providers’ programme reports list of the AVA’s development events and target audiences
Section 4 Academic standards	AVA evaluations of moderators’ reports moderators’ names, affiliation and length of service
Section 5 Commentary on statistical data programme data student data	Access to HE providers and programmes, with additions and deletions since last year a print-out of 2007-08 statistics submission previous two years’ statistics submissions
Section 6 Special Appendix
Response to AVA review recommendations (required where an AVA review has taken place since the last annual self-evaluation report was prepared)

Sections for inclusion in self-evaluation reports

Evidence of approval

The final version of the report must be formally approved by the AVA’s governing body. Approval should be evidenced through a formal statement, provided on a separate sheet at the front of the report, which a) specifies the date of the meeting at which the report was approved and b) is signed by the Chair of the governing body.

App 1: an outline of the process for the preparation and approval of the annual self-evaluation report. This may be shown in a diagram.

Contents page

Section headings with associated page numbers should be given. Appendices should also be clearly numbered (including any sub-divisions) and listed on the contents page.

Executive summary

A summary of the main outcomes of the AVA’s self-evaluation which is reported in detail within the main text of the report itself should be provided. Also include:
a brief contextual overview, providing basic introduction for readers new to the AVA (eg the organisation’s size and structure; geographical reach and scale of its activities)
the AVA’s position at the beginning of the year and its priorities for Access to HE
opportunities and major challenges in the external environment for the AVA’s work
internal factors influencing the organisation's progress in relation to Access to HE during the report year

how the events of the year have affected the organisation as a whole, its position at the end of the year, and key priorities for the coming year.

1 Governance

This section should evaluate the effectiveness of the AVA’s governance structures, with particular reference to any significant changes in legal or constitutional status.
Consider, in particular

legal, constitutional or governance changes that have been made and reasons for them
changes to the involvement of stakeholders (including receiving HE institutions and Access to HE providers), including any known reasons for changes
committees’ activity (in terms of the range of business undertaken, frequency of meetings and patterns of attendance)
policy decisions or measures taken or planned by the AVA in relation to developing the scope and range of its activity
other key decisions taken affecting the AVA’s work and how they arose.

The text of section 1 should be cross-referenced to relevant appendices:

App 2: a list of stakeholders, marked to show HEIs and Access to HE providers, and indicating any additions and deletions since the previous year
App 3: a diagram of the AVA’s committee structure, showing reporting lines
App 4: a list of the AVA’s committees and indication of their composition in terms of representation and/or named individuals and their institutional/organisational affiliation
App 5: a list of dates of committee meetings which took place during the report year, including any cancelled meetings or changes to dates
App 6: minutes of the AVA’s governing body
App 7: minutes of all meetings of key AVA committees, and the committee that has responsibility for finance, and for quality assurance of AVA business. If, for reasons of confidentiality, it is necessary to submit edited minutes, the points at which any material has been omitted, and the subject of omissions (eg ‘staff disciplinary hearing’) must be made clear.

2 Management

This section evaluates the success of the AVA’s strategic plans in developing Access to HE; and reports on its assessment of organisational risk and its management and operational functions.
Consider, in particular

Strategic planning process – evaluation of the AVA’s strategic planning process, including the process for the development, approval and monitoring of the strategic plan
Response to the report year’s targets - evaluation of the AVA’s progress in meeting its targets. Minor detail in this area will be included in the tabulated plan included in Appendix 8, but evaluation of the AVA’s progress with major targets should feature in the main text, with reference to difficulties experienced or circumstances which led to any targets being unmet
Targets for the coming year – reasons for the inclusion of new targets in the plan for the coming year should be identified: targets for Access to HE should be related to the AVA’s broader strategic aims, and to areas identified by the AVA as requiring action
Resources - evaluation of the organisation’s financial position and financial management; contribution and costs of AVA work to the organisation’s finances; accommodation and staffing and their adequacy for the AVA’s functions
Communications – overview and developments; evaluation of success of communications and promotional work with Access to HE providers and others
Self-assessment and risk-assessment –summary of the AVA’s internal processes for organisational self-assessment and risk-assessment processes undertaken during the year.

The text of section 2 should be cross-referenced to relevant appendices:

App 8: strategic plan and targets for the report year.

The format of strategic plans varies, but all AVAs’ plans should include - or make reference to a more detailed document which includes - formally approved targets for the AVA and its Access to HE work. If there is more than one plan, their status and relationship to each other, and the period to which they relate, should be clear. The appendix should show whether and how each target was achieved; how the AVA intends to pursue, or revise, any unmet targets, and whether such unmet targets have been revised, transferred to the coming year’s plan, or are now redundant

App 9: strategic plan and targets for the coming year. Tabulated plans relate strategic objectives to specific targets which are measurable and time-limited and which indicate designated individuals/groups with responsibility for meeting the targets

App 10: staffing structure and staff list, with job titles, and indicating fractional/temporary posts, and dates of new appointments.

App 11: Access to HE financial information to show how much of the AVA’s income and expenditure in 2007-08 (in absolute and proportionate terms) was derived from its Access to HE activity; the different sources of income (for example membership charges, validation/revalidation fees, certification fees, consultancy) and allocation of expenditure (for example staffing, administration, training, moderation, staff development, special projects) – and whether there was anything exceptional about income or expenditure in 2007-08

App 12: full audited accounts¹ for the most recent complete financial year. Accounts should be signed and dated.

3 Quality assurance, enhancement and development

This section evaluates the AVA’s quality assurance processes for Access to HE provision, noting developments and the reason for their introduction; improvements they are intended to bring about in the quality of Access to HE programmes or the AVA’s assurance of their quality; and an evaluation of their effectiveness since being introduced, where possible.
The section also evaluates the AVA’s success in establishing its position as a focus of development, through enhancing current Access to HE provision, initiating new developments, and engaging with developments initiated by others.
Consider, in particular

changes to the AVA’s quality assurance processes for course development; validation; revalidation; and course monitoring, with explanation for, and evaluation of, any changes
conclusions drawn about the quality of Access to HE provision from its processes for course monitoring, including the outcome of its evaluation of programme (or centre) reports, highlighting positive developments; general trends and patterns; and particular areas of difficulty, and areas for further development
summary and evaluation of development and enhancement events organised during the year, noting any particular outcomes
evaluation of the AVA’s involvement in regional or national projects, as well as smaller-scale local initiatives, designed to improve or develop Access to HE; evidence of productive cooperation outside the AVA with other organisations involved in widening participation in higher education and assessment of progress made in these areas.

The text of section 3 should be cross-referenced to relevant appendices:

App 13: samples of new forms designed for quality assurance purposes (evidence of minor amendments, or forms used for other purposes would not be expected)

App 14: summary evaluations of programme (or centre) reports, including key findings

App 15: a list of development or enhancement events organised by the AVA (with dates).

4 Academic standards

This section summarises outcomes of the AVA’s monitoring of academic standards on Access to HE programmes and of its procedures for monitoring standards, including assessment, standardisation, moderation and the award and issue of Access to HE certificates (or Diplomas).

Consider, in particular

assessment of moderation process, particularly in relation to the impact of any recently introduced refinements or changes to the AVA’s procedures
outcomes of measures undertaken to monitor moderators’ performance
analytical response to outcomes of moderation activity through the AVA’s evaluation of moderators’/examiners’ reports. As well as recording successes and notable achievements, the response will note any apparent weaknesses or areas of concern, and the AVA’s intended actions in relation to these. AVAs are not expected to make judgements about named programmes’ performance, but the self-evaluation report will mention how the AVA has addressed unsatisfactory quality or standards it has identified
trends relating to the quality and standards of Access to HE provision and student achievement, as noted by moderators in their reports
any contextual information that may be needed to different categories of moderation or moderators which are not self-explanatory.

The text of section 4 should be cross-referenced to relevant appendices:

App 16: summary evaluations of moderators’/examiners’ reports, including key findings

App 17: moderators’ names, affiliation and length of service.

5 Commentary on statistical data

This section provides an analytical and in-depth commentary on the statistical data collected for the AVA’s statistical report to QAA (see Part 2 – Data submission – guidance to AVAs), and provides the basis for a detailed discussion of how it relates to characteristics of Access to HE provision during the report year. The commentary provides the outcome of a process of analysis and interrogation of the data, identifying any anomalies and attempting to explain their cause: it attempts to explain, rather than describe, the data.
Where the statistics are self-evident or unremarkable, commentary may not be necessary, but the self-evaluation report should comment on the most striking features of the data and their significance and consider, in general

local circumstances which help to explain particular features of the data, where known (supporting anecdotal evidence can be illuminating: mere conjecture is largely unhelpful)
features which may not be evident or obvious from the raw data included in the report alone, particularly any factors that may have distorted the aggregated statistics presented
differences in, and ranges of, data received from different providers and known reasons for these. Although the self-evaluation report itself will probably not discuss the performance of individual providers by name, reference in general terms to considerations of differential performance may be needed for an external reader to have a proper understanding of aggregated or averaged statistics received from a number of different institutions and programmes.
reasons for incomplete data in any section, with details of action taken by the AVA to gain full data from all providers outlined
comparisons with previous years’ statistics, perceived trends and any known reasons for these
comparisons between the AVA’s own data and published national data, with any known reasons for variance from standard patterns. Comments may also include references to data gathered for other purposes, relating, for example, to comparative regional data.

The self-evaluation report will include comment on, in particular

Programme data - changes to the provision for which the AVA is responsible, and whether this has changed the overall profile of Access to HE provision. Areas to note may include areas of expansion and contraction; changes to providers; types of programmes recognised; progress with validations/re-validations; programmes withdrawn; and gaps in provision. Information about recruitment to different types of programme (whether described by mode, location, explicit targeting or defined intended progression route) is also useful
Learner data – comment on what is shown by learner profiles about the extent to which provision is reaching its target groups. Learner achievement and intended progression should receive comment, particularly with reference to emerging trends. Although the statistical returns do not require information on reasons for student withdrawal, information gleaned from providers about trends or correlations with particular target groups may prove useful.

The text of section 5 should be cross-referenced to relevant appendices:

App 18: a list of QAA-recognised Access to HE programmes and providers, with an indication of additions and deletions since the previous year

App 19: a hard copy of the statistics submitted electronically to QAA

App 20: previous years’ statistics for at least the last two years, sufficient to demonstrate trends described in the commentary.

Response to recommendations set at AVA review

Only AVAs which have recently been reviewed complete this section. Information is provided about how the AVA has responded to each recommendation. If the AVA has decided not to take action in response to any particular recommendation, the reason for this decision is clearly stated. If the AVA’s response to recommendations has been included in the AVA’s strategic (or other) plans, reference to each particular recommendation is explicit and easily identifiable.

6 Special appendix 2008

This year’s special appendix will provide an analysis and commentary on trends in the AVA’s data over the last three years (ie 2005-06; 2006-07; 2007-08), with a particular focus on the reasons for those trends. QAA is particularly interested to learn more about the factors that lie behind the declining numbers reported by most AVAs in recent years, so that it has a full understanding of the various mechanisms at work and is fully informed about what kind of action might be needed. Of equal interest, however, are any significant instances of growth and the kind of action that has led to an increase in provision or take-up, and examples of good practice in successful recruitment, such as effective mechanisms for converting enquiries to registrations.

There are two parts to this appendix:

a) a commentary by the AVA on trends shown in its own data – as collected and submitted to QAA since 2005-06 – based on its own understanding of factors influencing the provision. While it will draw on analysis that has been conducted previously, this commentary should be the result of fresh analysis and consideration of the data: it should not just summarise the data commentaries provided with each year’s annual report, and it should seek to explain and interpret the data. This part will consider the full range of data categories.

b) a survey by AVAs of all centres delivering QAA-recognised Access to HE provision. This survey will be undertaken by individual AVAs, but will use a common questionnaire constructed and analysed by QAA. The questionnaire builds on the outcomes of discussions at the 2008 Access to HE annual conference, taking the possible reasons for the trends that were put forward by delegates as the starting point for this more detailed and extensive survey. This part will focus on course numbers and learner registration numbers. The questionnaire will be forwarded separately to AVAs to allow them to forward it directly to providers for completion.

The process for completion of the questionnaire will involve these steps:

QAA supplies the questionnaire with standard questions to AVAs in electronic format.
AVAs forwards the questionnaire to each centre, with covering letter from QAA and instructions about how, and by when, to return the questionnaire.
Each centre will complete and return the completed form to QAA.
QAA will inform AVAs of which centres have returned completed questionnaire.
AVAs will chase any centre which has not returned the questionnaire to QAA.
QAA will process the data received from centres.
QAA will supply a data report to each AVA.
Each AVA will complete its response to the data report.

The final data report will be supplied to AVAs by the end of October. If this schedule does not allow sufficient time for the AVA to include part b) of the special appendix within its formally approved report, this section may be included separately, but at the same time, as the full report

¹The following common circumstances should be dealt with as indicated:
If the AVA’s financial year is not the same as the reporting year:
- submit the audited accounts for the most recent financial year (indicating the period that it covers), and management accounts for the period of the reporting year that is not covered by the audited accounts.
If the AVA is an unincorporated association and its accounts are not audited:
- submit detailed accounts which have been externally verified by some other means, and explain how they have been verified.
If AVA’s accounts are audited through a ‘host institution’:
- submit accounts that relate to the AVA, extracted from the ‘host institution’ accounts, if separately identifiable, and the management accounts presented to the AVA’s governing body. If the AVA’s accounts are not separately identifiable within the host institution’s accounts, explain in the main text how the AVA’s accounts are verified, and submit the management accounts presented to the AVA’s governing body.