AVA annual report Guidance notes, 2005-06

(report due at QAA: 1 December 2006)

Part 1 : Annual report – guidance to AVAs (2005-06)

Introduction
Preparation and presentation
Submission
Annual report checklist
Sections for inclusion in annual reports
Appendices

Introduction

Purpose

The consideration of Authorised Validating Agency (AVA) annual reports and statistics plays a key role in the monitoring of AVAs by the Quality Assurance Agency for Higher Education (QAA). Together with the system of AVA review, annual reports provide the means by which QAA can satisfy itself and assure others that an AVA is fulfilling the terms of its licence and continues to be a fit body for the validation of Access to HE programmes and the award of Access to HE certificates.

The preparation of annual reports involves AVAs in a process of review and evaluation of their work. This contributes to AVAs' own quality assurance and enhancement activities, and provides a means by which AVAs can inform others about their activities and developments.

Main emphases

The AVA annual report brings together the outcomes of the AVA’s monitoring procedures on quality and standards, and reports on activities aimed at developing and enhancing Access to HE. The report’s appendices include copies of formal plans, minutes and other records, providing an evidential base for the comments, analysis and judgements in the main text of the report. The process of self-evaluation involved in the preparation of the report requires AVAs to make an assessment of the effectiveness of their performance against QAA’s published AVA licensing criteria in relation to a range of matters including governance, management, academic standards, quality assurance, development and enhancement.

Scrutiny and outcomes

Annual reports are submitted to the Access Recognition and Licensing Committee (ARLC) which is charged with considering whether QAA can, on the basis of the report provided, be confident that each AVA is fulfilling the terms of its AVA licence.

Following the ARLC’s annual report scrutiny process, AVAs receive individual feedback on their reports and are notified of whether their reports have been:

approved
accepted (with a request for further clarification or information) or
not approved (in which case further action is taken).

If the ARLC’s scrutiny of the report and any follow-up suggests that there is some doubt about whether the AVA is meeting the terms of its licence, there may be a request for further information or a special meeting or review may be arranged. If the ARLC’s concerns are confirmed by this further investigation, it may require the AVA to take certain actions as a condition of continuation of the licence.

Annual reports also make an important contribution to the AVA review process, forming part of the source material on which review teams make their assessment of AVAs. The ARLC may ask review teams to follow up queries raised by annual reports, and AVAs are asked to provide a response to recommendations resulting from review findings in their subsequent annual report. AVA annual reports thereby have a central role of this quality monitoring circle.

Preparation and presentation

The AVA annual report reflects on Access to HE-related activities during the year which is the subject of the report, from the perspective of the following year. It looks back to the achievements and challenges of the previous year and reviews the AVA’s activities and developments, adopting an evaluative approach which seeks to be objective and analytical, rather than merely descriptive or celebratory. As well as noting successes and positive outcomes, the report also considers problems or difficulties and discusses how these affected the AVA’s work during the report year and how the AVA has responded.

There are standard requirements for all annual reports, but AVAs may also include additional information within the section headings provided, if it helps to give a more complete account of their activities. AVAs are encouraged to emphasise the particularity of their activities and of the Access to HE provision for which they hold awarding body responsibilities.

In preparing the report, AVAs should ensure that:

full account is taken of feedback provided by QAA to the AVA in previous years
material is organised according to the sections given in this guidance (see Sections for inclusion in AVA annual reports), with the headings listed here used as section headings for the report. Where additional material is included, or material could be included in more than one section, AVAs apply their judgement about where it is best placed for clarity and in order to avoid repetition
a tone and style which is appropriate to the report’s formal status is used throughout
acronyms, local terminology and references to local circumstances are explained in sufficient detail to allow readers unfamiliar with the AVA to understand its work
lengthy, detailed narrative is avoided: the main text (excluding appendices) is about 5,000-7,000 words, but use is made of sub-headings, lists, tables, charts and process diagrams, if it helps to provide information more succinctly and directly
full analysis is provided where requested – particularly in relation to the AVA’s data
appendices are used to support and verify statements in the main text: appendices are clearly cross-referenced to the main text of the report and material is not included in appendices without some reference to its significance in the main text of the report
double-sided copying is considered for lengthy documents, in order to reduce the overall bulk of reports
the report is securely bound, and presented as a single document under one set of durable covers which will withstand several readings and journeys.

Before submitting the report, AVAs should check each copy of the report and ensure that:

the report has been proof-read and any corrections have been made to each copy
all specified material has been included, including the required appendices (see the annual report checklist, page 4)
all sections of the main text of the AVA’s report use the same titles as in this guidance
appendices appear together at the end of the report and dividing pages are used to separate appendices from each other and from the main text of the report
appendices are clearly numbered and titled (using the same title on the document as on the contents page and in any references made in the main text)
cross-referencing is accurate
page numbering is used throughout and is consistent with the numbers given in the contents page
pages have been properly collated.

Submission

AVAs should submit one electronic copy and five hard copies of the annual report, to arrive at QAA’s offices no later than Friday 1 December 2006.

Electronic copies of reports should be sent by email attachment to access@qaa.ac.uk AVAs should also send a separate email to the same address, stating that the report has been sent. (This will alert QAA to any difficulties that might arise with the electronic delivery of the report.)
Hard copies of reports should be sent to:

Ann-Marie Karadia
Project Officer (Access to HE)
Quality Assurance Agency for Higher Education
Southgate House
Southgate Street
Gloucester
GL1 1UB

Annual report checklist (full details provided below)

Statement of formal approval (signed and dated)
Contents page (including sections of main text and appendices)
Main text	Appendices
Introduction	Annual report preparation process diagram.
1 Legal, constitutional and governance matters	Diagram of the AVA’s committee structure, with reporting lines. List of the AVA committees (include membership/representation). Dates of committee meetings during the report year (any cancelled?). Committee minutes. List of AVA stakeholders (marked to show HEIs and Access to HE providers), with additions and deletions since last year.
2 Strategic, management and operational matters	Strategic plan and targets for report year. Strategic plan and targets for the coming year. Staff structure (eg organisational diagram) and list (job titles; fractional and temporary posts; dates of new appointments). Externally verified accounts.
3 Quality assurance, enhancement and development	Samples of new quality assurance documentation. Evaluations of programme reports . List of the AVA’s development events and target audiences.
4 Academic standards	Evaluations of moderators’ reports. Moderators’ names, affiliation and length of service.
5 Commentary on statistical data programme data student data	Access to HE providers and programmes, with additions and deletions since last year. A print-out of 2005-06 statistics submission. Previous years’ statistics submissions.
Response to AVA review recommendations (where an AVA review has taken place since the last annual report was prepared)
Conclusions
	Special Appendix

Sections for inclusion in AVA annual reports

Evidence of approval

The final version of the report should be formally approved by the AVA’s governing body. This approval should be evidenced through a statement which a specifies the date of the meeting at which the report was approved, and b is signed by the Chair of the governing body. Please provide this evidence of the report’s approval on a separate sheet at the front of the report.

Contents page

Section headings with associated page numbers should be given. Appendices should also be clearly numbered and listed on the contents page.

Introduction

This section should provide an overview of the AVA's activities and achievements, noting major changes and developments in the environment for the AVA’s work.

Include, in particular:

contextual information to provide basic introduction for readers new to the AVA (eg the organisation’s size and structure; geographical reach and scale of its activities; relationship between Access to HE and any other areas of activity)
the AVA’s position at the beginning of the year, its priorities for Access to HE,
and the major challenges and opportunities it faced
internal and external factors influencing the organisation's progress in relation
to Access to HE during the report year
process for the preparation of the annual report (appendix optional).

1 Legal, constitutional and governance matters

This section should describe any significant changes in legal or constitutional status and assess the effectiveness of the AVA’s governance structures.

Include, in particular:

legal, constitutional or governance changes and reasons for them
changes to the involvement of stakeholders (including receiving HE institutions and Access to HE providers), including any known reasons for changes
committees’ activity (in terms of the range of business undertaken, frequency of meetings and patterns of attendance)
policy decisions or measures taken or planned by the AVA in relation to developing the scope and range of its activity
other key decisions taken affecting the AVA’s work and how they arose.

The text of section 1 should be cross-referenced to relevant appendices, including:

a list of stakeholders, marked to show HEIs and Access to HE providers, and indicating any additions and deletions since the previous year
a diagram of the AVA’s committee structure, showing reporting lines
a list of the AVA’s committees and indication of their composition in terms of representation and/or named individuals
a list of dates of committee meetings which took place during the report year, including any cancelled meetings
minutes of all meetings of key AVA committees, including the AVA’s governing body and the committee that has responsibility for finance and for quality assurance . (If, for reasons of confidentiality, it is necessary to submit edited minutes, the points at which any material has been omitted must be made clear.)

2 Strategic, management and operational matters

This section describes strategic and policy developments; it evaluates the success of the AVA’s plans in developing Access to HE; and assesses the impact and effectiveness of changes to the AVA’s management and operational functions.

Include, in particular:

strategic planning - summary of the AVA’s approach, including the process for approval of the AVA’s strategic plan and for monitoring targets. (A new strategic plan will not necessarily be devised on an annual basis, but there should be an indication of when, and by whom, the plan is reviewed and targets updated.)
The format of strategic plans varies, but all AVAs’ plans should include - or make reference to a separate document (for example, a ‘business plan’ or ‘action plan’) which includes - formally approved targets for the development of the AVA and its Access to HE work. The status of any plans, their relationship to each other, and the period to which they relate are clearly stated. Tabulated plans (included either in the main text or the report’s appendices) relate strategic objectives to specific targets which are measurable and time-limited and which indicate designated individuals/groups with responsibility for meeting the targets.
response to the report year’s targets - review of the AVA’s progress in meeting targets it set for the year, including comment on each target, with an indication of whether, and how, it was achieved. Where an action has not been achieved, the report should explain why it has not, and how the AVA intends to pursue it. Any unmet target should be shown as redundant or revised to respond to changes; or deferred and transferred to the coming year’s action plan with a new target date.
targets for the coming year - the new or revised plan for the coming year, with updated targets: targets for Access to HE should be related to the AVA’s broader strategic aims, and to areas identified elsewhere in the report (or in the previous year’s AVA report) as requiring action.
resources - assessment of the organisation’s financial position and changes to financial management; contribution and costs of AVA work to the organisation’s finances; accommodation and its adequacy for the AVA’s functions; staffing (including new posts with, or re-allocation of, Access to HE responsibilities)
communications – overview and developments; evaluation of success of communications and promotional work with Access to HE providers and others
self-assessment and risk-assessment – summary of outcomes of internal processes undertaken by the AVA during the report year.

The text section 2 should be cross-referenced to relevant appendices, including:

strategic plan and targets for the report year
strategic plan and targets for the coming year
the AVA’s detailed accounts for the most recent complete financial year. If the AVA has company status, it should submit its full audited accounts. If the AVA is an unincorporated association which has its accounts audited through a ‘host institution’, it should submit accounts extracted from those of its ‘host institution’ (where the AVA’s accounts are separately identifiable), together with the management accounts which are presented to the AVA’s governing body . In all instances, whatever the legal status of the organisation and its relationship with any other organisation, the accounts submitted must have been externally verified (and the means of verification made clear), and must be in sufficient detail to indicate robust internal and external financial monitoring. Where the AVA’s accounts for the report year have not been audited by the time of the submission of the report, draft accounts should also be submitted
staffing structure and staff list, with job titles, and indicating fractional/temporary posts, and dates of new appointments.

3 Quality assurance, enhancement and development

This section records developments in quality assurance processes relating to Access to HE; the reason for their introduction and what improvements they are intended to bring about in the quality of Access to HE programmes or the AVA’s assurance of their quality; and an assessment of their effectiveness to date, where possible.

The section also assesses the AVA’s success in establishing its position as a focus of development, through enhancing current Access to HE provision, initiating new developments, and engaging with developments initiated by others.

Include, in particular

changes to the AVA’s quality assurance processes for course development; validation; revalidation; and course monitoring, with explanation for, and evaluation of, any changes
conclusions drawn about the quality of Access to HE provision from its processes for course monitoring, including the outcome of programme (or centre) reports, highlighting positive developments; general trends and patterns; and particular areas of difficulty, and areas for further development
summary and evaluation of development and enhancement events organised during the year, noting any particular outcomes
description and evaluation of the AVA’s involvement in regional or national projects, as well as smaller-scale local initiatives, designed to improve or develop Access to HE; evidence of productive cooperation outside the AVA with other organisations involved in widening participation in higher education and assessment of progress made in these areas .

The text of section 3 should be cross-referenced to relevant appendices, including:

samples of new forms designed for quality assurance purposes (evidence of minor amendments, or forms used for other purposes would not be expected)
summary evaluations of programme (or centre) reports, including key findings
a list of development or enhancement events organised by the AVA (with dates) and their target audience.

4 Academic standards

This section reports on the outcome of the AVA’s procedures for monitoring academic standards on Access to HE programmes, and measures taken to assure their consistency and sufficiency. It also considers, and assesses the impact of, activities initiated by the AVA relating to assessment, standardisation, moderation and the award and issue of Access to HE certificates.

Include, in particular:

assessment of moderation process, particularly in relation to the impact of any recently introduced refinements or changes to the AVA’s procedures
outcomes of measures undertaken to monitor moderators’ performance
analytical response to moderators’/examiners’ reports. As well as recording successes and notable achievements, the response will note any apparent weaknesses or areas of concern, and the AVA’s intended actions in relation to these. AVAs are not expected to make judgements about named, individual programmes’ performance in their reports but the report mentions how the AVA has addressed unsatisfactory quality or standards it has identified
trends relating to the quality and standards of Access to HE provision and student achievement, as noted by moderators in their reports
any necessary explanation, especially where references are made to different categories of moderation or moderators which are not self-explanatory.

The text of section 4 should be cross-referenced to relevant appendices, including:

summary evaluations of moderators’/examiners’ reports, including key findings
moderators’ names, affiliation and length of service.

5 Commentary on statistical data

This section provides an analytical and in-depth commentary on the statistical data collected for the AVA’s statistical report to QAA [see Part 2 – Data submission – guidance to AVAs.], and provides the basis for a detailed discussion of how it relates to the characteristics of Access to HE provision during the report year. The commentary provides the outcome of a process of analysis and interrogation of the data, identifying any anomalies and attempting to explain their cause: it attempts to explain, rather than describe, the data.

Where the statistics are self-evident or unremarkable, commentary may not be necessary, but the report should comment on the most striking features of the data and their significance and consider, in general:

local circumstances which help to interpret or explain particular features of the data, where this is known (supporting anecdotal evidence can be illuminating: mere conjecture is largely unhelpful)
features which may not be evident or obvious from the raw data included in the report alone, particularly any factors that may have distorted the aggregated statistics presented
differences in, and ranges of, statistical data received from different providers and whether there are reasons to explain these. Although the report itself will probably not discuss the performance of individual providers by name, reference in general terms to considerations of differential performance may be needed for an external reader to have a proper understanding of aggregated or averaged statistics received from a number of different institutions and programmes.
reasons for large numbers of ‘unknowns’: if data is incomplete because certain providers have not submitted full data, this should be explained and the AVA’s actions to gain full data from all providers outlined
comparisons with previous years’ statistics, perceived trends and any known reasons for these
comparisons between the AVA’s own data and published national data, with any known reasons for significant variance from standard patterns. Comments may helpfully also include references to statistical data gathered for other purposes, relating, for example, to comparative regional data.

The report will include comment on, in particular:

programme data - changes to the provision for which the AVA is responsible, and whether this has changed the overall profile of Access to HE provision. Areas to note may include areas of expansion and contraction; changes to providers; types of programmes recognised; progress with validations/re-validations; programmes withdrawn; and gaps in provision. Information about recruitment to different types of programme (whether described by mode, location, explicit targeting or defined progression route) is also useful
learner data – comment on what is demonstrated by learner profiles about the extent to which provision is reaching its target groups. Learner achievement and progression should receive comment, particularly with reference to emerging trends. Although the statistical returns do not require information on reasons for student withdrawal, information gleaned from providers about trends or correlations with particular target groups may prove useful.

The text of section 5 should be cross-referenced to relevant appendices, including:

a list of QAA-recognised Access to HE programmes and providers, with an indication of additions and deletions since the previous year
a hard copy of the statistics submitted electronically to QAA
previous years’ statistics for at least the last two years, sufficient to demonstrate trends described in the commentary.

* Response to recommendations set at AVA review
Only those AVAs which have recently been reviewed will include comment in this section. Information is provided about the way in which the AVA has responded to each recommendation. If the AVA has decided not to take action in response to any of the particular recommendations, the reason for this decision will be clearly stated. If the AVA’s response to review recommendations has been included in the AVA’s action plan, reference to each particular recommendation will be explicit and easily identifiable.

6 Conclusions

This section provides an opportunity for the AVA to provide a brief statement on how the events of the year have affected the organisation as a whole, its position at the end of the year, and the major challenges and opportunities and key priorities which it faces in the coming year.

Appendices

Standard appendices

For a list of the required appendices, see the annual report checklist (page 4).

Information in appendices supports the commentary in each section, as indicated under each of the section descriptions given in the previous section. It is expected that most appendices, with the exception of the Special Appendix (see below), will be documents which have been produced prior to the preparation of the annual report for the AVA’s internal purposes.

In instances where an appendix is not available in electronic format, the contents page should state that the appendix is available in hard copy only.

2006 special appendix

AVAs should provide an outline of their plans for the implementation of the Access to HE Diploma and credit specifications. It should include the AVA’s overall strategy and approach to the implementation of the new specifications as well as articulating clearly specified actions and detailed timings. AVAs should build upon the update submitted to QAA in June 2006 to illustrate those aspects of its plan that have been completed, those that are ongoing and to highlight particular challenges that need to be overcome.

It is recognised that the special appendix will, on this occasion, require AVAs to provide information about activity which takes place outside the report year.