Evaluation Manual
for the Academy institutes
2009-2014
Revised edition May 2012
1 Introduction ... 1 2 External evaluation ... 2 2.1 Terms ... 2 2.2 Procedure ... 3 3 Midterm evaluation ... 6 3.1 Terms ... 6 3.2 Procedure ... 6
Evaluation Manual for the Academy Institutes
Revised version – May 2012 1
1
Introduction
Evaluations are a key component of the Royal Academy of Arts and Sciences research or-ganisation’s quality assurance system. The Academy differentiates between two different categories of evaluation: external evaluations and midterm evaluations. This memoran-dum explains the process involved in both categories.
Before beginning, we briefly describe the two categories of evaluation below. A more de-tailed description can be found in sections 2 (external evaluation) and 3 (midterm evalua-tion).
External evaluation
The Academy institutes are evaluated by an external evaluation committee once every six years. The committee is appointed by the Academy. An external evaluation concerns the institute’s research task, but it may also consider other tasks (collections, services).
To prepare for the evaluation committee’s site visit, the institute draws up a self-evaluation, following the Standard Evaluation Protocol for Public Research Organisations 2009-2015 (SEP). Where possible and useful, a bibliometric analysis is carried out. The evaluation committee then visits the institute for two days . The committee runs through a programme during that period that has been agreed in advance by the institute, the evaluation committee and the Academy.
After the site visit, the committee draws up a report (evaluation report) of its findings that also complies with the SEP. The report is then submitted to the Academy, which asks the institute management and scientific committee (and in some cases the social advisory council) to comment on the report. Finally, the Academy draws up its own position paper. After the evaluation has been concluded, the complete evaluation file is published on the website of the Academy.
Midterm evaluation
A midterm evaluation is conducted midway between two external evaluations (in other words, three years after an external evaluation). The institute draws up a brief self-evaluation report describing internal developments since the most recent self-evaluation. It also reports on how it has dealt with the most recent evaluation recommendations. The report is submitted to the Academy, after which the institute’s scientific committee is asked to comment. The report and the scientific committee’s comments are then discussed with the institute.
2
External evaluation
2.1 Terms
The most important terms associated with the evaluation process are defined briefly be-low, followed by a detailed schedule.
Self-evaluation
In order to prepare for the evaluation committee’s site visit, the institute draws up a self-evaluation. The self-evaluation must be sent to the members of the evaluation committee and the Academy’s Board of Management no more than four weeks before the site visit. Chapter 5 of the Standard Evaluation Protocol 2009-2015 describes the information that must in any event be covered in the self-evaluation. This information is largely the same as the information provided in the factsheets (which are filled in every year by the insti-tutes).
Evaluation committee
The evaluation committee has five members at most. The members are prominent Dutch and foreign peers (the chairperson is Dutch, three peers are from abroad including one non-European member). The committee members may not have a business or personal relationship with the institute or any of its staff that could give rise to a conflict of interest. The members are appointed and discharged by the Academy. They are reimbursed for their travel and accommodation expenses (see next paragraph for details). They also re-ceive a fee of EUR 1,000 for participating in a site visit.
The evaluation committee has an official secretary, either an Academy employee or some-one hired externally by the Academy.
Travelling and accommodation
Flights can be book by either the Academy or the committee member him/herself. Flight costs for economy class travelling will be imbursed by the Academy. Costs for travelling by taxi or train (First Class) from and to the airport are also reimbursed. For Dutch members train costs (First Class) or the use one’s own car (€ 0,19 per kilometer) are reimbursed. Taxi’s from the hotel to the institute and vice versa will be booked and paid by the Acade-my. Dinners on the first and second evening will be also booked and paid by the AcadeAcade-my. Hotel accommodation for two nights, including breakfast, is booked by the Academy. All extra costs made in the hotel are to be paid by the committee member and will not be re-imbursed by the Academy.
Site visit
re-Evaluation Manual for the Academy Institutes
Revised version – May 2012 3
Evaluation report
After the site visit, the committee draws up a report (evaluation report) of its findings. The report complies with the SEP. Within a few weeks it is submitted for a factual check to the institute, which corrects any inaccuracies. The report is then submitted to the Academy,
Comments of scientific committee and institute management
The Academy asks the scientific committee and the institute management to comment on the substance of the evaluation committee’s report.
Academy position
Based on the evaluation report and the comments of the scientific committee and institute management, the Academy draws up its position paper and discusses it with the institute. The position paper is submitted to the members of the evaluation committee, who are then discharged from their task.
Publication of the evaluation file
The evaluation file consists of the self-evaluation, the evaluation committee's report, the comments of the institute management and scientific committee (and the social advisory council, where relevant) and the Academy's position paper. The evaluation file is pub-lished on the Academy’s website after the conclusion of the evaluation process. The evaluation committee’s report and the Academy position paper are also submitted to the Dutch Minister of Education, Culture and Science.
2.2 Procedure
I. SIX MONTHS BEFORE THE SITE VISIT
Intention to conduct evaluation The Academy’s Board of Management informs the institute management of its inten-tion to conduct an evaluainten-tion and draws up a general schedule in consultainten-tion with the management.
The institute management reports the Academy’s intention to conduct an evaluation to the scientific committee and its staff (including the institute works council).
The Academy's Board of Management asks the institute and others (scientific commit-tee, members of the society) to propose a potential chairperson and members of the international evaluation committee and a potential official secretary.
The Academy’s Board of Management asks the institute to decide which documents will form part of the self-evaluation according to the Standard Evaluation Protocol and to make the necessary preparations (the self-evaluation must be ready one month prior to the site visit).
Where applicable, the Academy’s Board of Management orders an external organisa-tion to carry out a bibliometric analysis.
Composition of evaluation committee
The Academy’s Board of Management approaches the proposed chairperson after con-sulting the institute management.
The Academy’s Board of Management consults the chairperson about the other com-mittee members and the secretary, approaches the candidates and consults the
insti-tute management concerning the final composition of the committee. The committee has between three and seven members.
The Academy’s Board of Management may issue specific instructions to the commit-tee; if so, it will inform the institute management accordingly.
The Academy Board finalises the committee and evaluation tasks and so informs the committee members and the institute.
II. THREE MONTHS BEFORE THE SITE VISIT
Finalising the site visit programme The institute management proposes a programme for the site visit.
The chairperson assesses the programme and suggests changes where necessary. The Academy's Board of Management assesses the final programme.
Organising travel and accommodation for committee members during a site visit The Academy secretariat books the trips and hotel rooms in consultation with the
committee members.
The institute secretariat arranges meeting rooms and catering at the institute.
Where necessary, the Academy secretariat arranges meeting rooms and catering at the Trippenhuis Building.
The Academy secretariat reserves the restaurant tables (on the first evening, for the committee and the Academy's director of Research; on the second and, possibly, third evening, only the committee) and arranges transport to the restaurants.
III. ONE MONTH BEFORE THE SITE VISIT
Sending out the self-evaluation The institute management sends the self-evaluation to the members of the evaluation committee, the official secretary and the Academy’s Board of Management (no more than four weeks before the site visit).
Logistical information
The Academy secretariat e-mails (or mails) the committee members the latest infor-mation about their travel arrangements, reimbursement of expenses, the hotel, the in-stitute, the Academy, and so forth.
IV. SITE VISIT
Conducting the site visit
The evaluation committee conducts the site visit according to the final programme (duration: approximately two days). On the evening prior to the site visit, the
commit-Evaluation Manual for the Academy Institutes
Revised version – May 2012 5
Evaluation report
The evaluation committee forwards a draft evaluation report to the institute. The institute management checks the draft report for factual errors and reports any
inaccuracies to the committee.
The evaluation committee sends the final report to the Academy’s Board of Manage-ment.
The Academy’s Board of Management sends the final report to the institute and the scientific committee for comment and, where relevant, to the social advisory council. The institute management and the scientific committee send their comments to the
Academy’s Board of Management.
The Academy’s Board of Management drafts the Board position paper and discusses it in outline with the institute management.
The Academy Board adopts the final position paper. Completing the evaluation process
The Academy Board sends the position paper to the institute and the scientific com-mittee.
The institute management informs the institute’s staff (including the institute works council) of the outcome of the evaluation.
The Academy Board sends the Board’s position paper to the evaluation committee and discharges its members.
The Academy Board sends the evaluation report and Academy position paper to the Dutch Minister of Education, Culture and Science.
The Academy Board sends the evaluation report and Academy position paper to the works council.
3
Midterm evaluation
3.1 Terms
Self-evaluation
A self-evaluation that is part of a midterm evaluation is briefer than one that is part of an external evaluation. A midterm evaluation consists of three parts:
A review of the recommendations of the previous external evaluation, focusing on two aspects: 1) the institute’s direction in terms of its research; 2) the administrative agreements made then.
A SWOT analysis of the institute (see chapter 5 of the SEP). Unlike the peer evaluation, the SWOT analysis is intended for internal use. This means that the analysis can be less in-depth, and that it is less formal in nature, allowing the institute to push the limits of the analysis in order to engage in an open-hearted discussion of the institute’s future. A preview of new objectives and aims for the medium term, in part with a view to the
next external evaluation.
Discussion with Academy’s Board of Management
The self-evaluation report is normally discussed prior to or immediately after the spring or autumn consultations. The scientific committee (or a member thereof) is also invited to take part in this discussion. If the scientific committee is unable to attend, it is asked to submit its comments in writing. Minutes are kept of the discussion (including the scientific committee’s comments). The self-evaluation and the minutes together make up the mid-term evaluation file. This file is usually not published.
3.2 Procedure
Preparing for a midterm evaluation – six months prior to the evaluation
The Academy’s Board of Management informs the institute six months in advance that a midterm evaluation has been planned and determines when the self-evaluation will be discussed.
The institute sends the self-evaluation report to the Academy’s Board of Management at least six weeks prior to the scheduled evaluation.
The Academy’s Board of Management sends the report to the scientific committee for information purposes, asking it to attend the discussion of the report. If the scientific committee is unable to attend, it is asked to comment on the report in writing.
Evaluation Manual for the Academy Institutes
Revised version – May 2012 7
Appendix:
Schedule of external evaluations and midterm evaluations,
2009-2014
The table below shows the schedule up to and including 2014. The years thereafter follow the SEP schedule, with minor adjustments. The result is an uneven distribution of evalua-tions over time. The Academy will adhere to this schedule for the time being, but the aim is to arrive at a more even distribution of evaluations in a few years’ time.
Instituut Externe evaluaties 1993-2008 2009 2010 2011 2012 2013 2014
CBS 2002, 2008 midterm extern
DANS n.v.t. extern midterm
Huygens 1997, 2008 midterm extern
FA 1995, 2002 extern midterm
Hubrecht 2002, 2008 midterm extern
ICIN 1994, 2006 midterm extern
IISG 1995, 2003 extern midterm
KITLV 1999, 2006 midterm extern midterm
Meertens 2000, 2007 midterm extern
NIAS 2000, 2008 midterm
NIDI 2008 midterm extern
NIN midterm extern
NIOD 2003 extern midterm
