Amsterdam, 4 October 2016
Manual for research assessments
of the Academy Institutes
1 INTRODUCTION
A manual for using SEP 2015-2021
Research assessments are a key component of the quality assurance system of the Royal Netherlands Academy of Arts and Sciences institutes organisation. In the period 2015-2021, the Academy will evaluate all research of its institutes according to the Standard Evaluation Protocol for Public Research
Organisations 2015-2021 (SEP 2015-2021, version 2016).
This manual explains the procedures involved in an assessment according to SEP 2015-2021. We will begin by describing the most important changes in this SEP, compared to the previous one.
Academy institutes also carry out midterm assessments, halfway between two external assessments. The final chapter of this manual describes the procedure for these ‘light’, internal assessments.
2 CHANGES IN SEP 2015-2021
Below the eight most important changes in SEP 2015-2021 (compared to SEP 2009-2015) are described.
1 Strategy
The strategic targets/goals of the institute are leading in the assessment process. These targets should be kept in mind when composing the assessment committee, writing the Terms of Reference, writing the programme for the site visit, and writing the self-assessment report.
2 Self-assessment report
In the self-assessment report the institute describes its strategic targets, both scientific and societal, of the past years, the efforts it has made to realise these targets, and the results of these efforts. In this way the committee assesses whether the institute has done the right things and whether it has done these things right.
3 Assessment criteria
The number of assessment criteria is reduced from four to three: ‘scientific quality’, ‘relevance to society’, and ‘viability’ (SEP 2015-2021, p. 7).1The former criterion ‘productivity’ is not a separate criterion anymore, but is now part of the criterion ‘scientific quality’. This can be seen as a response to the criticism that in certain disciplines the publication culture puts too much stress on the number of publications, instead of their quality.
The three criteria are applied as follows:
• Scientific quality: the assessment committee assesses the scientific quality of the institute’s research by relating the scientific output to the institute’s strategic targets. Each institute should specify which indicators are relevant in judging whether it has reached its strategic targets on scientific quality. • Relevance to society: the societal relevance of the institute’s work is assessed by relating the societal
output to the institute’s strategic targets. Each institute should specify which indicators are relevant in judging whether it has reached its targets on relevance to society.
• Viability: the institute describes its strategic targets (both scientific and societal) for the next five to ten years. The committee assesses whether the institute makes the right choices and is well-equipped to realise the targets it has formulated.
As such, the criteria ‘relevance to society’ and ‘viability’ are assessed more systematically in this SEP, compared to the previous SEP.
4 Matrix of output indicators (Table D1)
SEP contains a matrix for indicators of scientific quality and relevance to society. As said, institutes should specify their strategic targets in both areas, as well as the relevant indicators. See for more information on how to use Table D1 section 3.1.2 below.
Especially for relevance to society institutes should include at least one narrative in their self-assessment report (see Appendix D2 of SEP and section 3.1.2 below).
5 Judgments: text and categories
The assessment committee presents it judgments regarding the three main criteria (scientific quality,
relevance to society, and viability) in writing. These qualitative judgments are supplemented by a discrete
1 SEP 2015-2021 calls this criterion ‘research quality’. We prefer the broader term ‘scientific quality’, which may refer to both the research tasks of the Academy institutes, and their collection or other tasks.
category for each criterion: 1 = excellent, 2 = very good, 3 = good, 4 = unsatisfactory. The core of the judgment, however, remains the written text.
Please note: the current categories differ from the scores used in the previous SEP. Also, there are no
intermediate categories (for example no 3,5). Evaluators have to choose.
6 PhD programmes
The assessment committee also evaluates the PhD programmes associated with the institute. See section 2.4 in SEP, p. 7-8, and Appendix E, p. 31. The institute has to report on this subject in its self-assessment report, see Appendix D, p. 23.
7 Research integrity
The assessment committee also evaluates the institute’s policy on research integrity and the way in which violation of research integrity is prevented. See section 2.4 in SEP, p. 9, and Appendix E, p. 31. The institute has to report on this subject in its self-assessment report, see Appendix D, p. 23.
8 Diversity
In the 2016 version of SEP 2015-2021 diversity is added to the aspects that are evaluated qualitatively. See section 2.4 in SEP, p. 9, and Appendix E, p. 31. The institute has to report on this subject in its self-assessment report, see Appendix D, p. 23.
9 Aggregation level of the assessment
The previous SEP asked for judgments on two levels of aggregation: 1 the level of the research groups and
2 the level of the institute. SEP 2015-2021 only asks for judgments on the higher level: that of the institute as a whole (i.e. of the whole organisational unit that is assessed).
This is related to the focus of the new SEP which is on strategy: the assessment committee should assess the results of the unit in relation to the strategy it has formulated for the past years, and should give recommendations for the strategy formulated for the years to come.
Please note: if the institute’s director wishes judgments per group, (s)he can ask the assessment
3 EXTERNAL ASSESSMENT
3.1 Terms
The most important terms associated with the assessment procedure are described below, followed by a schedule for the evaluation of an individual institute.
3.1.1 Assessment committee
The assessment committee generally has five members. The chairperson is Dutch, in general three members are from abroad; two of them are European.
It is evident that the committee must be qualified to assess the scientific quality, the relevance to society (‘the applicability of the research unit’s research and its relevance to society’, SEP, p. 11), and the viability of the institute. In order to be able to assess the relevance to society, it is important to:
1 include in the assessment committee someone with specific knowledge and experience to assess this aspect;
2 include in the programme for the site visit a session with representatives of the institute’s societal stakeholders.
The committee members should not have had a scientific, business or personal relationship with the institute or any of its staff in the past years that could give rise to a conflict of interest. Before starting the assessment procedure, the committee members sign a statement of impartiality and confidentiality.
The assessment committee is assisted by an independent secretary.
The members and chairperson are appointed and discharged by the Academy Board on recommendation of the director general after consulting the institute’s director. The director general will also consult an Academy member or other renowned researcher and the scientific committee about the potential candidates for the assessment committee.
The members of the assessment committee receive a fee of € 1,000 (the chairperson receives € 2.000) for participating in an assessment and are reimbursed for their travel and accommodation expenses. More information on practical matters such as travelling, accommodation, and reimbursements can be found in the document Evaluations of KNAW institutes: Practicalities.
3.1.2 Self-assessment
In order to prepare for the assessment committee’s site visit, the institute draws up a self-assessment report. This document is sent to the members of the assessment committee and to the director general at least four weeks prior to the site visit.
In the self-assessment report, the institute describes its performances during the past six years, related to its mission and strategic targets. Chapter 5 and Appendix D of the SEP 2015-2021 describe the information that must be covered in the self-assessment. It entails both the scientific and societal strategy, targets, and results of the past six years. The institute also describes its strategic targets (both scientific and societal) for the years to come.
Other topics that need to be addressed are PhD programmes and research integrity.
The self-assessment report should be no longer than 15 pages (excluding appendices and tables). Please note that SEP 2015-2021 assesses the institute level, and not the individual group level (see also the final section in Chapter 2 of this manual).
Instructions for appendices D1-D4, to be included in the self-assessment report
SEP 2015-2021 contains several appendices that are to be included in the self-assessment report. Below we give an instruction of how to use these appendices.
Instructions for appendix D1 and D2
Appendix D1 is the key table of the SEP 2015-2021. It contains a matrix with six cells in which examples of output indicators are given.2 The idea is that disciplines or research fields for each cell agree on a number of indicators best representing the activities in their field.
Academy institutes must indicate in their self-assessment report the indicators they have chosen for each of the six cells in the output table. The choice of indicators of an individual Academy institute should be in line with a) indicators generally used in the institute’s discipline or research field, b) the specific mission and strategy of the institute, and c) the agreements the institute has made with the director general.
SEP 2015-2021 states that institutes ‘must complete all cells [of the table with output indicators] unless certain cells are not relevant’ for that institute (p. 13). In that case, the institute must in its self-assessment explain why this is the case.
For each indicator, the institute provides evidence pertaining to the past six years. Most of the quantitative evidence is delivered by the Pure research information system. This evidence must be provided with an explanation of the results, answering questions such as: what do the numbers show; how do they relate to the mission and the strategic targets of the institute; which deviations deserve an explanation?
Per institute, qualitative evidence must be delivered via one or two narratives (also called ‘case studies’). A narrative is a text in which societal relevance is reported by describing a specific case. See the
instruction in Appendix D2 of the SEP 2015-2021. SEP has taken this from the Research Excellence Framework (REF) in the United Kingdom; for examples of narratives across disciplines see
http://www.ref.ac.uk/about/background/pilot/.
Instructions for appendix D3 and D4
The institute must complete the tables in Appendix D3 of the SEP 2015-2021, except for
Table D3b, as the categories in this table are already included in the table with output indicators the institute draws up (Table D1).
Appendix D4, the SWOT analysis, must also be included in the self-assessment.
3.1.3 Site visit
The assessment committee visits the institute for two consecutive days. This visit is referred to as the site visit. The institute proposes a programme for the visit, which is subject to the approval of the chairperson of the assessment committee and the director general. It is important that the site visit also includes a meeting of the assessment committee with a representation of the societal stakeholders of the institute. Other guidelines for the programme of the site visit are described in section 6.2 of the SEP 2015-2021.
On the evening before the site visit, the committee members have a dinner together. During this private kick-off meeting the committee members discuss the assessment procedure, the programme for the site visit, and their findings on the documentation provided by the institute.
3.1.4 Assessment report
After the site visit, the committee draws up a report of its findings (assessment report) in compliance with the SEP-guidelines (chapter 7 and Appendix E). Within four to six weeks after the site visit the draft report is submitted to the institute for a factual check. The factual inaccuracies, if any, are corrected by the
assessment committee, after which the final report is submitted to the director general, the institute’s management, and the institute’s scientific committee.
Comments of institute’s management and scientific committee
The director general asks the institute’s management and the scientific committee (and, where relevant, also the societal advisory council) to comment on the assessment committee’s report.
Position of the Academy Board
Based on the assessment report and the comments of the institute’ management and the scientific
committee (and, where relevant, also those of the societal advisory council), the Academy Board draws up its position paper. The draft position paper will be drawn up by the director general who will consult an Academy member or other senior scientist. This draft paper is discussed, in outline, with the institute’s director and the chair of the scientific committee. The final position paper is submitted to the members of the assessment committee, who are then discharged from their task.
Publication of the assessment file
After the assessment procedure has been completed, the assessment file is published on the Academy’s website. This file consists of these documents:
• the self-assessment report;
• the assessment committee's report;
• the comments of the institute’s management; • the comments of the scientific committee
• where relevant, the comments of the societal advisory council; • the Academy's position paper.
In its annual report the Academy indicates which institutes have been assessed, as well as the most important conclusions and recommendations, and the follow-up actions that have been formulated.
3.1.5 Follow-up
The follow-up of the assessment is monitored in the half-yearly meetings between the Academy Board and the director general with the institute’s management. Each year the institute’s management delivers a text in which it formulates the progress that has been made on the various aspects in the past year, as well as its specific objectives for the year to come. Delays or changes in the objectives are indicated and motivated in the text.
Three years after the external assessment has taken place, a midterm assessment is carried out (see Chapter 4 of this manual).
3.2 Plan of action for an individual external assessment
Below the plan of action for an external assessment is given. For each of the actions performed in the procedure of an external assessment, this plan indicates by whom it is performed, and when.
A. BEFORE THE SITE VISIT Six months before the site visit
Announcement to conduct an assessment
• The director general informs the institute’s director of its intention to conduct an external assessment and draws up a general schedule in consultation with the institute’s director.
• The institute’s director reports the Academy’s intention to conduct an assessment to the scientific committee and its staff (including the institute works council).
• The director general asks the institute’s director to decide which documents will be part of the self-assessment and to make the necessary preparations (the self-self-assessment report must be ready one month prior to the site visit).
• Where applicable, the institute’s director requests a bibliometric analysis by a qualified external institute. The costs of this analysis are shared by the institute and the Academy. However, the Academy reimburses up to a maximum of € 3.000.
Composition of assessment committee
• The director general asks the institute’s director to propose a longlist of potential chairpersons and members of the international assessment committee.
• The director general will consult an Academy member or other renowned researcher and the scientific committee of the institute about the proposal before sending it to the Academy Board. • The Academy Board discusses the proposed composition of the assessment committee.
• The institute’s director indicates whether (s)he agrees with the Board that the committee as proposed by the Board will be capable of adequately assessing the quality of the institute’s work.
• The Academy Board decides on the composition of the committee and so informs the institute’s director.
• The director general appoints the chairperson and members on behalf of the Academy Board. • The director general asks the committee members to sign a statement of impartiality and
confidentiality (SEP, Appendix C).
• An independent secretary will be selected and appointed by the director general.
Terms of reference
• The director general draws up the draft Terms of Reference (assessment instructions) for the assessment committee. The draft version is discussed with the institute’s director.
• In the Terms of Reference an appropriate benchmark is selected for the institute (in consultation with the institute’s director). Other guidelines for the Terms of Reference are given in the SEP (section 4.2 and Appendix B).
• The Academy Board decides on the Terms of Reference and so informs the institute’s director.
Three months before the site visit
Composing the site visit programme
• The institute’s director proposes a programme for the site visit.
• The director general discusses the proposed programme and the finalised Terms of Reference with the chair and the secretary of the assessment committee. They may suggest changes for the programme.
• The director general assesses the final programme.
Organising travel and accommodation for committee members
• The Academy secretariat books the trips and hotel rooms in consultation with the committee members.
• The institute secretariat arranges meeting rooms and catering at the institute.
• Where necessary, the Academy secretariat arranges meeting rooms and catering at the Trippenhuis building.
• The Academy secretariat reserves the restaurant tables (for the first evening and the second evening, and, possibly, also the third evening) and arranges transport to the restaurants.
One month before the site visit
Sending out the self-assessment report
• The institute’s director sends the self-assessment report and appendices to the general director, the members of the assessment committee, and the official secretary (at least four weeks before the site
visit).
Logistical information
• The Academy secretariat sends to the committee members the latest information about their travel arrangements, reimbursement of expenses, the hotel, the restaurants, the institute, the Academy, and so forth.
B. AFTER THE SITE VISIT
Within three months after the site visit
Assessment report
• The assessment committee sends a draft assessment report to the director general within four to six weeks after the site visit. The director general sends the draft report to the institute’s director. • The institute’s director checks the draft report for factual inaccuracies and reports these, if any, to the
director general. The director general reports these to the assessment committee.
• The assessment committee corrects the inaccuracies and sends the final report to the director general.
• The director general sends the final report to the institute’s director and the scientific committee (and, where relevant, also to the societal advisory council), asking them for their written comments. • The institute’s director and the scientific committee (and, where relevant, also the societal advisory
council) send their comments to the director general.
• The draft position paper will be drawn up by the director general who will consult an Academy member or other renowned researcher. The director general discusses it in outline with the institute’s director and the chair of the scientific committee.
• The Academy Board assesses the position paper.
Assessment procedure
• The Academy Board sends the position paper to the institute’s director, to the scientific committee (and, where relevant, also to the societal advisory council).
• The institute’s director informs the institute’s staff (including the institute works council) of the outcome of the assessment.
• The Academy Board sends the Board’s position paper to the assessment committee and discharges its members.
• The Academy Board publishes the assessment file (including the self-assessment report, the assessment report, the comments by the institute’s director and by the scientific committee (and, where relevant, also those by the societal advisory council), and the Board’s position paper) on the Academy’s website.
• The Academy Board reports on the assessment and the follow-up actions in its annual report.
Payment of expense claims
• The Academy secretariat ensures that the committee members’ fees and expense claims are paid.
3.3 Schedule of assessments of Academy institutes 2015-2021
The Appendix gives a schedule of the external and midterm assessments of the Academy institutes for the years 2015-2021.
4 MIDTERM ASSESSMENT
4.1 Introduction
A midterm assessment is conducted halfway between two external assessments (i.e. three years after an external assessment). This is obligatory for all Academy institutes. This assessment should be a ‘light’ and internal procedure, as is described below.
4.2 Self-assessment letter
The institute’s management is asked to write a self-assessment letter to the general director which contains at least the following information:
1 A review of the recommendations of the previous external assessment, and of the way the institute has dealt with these recommendations in the past three years.
2 A review of relevant internal and external developments in the past three years. 3 A SWOT analysis of the institute (see Appendix D4 of the SEP).
Unlike the SWOT analysis for the external assessment, the SWOT analysis in the midterm assessment is intended for internal use. This means that the analysis can be more in-depth, and that it is less formal in nature, allowing the institute to push the limits of the analysis in order to engage in an open-hearted discussion of the institute’s future.
4 A preview of new objectives and aims for the next few years, with a view to the next external assessment.
The letter is submitted to the director general, after which the director general asks the institute’s scientific committee to comment.
4.3 Discussion
The midterm assessment is discussed during one of the half-yearly meetings of the Academy Board and the director general with the institute’s management. The scientific committee (or a member thereof) is also invited to take part in this discussion.
APPENDIX 1 - SCHEDULE EXTERNAL AND MIDTERM ASSESSMENTS KNAW INSTITUTES
Institute Previous external assessment 2015 2016 2017 20183 2019 2020 2021
CBS 2014 midterm external
DANS 2011 external midterm
Huygens ING4 2008 midterm external midterm
Fryske Akademy 2009 external midterm external
Hubrecht Institute 2014 midterm external
IISG 2011 midterm external midterm
KITLV 2011 midterm external midterm
Meertens Institute 2011 midterm external midterm
NIAS5 2008 external midterm
NIDI 2013 midterm6 external
Netherlands Institute for Neuroscience 2012 midterm external midterm
NIOD 2011 midterm external midterm
NIOO 2012 midterm external midterm
Spinoza Centre Founded in 2015 midterm external
3 The planned external assessments and midterm review in 2018 must be fully completed by mid-2018
4 The assessment of Huygens ING is postponed due to the merger of Huygens Institute-KNAW with ING-NWO in 2010-2011 and changes of directors. 5 The assessment of NIAS is postponed due to changes of rectors and major decisions on the organisation and location of the institute.
