6th February 2017.
Raffaella Rumiati, Scuola Internazionale Superiore di Studi Avanzati (SISSA), Trieste
and Agenzia Nazionale di Valutazione del Sistema Universitario e della Ricerca (ANVUR)
Venue and time: College lecture theatre, 6.00 pm. The poster of the seminar can be downloaded here. The poster of the series can be downloaded here. The seminar will be streamed at: http://smarturl.it/CollegioVoltaStream
Introduction
Over the last two and half centuries research has transformed from the endeavour of a few and wealthy gentlemen to a resource intensive and institutionalised enterprise, largely sustained by tax payer money. This seminar series contrasts different procedures of research assessment in use in several European countries in the search for new or improved methodology. Starting from the mid 1980s, however, several countries have began questioning the return from investment in research and devised procedures for research assessment in the name of accountability. Thirty years on there is broad agreement that research funded by the public ought to be assessed, but little consensus about the methodology to be employed. This seminar series contrasts different procedures of research assessment in use in several European countries in the search for new or improved methodology.
Abstract
(by RI Rumiati and M Malgarini)
The ‘Agenzia Nazionale per la Valutazione del sistema Universitario e della Ricerca’ (ANVUR) is responsible for evaluating universities and research institutes in Italy. ANVUR bears a wide range of responsibilities including the evaluation of research and quality assurance, of the impact and of the learning outcomes. Of these roles, the evaluation exercise is probably the one that has received more attention for it is linked to incentives that the Italian Ministry of University and Education (MIUR) grants to the universities in the yearly budget. Historically, these evaluation exercises have been based on peer review methods. More recently, there has been a growing tendency to adopt bibliometric indicators, at least for STEM areas. To date there have been two research evaluation exercises (Valutazione della Qualità della Ricerca or VQR in short), one related to the period 2004-10, the other that has just closed now concerning the period 2011-2014. The 2004-10 VQR has analyzed almost 185,000 articles, books, patents, and other scientific outcomes submitted for evaluation by Italian universities and other public research bodies. Fourteen areas of research were considered for each of which ANVUR selected a group of evaluation experts (Gruppo di Esperti Valutatori, GEV), for a total of 450 experts. The methodology adopted for such evaluation was based on a combination of peer review and bibliometric methods, based on indicators concerning the number of citations received by individual papers and the impact factor of the journal in which they are published.
More specifically, bibliometric indicators have been applied to mathematics, natural sciences, engineering and life sciences and, albeit with a slightly different approach, in economics and statistics; for humanities and social sciences evaluation has been accomplished by relying only upon peer evaluation. Bibliometric evaluation was based on a combined use of article-level and journal-level metrics; wherever the two metrics provided converging results (i.e., when both indicators assigned an article to the same class of merit), bibliometric evaluation will be intended as conclusive, before being finally approved by the GEV. However, when the two metrics ended up in different classes of merit, the GEV assigned a specific weight to the two indicators in order to decide the final class of merit to be assigned to the article. For instance, in the areas characterized by slow accumulation of citations, a predominant weight was assigned to journal-level metrics, while the opposite was true for areas characterized by more frequent citation practices. Similarly, evaluation of more recent articles, for which the citations count may still be considered as inconclusive, relies more heavily on journal-level metrics, while the opposite is true for articles published at the beginning of the evaluation period. If article-level and journal level metrics were considered to be “too” far apart (usually when they differ by more than two classes of merit), the GEV peer reviewed the article, either internally or assigning it to an external reviewers. External reviewers were selected on the basis of an external list of experts in the field; the list was carefully crafted on the basis of an ex-ante scientific accreditation and taking into consideration possible conflicts of interest. Peer evaluation was based on a pre-determined evaluation form, containing specific questions based on the evaluation criteria described above; the reviewer also provided a final brief comment explaining the choice made concerning the class of merit assigned to the article.
Full professors, associate professors and assistant professors, affiliated to a research universities and institutes, took part in this exercise by submitting three research outcomes, including books, books chapters, articles and other products (among which are patents, databases, works of art, architectonic projects and others). Departments and Universities are therefore evaluated within homogeneous scientific areas on the basis of the number of research outcomes submitted and the corresponding scores. Only results at the level of departments and universities are made public while individual researcher’s output is restituted privately.
The 2011-14 VQR was performed adopting by and large the same methodology as in the first exercise. However some novelties were inserted. First, 16 GEV were nominated each for each research area, for a total of 400. Second, Italian faculty submitted two research outcomes, whereas each researcher from research center submitted three outcomes. In order to be considered for evaluation they had to possess an ORCID identifier. Finally, the bibliometric algorithm used in STEM areas has been enhanced in order to better calibrate the distribution of research outcomes into the evaluation classes defined in the Ministerial decree. Overall, over 118,000 research outcomes have been evaluated. Evaluation has been based on criteria such as originality, methodological rigor and scientific (actual or potential) impact. Each GEV defined specific evaluation criteria for each Area. The analysis of each research outcome resulted in a synthetic appraisement of its scientific merit, across five levels: excellent, high, fair, acceptable, and limited, with score being been assigned to each level of merit (respectively equal to 1; 0,7; 0,4; 0,2; 0).
Evaluation methods. Evaluation is based on the method of informed peer review: wherever possible, peer evaluation has been supported by bibliometric information stemming from the two main databases available worldwide (ISI-Web of Science and Scopus); specifically, information about the scientific impact of both the articles (as expressed by the number of citations) and the journal in which they are published in (as expressed by the Impact Factor © and other similar indicators) have been used in the evaluation. Bibliometric indicators were duly normalized, in the widely agreed assumption that citational practices differ significantly across type of publication, scientific field and year of publication. No automatic evaluation has taken place, with the final assessment concerning the scientific merit of a publication always falling under the responsibility of the GEV. At any rate, more than 50% of publications has been evaluated purely with peer review methods.
For the 2011-14 VQR, results of the evaluation of research products accounted for 75% of the global evaluation of Universities and Departments; another 20% of the total score will be based on the quality of publications of researchers that has been hired or promoted during the period 2011-14: in this way, evaluation also takes into account the ability that the Italian Universities showed in selecting researchers with the higher scientific impact and quality. A final 5% of the final score reflects the competitive fundings granted to Universities and their higher education activities (doctoral programs). On an experimental basis, ANVUR has also been elaborating a system of indicators supporting the peer review of impact activities, related to scientific transfer and building of social capital; results of the experimentation will be presented in the final VQR report, however will not translate into monetary incentive as yet. In conclusion the final goal of the research exercise consists in evaluating the quality of the research carried out in the Italian Universities, and to rank them and their Departments in each of the 16 areas of evaluation. In times of growing political pressure to control public finances, it becomes increasingly important to ensure that expenditures are cost-effective and capable of reaching their objectives.
Biographical sketch
Raffaella I Rumiati obtained an MSc in Philosophy with Psychology curriculum in 1990, and a Doctorate in Psychology in 1995 (both from the University of Bologna). The research towards Doctorate was carried out at the School of Psychology of Birmingham University UK for 4 years, focusing on the interface between perception and action, a field in which she has been working until now. After completing the Doctorate she moved to the International School of Advanced Studies (SISSA) in Trieste where from a postdoctoral position she moved up to a full professorship in cognitive neuroscience in 2011. To date she has about 130 publications mostly peer-reviewed articles, but also reviews, commentaries and book chapters. She was the recipient of the “von Humboldt Foundation” Bessel Prize in 2003, and of the Women in Cognitive Science Mentorship Award in 2006. She is an editor for Brain and Cognition and Cortex, and a member of the board of Cognitive Neuropsychology and acts as a reviewer for many journals as well as funding agencies. About gave about 80 invited talks in Italy and abroad and spent some working periods abroad (e.g., U.K., Japan, Germany). During 20 year in SISSA she had several organizational roles including head of the PhD in Cognitive Neuroscience, head of the SISSA’s Ethics Committee, member of SISSA’s Board of Governors, and member of the Academic Senate. From November 2016 she joined ANVUR, of which she is vice-president.
Image: A drawing of eminent Italian scientist Alessandro Volta, after whom this College is named and the battery he invented.
