Quality can come in many sizes and shapes

The most valuable peer review is based on clear criteria and guidelines, allowing a good dialogue between the researcher, the evaluator and the representatives of the scientific committee of the publication channel, and ultimately leading to the improvement of the research to be published (...). Any form of open initiative contributing to the generalisation of such a system is valuable to me.

Ioana Galleron is a Senior lecturer in French language and literature at Université de Bretagne-Sud. She is involved in research evaluation projects, such as EvalHum. Since April 2016, she is the Chair of the COST Action CA15137.

Michał Starczewski: What is the difference between the evaluation systems in STEM and SSH disciplines? How are the European Union and its member states facing this issue? Is it possible to find common criteria for all academic  disciplines? Should we try?

Ioana Galleron: While systems of evaluation differ with regards to their principles (ex-post/ex-ante, performance based/size based), there are not separate systems evaluating STEM and SSH disciplines. This is not a problem in itself, except when it comes to some of the methods applied, since both the SSH and the STEM disciplines do not form homogenous groups, and there are enormous differences, with regards to the publications habits, between biology and mathematics, for instance, as well as between the SS and H, and within these broad areas. There is strong evidence that quantitative metrics such as impact factor, H-index and alike do not reflect quality or even allow to correctly observe productivity in the SSH, and therefore should not be used for evaluating these disciplines, all the more so as the data is incomplete and not representative. However, such methods are still in use, partly because national evaluation systems are often influenced by STEM scientists and inspired by what goes for these disciplines, partly because of a belief that competition to publish in international journals will improve research, partly because it produces easy to use numbers, and partly because the alternatives appear unclear or not very reliable. Indeed, SSH research evaluation has to improve its protocols, which are sometimes perceived by many SSH communities as being unfair, ill-adapted and non-transparent; also, opposition to the very principle of evaluation is to be found more often in the SSH communities, for ideological reasons, or because of traditions of collegiality. From an epistemological point of view, it was even suggested that STEM and SSH disciplines have very different positions with regards to evaluation, since in STEM there is a long tradition of progress related to the discussion and the refutation of the findings by other researchers, while such a paradigm is less frequent in the SSH. This tends to suggest that while the two fields may converge towards certain common criteria, there are intrinsic differences to be taken into account when evaluating the ones and the others.

In short, SSH evaluation is a complicated issue, both for the evaluators and for the evaluated. And while some European states have acknowledged the situation and look for appropriate solutions, in many other places this is not the case. A European initiative for the SSH is therefore much needed, so as to create a space for discussion between those who experiment more adapted and fair procedures for the SSH, and to give the issue the visibility it deserves.

You are the Chair of the COST Action European Network for Research Evaluation in the Social Sciences and the Humanities (ENRESSH). Could you describe the aims of this project?

The “European Network for Research Evaluation in the Social Sciences and the Humanities” is a COST Action starting in April 2016 and ending in April 2020, aimed at proposing clear best practices in the field of SSH research evaluation. The Action brings together various experts such as researchers in evaluation studies, policy makers and members of evaluation units, as well as researchers from SSH disciplines. The project compares strands of work dedicated to SSH research evaluation, currently under development in different parts of Europe, in order to avoid unnecessary duplication and to upscale results; it started with exchanges of experience in order to build a picture of practices across Europe and the state of the art in research on evaluation. During the subsequent years, the project team will organise conferences, workshops and meetings to bridge the gap between scholars in SSH research evaluation, research managers and policy makers.
I also seize this occasion to mention the EvalHum Initiative. This is a broad based association that seeks to bring together all those working in the field of research evaluation and impact in the social sciences and humanities. If this sounds pretty similar to ENRESSH, it is not surprising to find that it was EvalHum members who proposed the action. However, the two are strictly separate, and the association will continue when the COST Action will be over. It is also EvalHum that runs the successful RESSH conference series, with the next taking place in Antwerp in July 2017.

Sometimes the concepts used to describe scholarly communication mean very different things in different countries. This is the case of ‘monograph’, ‘paper’, ‘review’ etc. In some countries a monograph is legally defined by the number of pages, while in others by the number of authors. How much confusion does this cause on the international level?  Do you think that some controversies would be easier to solve if people were more aware of this ambiguity?

Too often, evaluation procedures focus upon formal attributes of publications, without being aware that they are very rough proxies for quality, that labels (‘(peer) reviewed’, ‘university press’, ‘book’) can cover very different realities, and that numbers (of pages, of signs, of contributors…) are not relevant in themselves. There is no reason to look down on proceedings of a conference as being in principle “less good” than a single-authored book, and no reason to push towards publishing papers rather than books – or vice-versa! – since quality can come in many sizes and shapes. Such shortcuts or trends would make sense only if there was a form of consensus, within the SSH communities, as about the most adapted types of publication with regards to certain types of scientific results, and if one could have a fair amount of confidence in the thoroughness of the quality checking realised by a scientific committee (be it of a journal, a conference, a publishing house, a reading committee, etc.). Unfortunately, when one looks at the reality behind the words, especially from a pan-European point of view, one realises these two conditions are far from being met.
So, I don’t know if this will help solving controversies, but being aware about what kind of processes are involved in the publication of a scientific book in, for example, Riga as compared to what happens with a book in Venice will probably help to progress collectively towards common standards, and to promote better science rather than to keep imposing (publication) choices to the scholars.

Does open science make evaluation more effective? How?

Open science will change research practices in the SSH, and can impact evaluation methods and protocols. But there are also several questions arising, such as are SSH scholars aware of the open access debate and its implications, and do SSH scholars have the financial means to publish in open access? Article Processing Charges (APC) for articles are a problem for what is often unfunded research and book publishing is prohibitive. However, to return to the question, the very fact that certain scholars choose to publish openly their research and, moreover, their research data, should be taken into account in an evaluation grid. Open science is also better disseminated, therefore easier to reuse, and to refute, two other important aspects which can be evaluated in turn.
Moreover, open science allows researchers, in certain countries, to shortcut or to go beyond a publishing system which is, for the reasons above mentioned, quite often dysfunctional in the SSH.

How could data sharing be evaluated? Which aspect is the most important: immediate access, legal licenses or full metadata?

Data sharing is a way to make the research process more transparent and make the job for reviewers easier. In the mean time, it can spare unnecessary duplication, since studies can be replicated or re-analysed from another perspective. It is not easy, however, to conceive a protocol for evaluating data sharing, and we will probably need some time so as to see how scholars engage with the issue.
From an evaluation point of view, metadata are crucial; they are also, maybe, easier to share and to protect. In order to build robust evaluation protocols, we need to know what happens in the field, and to date this knowledge is very incomplete concerning the SSH, at national as well as at a European level. Also, the situation is very contrasted from a discipline to another, with psychology and economy being much better covered by the international databases, while research in languages and literatures are lagging largely behind. Having access to the metadata – not only of the data but also of various publications – would allow us to progress towards a European database of SSH production, so as to observe on a larger and more robust scale publication habits, collaborative practices, and even thematic trends.

Communications technology is changing very fast. Scholarly communication is changing as well. Do you think that research evaluation should keep up or should it be a little more conservative and evolve at its own pace?

Research evaluation has to be aware of research processes and the scientific habits in the communities it evaluates, so yes, for credibility’s sake it has to keep pace with the changes in scholarly communication. There is little room, in the current evaluation protocols, for scientific blogs, contributions to newsletters, and in most countries for many forms of involvement with the society at large, to give but some examples. However, all these attempts to communicate otherwise, within and without academia, create room for innovation. In the SSH, there is a huge potential unspotted by decision makers, in many cases because they are scrutinising indicators and results which are not actually reflecting what really happens in the field.
On the other hand, let’s also bear in mind that indicators used in research evaluation need time to be tested and proven to be working, and that running after each trend doesn’t make for a very robust evaluation protocol.

What is the role of social media in scholarly communication? Should academic institutions invest in tools counting tweets or posts on Facebook, such as Altmetric?

Social media have done a lot of good to the SSH, since they allowed to communicate with the large public about our topics and researches, something which did not often happen with the traditional media, more focused on discoveries from the ‘hard’ sciences. Also, as mentioned above, ‘traditional’ metrics do not work very well for the SSH. A SSH scholar with a very low H-index is not necessarily an unproductive scholar, and the number of downloads of his or her papers in repositories can demonstrate this. However, one must be aware that altmetrics would not reflect either what exactly happens in the field, because of the conservatisms in many communities. Excellent researchers not communicating through Twitter, posts on Facebook or other academic social media can still be widely read in libraries. Also, data from many Web 2.0 platforms are not reliable enough, and, as all data, they can be shaped by the powerful sciences (physics and medicine) so as to count what they are good at (stories of discovery and curing cancer) rather than our SSH stories in which people are often more interested. In short, institutions need to be aware about scholarly communication through social media, but also careful in interpreting the information they get, and cautious not to impose behaviours where these are not natural.

You might say that there is a chasm between science and society. 'Ordinary people' don’t know what science is about and they can’t participate in it. At the same time scientists often are seen as closed in an ivory tower, ignoring the real problems of today’s world. However, “citizen science” projects aim to build bridges and connect both sides. Do you think that science is becoming more egalitarian?

I am always surprised when I hear about a “chasm” between science and society, and even more surprised of people focusing on our (supposed) ivory tower, rather than, say, on bankers in their golden one… Be that as it may, is this a new phenomenon, and is this something to worry about? After all, ‘ordinary people’ living in the 16th century were even less aware about science, as the great majority was unable to read, and scientists appeared to many as dangerous. Science is better perceived and even understood in our modern, educated societies – and maybe the feeling that scientists don’t open enough to society is related to this greater capability of the ‘lay public’ to understand science, which makes people, in turn, more interested to participate in it. Science is not rejecting people, but it asks, and it is in its very nature to do that, for knowledge and skills so as to participate in it. There are also a lot of great stories of societal engagement of scholars from all disciplines, and especially from the SSH, showing that we are far from being so above the maddening crowd as the traditional saying pretends.
From a sociological point of view, what we see is a steady and continuous change of the community of researchers, whose origins and backgrounds are more and more diverse; so, in this sense also science seems to become more ‘egalitarian’.

There are lots of ideas how to improve peer review and make it more transparent. What kinds of open peer review seem to be the most valuable for you?

In spite of known limitations, traditional peer review has been proved to be useful to generate better science, provided it is done properly; what is called so by certain SSH publishing houses or journals is sometimes a very light process, resulting in one line (or even one word!) appreciations of pieces of research. This is not to deny the very good job performed by numerous journals or publishing houses but, in many other cases, who gets published depends less on the quality of the work, than on the personal relationships some researchers maintain with certain editors, or, even worse, on the capability to pay to be published. I tend therefore to say that the most valuable peer review is, above one, the real one (as opposed to what I call “mocking” or “pretence” peer-review), based on clear criteria and guidelines, allowing a good dialogue between the researcher, the evaluator and the representatives of the scientific committee of the publication channel, and ultimately leading to the improvement of the research to be published. The recent proposals by the open access biology journal eLife opens interesting perspectives in allowing a pre-publication dialogue between the anonymous reviewers and the researcher, as well as extended forms of peer review where stakeholders participate to evaluate research with a societal mission. More generally, any form of open initiative contributing to the generalisation of such a system is valuable to me.

Szczegóły

Michał Starczewski