J e r z y B r z e z i ń s k i
Adam Mickiewicz University in Poznań, Poland
CONTROVERSIES AROUND THE EVALUATION OF
RESEARCH UNITS IN THE FIELD OF SOCIAL SCIENCES
Th e parametric evaluation of research units carried out last year (approximately 950 academic units, Polish Academy of Sciences research centres and research and development units participated in it on a voluntary basis) evoked heated debate in scientifi c circles. I took part in it myself, as a member of the Council for Science. Holding the position of the chairman of the Humanities and Social Sciences Work-ing Team (ZR-1), together with the other members of the team I was responsible for this evaluation in about 1/3 of the units (the ones that were assigned to the category of the humanities, social sciences and artistic studies). Together with prof. Antoni Sułek (a sociologist from Warsaw University), we dealt with those units which were assigned to a homogenous group “N-4: Social Sciences1”.
In this paper I would like to share my experience from this parameterization of research units as well as from my previous work connected with the evaluation of such institutions, which I did as a member of the State Committee for Scien-tifi c Research for two terms. I wold also like to propose some new solutions. In order to make my argumentation clearer, I am going to divide my comments/ proposals into six thematic groups:
Problem 1. What is (is not) a research unit and how should its minimum size be established (the problem of so-called “small Ns”)?
1 Prof. A. Sułek wrote about these works in “Forum Akademickie” [Academic Forum]. See:
idem, Parametryzacja nauk społecznych 2010 [Th e Parameterization of Social Sciences 2010], “Forum Akademickie” [Academic Forum] 2011, No. 3, p. 38-40.
Problem 2. How should a homogenous group of research units be defi ned? Problem 3. Whose results of research work (converted into points) are counted as the ones of a research unit (they are located in the denominator of index E) and who should be located in the denominator of index E?
Problem 4. Should the number of a research results presented by a unit be re-duced?
Problem 5. What eff ects of research work of a unit’s employees should be taken into consideration (and awarded points proportionally to their weight)?
Problem 6. Should one publication be taken into account several times if its authors work in diff erent units?
* * *
PROBLEM 1. What is (is not) a research unit and how should its minimum size be established (the problem of so-called “small Ns”)?
In order to carry out a truly accurate evaluation of the scientifi c level of a given research unit, fi rst we need to defi ne the term “research unit”. In accordance with binding legislation it is: (a) a research centre of the Polish Academy of Sciences, (b) a research and development unit of a specifi c ministry – e.g. the Institute for Western Aff airs in Poznań founded by the Ministry of Foreign Aff airs, (c) an aca-demic unit: faculty, institute, department, division – it is stipulated in university regulations (however, it is usually a faculty – e.g. the Faculty of Philosophy of the Jagiellonian University of Cracow).
As far as the units from groups (a) and (b) are concerned, the evaluation does not cause any problems because they seem to be quite homogenous in terms of a research profi le – they could be identifi ed as, for example, sociological, economic, psychological or historical ones.
Some diffi culties arise when we deal with academic units from group (c). When universities established faculties, they did not always do it by combining a few units of lower level: such as institutes, departments or divisions which have a similar research profi le. Th us, there are universities which have a relatively big number of basic units (such as the University of Warsaw) and the ones which have defi nitely too few of such units. Consequently, a faculty may be homogenous in a strictly scientifi c sense (e.g. the Faculty of Psychology of the University of Warsaw or the Faculties of Political Science and Journalism of the University of Warsaw and of the Adam Mickiewicz University in Poznań) or it may be strongly heterogeneous,
composed of a few institutes or institutes and departments of a diverse research profi le (such as the Faculty of Social Sciences of the Adam Mickiewicz University consists of four institutes: the Institute of Philosophy, the Institute of Cultural Studies, the Institute of Psychology and the Institute of Sociology). Apart from the organizational formula, what link is there between diff erent units which form the Faculty of History and Pedagogy of Opole University: the Institutes of History, Pedagogical Sciences, Psychology, Educational Studies, Political Science, Philoso-phy, Sociology and Arts? If it was possible to categorize them separately, these institutes would be assigned to three or four homogeneous groups.
When the parameterization was carried out in 2010, legal regulations did not allow dividing such internally diff erent basic units of universities (faculties) into their de facto homogeneous, smaller units – institutes. As a result, one homogene-ous group (see p. 2) was composed of units which were too strongly diff erentiated. I will provide a simple, as it seems, example. Th e Institute of Philosophy and So-ciology of the Polish Academy of Sciences has two research profi les: the philo-sophical one [by this virtue it could be assigned to the homogenous group N-3 (cultural studies, philosophy and theological studies)] and the sociological one [by this virtue it could be assigned to the homogeneous group N-4(social sciences)]. It was oft en the case, however, that a given institute, which was a part of a large university faculty, could be assigned to a few heterogeneous groups! Th ere were units of a historical-social profi le or of a historical-linguistic-social one. Th ere are also other – not less “exotic”– combinations.
Th e act of assigning a basic unit to a specifi c group, according to the defi nition of a homogeneous research profi le, was not always substantiable. It must also be added that it was not legally possible! It obviously posed further diffi culties in comparing (and ranking substantively) such units. In other words, were units of a similar research profi le compared (which was the basic methodological assump-tion of the whole procedure)? It was mostly true but a certain group of units did not fi t in the proposed framework. What could I suggest? I recommend returning to a solution from the times of the Committee for Scientifi c Research, when at the request of a unit director and a university vice-chancellor it was possible to sepa-rately evaluate lower level units which composed faculties (e.g. institutes).
If we want to defi ne a research unit accurately, we need to establish its mini-mum size. Th e experience of the fi rst parameterizations conducted by the Com-mittee for Scientifi c Research shows that the parametric evaluation of research units was carried out on the basis of the so-called eff ectiveness index (parameter E) calculated as the ratio of the sum of points gathered in the period under ap-praisal (the years 2005-2009 were evaluated) by “employees conducting scientifi c
research” (the nominator of parameter E) to the average number of “employees conducting scientifi c research” for whom a given unit was the primary place of employment.
If we took all units into account, even those which consist of just a few people (yes, even such units were indicated in the requests!), their indexes would be very high. Consequently, the needle eff ect would occur. Aft er all, it is possible to imag-ine a team composed of a professor and two assistants (N = 3), who would publish a total of 9 (as 3N=9) highly scoring works (monographs and articles from the ISI Master Journal list or even the European Reference Index for the Humanities – ERIH). In order to prevent this phenomenon, which distorts the global result, it was decided (by means of a simulation) that a unit undergoing evaluation cannot employ fewer than 21 people.
Conclusions:
1. Using abundant materials provided by last year’s parameterization we should consider diff erent variants of division into homogeneous groups. Now “our” N-4 is not homogeneous at all.
2. On the basis of a new simulation, we should establish the size of a small research unit. It should not be too small. In my view, having analysed the results of last year’s parameterization of research units, the lower limit of the size of a research unit should be increased to N = 30.
PROBLEM 2. How should a homogenous group of research units be defined?
Apart from defi ning the smallest possible scientifi c unit (non-divisible into smaller ones), the issue of dividing all scientifi c units (not taking into consideration their location – at universities, the Polish Academy of Sciences, ministries) into homoge-neous ones (on account of their similar research profi le) is the second problem that needs to be solved in order to establish the accurate parameterization procedure.
In accordance with legislation in force the following division pattern of research units is binding:
Level I. Th ree basic groups of sciences and arts2:
A. Th e humanities and social sciences (legal sciences, humanistic sciences, theological studies, economics and military science);
2 In accordance with attachments no. 2-4 to the directive of the Minister of Science and Higher
B. Exact and technical sciences, life sciences (chemical sciences, physical sci-ences, mathematical scisci-ences, technical scisci-ences, biological scisci-ences, phar-maceutical sciences, forest sciences, medical sciences, physical culture sci-ences, Earth scisci-ences, health scisci-ences, agricultural sciences and veterinary sciences.
C. Disciplines of arts [fi lm arts, musical arts (conducting, instrumental studies, composition and theory of music, audio engineering, eurhythmics, and vocal studies), plastic arts (fi ne art, design, conservation and restoration of works of art), and theatre studies].
In this paper we shall deal exclusively with group A. Level II. Scientifi cally homogeneous groups:
N-1 – language studies and bibliology, N-2 – historical studies,
N-3 – cultural studies, philosophy and theological studies, N-4 – social sciences,
N-5 – economic sciences, N-6 – legal sciences,
Nm – small groups of the humanities and social sciences – units with N ≤ 20, N-14 – arts – units with N > 20,
N – 14m – arts – units with N ≤ 20.
As I indicated at the beginning of this paper, we will be concerned only with the sciences which compose the homogeneous group N-4. Th is group included units which represented a “pure” research profi le of such disciplines as: pedagogy, political science, psychology, sociology, or – in the case of faculties – those consist-ing of institutes, departments and divisions of a “social” profi le (on the basis of being “arbitrarily” assigned to N-4 or on the basis of their own choice, later ac-cepted by the minister, when they did not agree with the decision of the Commit-tee for Scientifi c Research to assign them to a specifi c homogeneous group).
Is it possible to construct homogeneous groups which would enable conducting an unencumbered ranking evaluation of all units assigned to a given homogeneous group (here: N-4)? It is obviously not. Th e evaluation such as the one carried out in 2010 was not free of weaknesses. We should remember that only 30% of the units (in accordance with the above-mentioned directive) from each homogeneous group could be assigned to the most desirable and prestigious category A (cat. 1). By forming small homogeneous groups (there will be more of them instead), we increase their degree of comparability, but at the same time we reduce the number of units which could be assigned to category 1. Th e units which found themselves
“below the line” and were assigned to category 2(B) could be included in category 1 if a small group they belonged to comprised more units.
Let me illustrate this with an example. Our group N-4 consisted of as many as 61 units, 30% of which (18 units) were assigned to category 1 (A). Th e value range of index E was: 65.22 – 31.29 (the total value range of index E was: 65.22 – 7.04). Th e next two units had the following values of index E: 30.23 and 30.03. If the group was larger, they would also fi t in category 1. I will provide two examples. Let us consider a relatively small group N-2 (historical studies), which consisted of 16 units.
It included 6 units of category 1 (which is more than 30%) with the value range of index E: 54.12 – 41.28, with the total value range of index E: 54.12 – 18.32. Th e next unit had the value of index E = 39.4. If this unit was included in a group with the number of units equal to the number of units in group N-4, it would be classi-fi ed in category 1 together with four other units 9 (E > 36). I wanted to show that the high value of index E absolutely guaranteed being given an equally high (cor-responding to category 1) position in a given homogeneous group – irrespective of its size. In turn, a unit which was placed in the medium value range of index E and was included in a small group, could be located in the lower value range of index E, category 1, in a group consisting of many units. Th us, why were the groups not equinumerous? Firstly, because some disciplines are more popular and more oft en taught at universities and others are less numerous. Secondly, the scientifi c level (measured in the same way!) in the whole group of the humanities and social sci-ences was clearly diff erentiated. Th irdly, the complexity of university faculties (there were only a few truly homogeneous ones) had a distorting eff ect.
Conclusions:
1. Given the percentage classifi cation of units to specifi c homogeneous groups, the size of groups infl uenced the lower sub-range of index E values. In small groups, such a unit would have to “beat” a higher threshold value of E. 2. I believe that at universities only genuinely homogeneous units, i.e., institutes
rather than complex, “combined” faculties, should be evaluated. We should remember that, apart from only a few exceptions, the Polish Academy of Sciences consisted of de facto homogeneous units. Th e same was the case with research and development units.
3. I would also consider a diff erent method of selecting the best units (categories 1 and 2), which might not be as comprehensible and easily interpretable as the percentage one, but which would take the actual complexity of a unit into account.
PROBLEM 3. Whose results of research work (converted into points) are counted as the ones of a research unit (they are located in the denominator of index E) and who should be located in the denominator of index E?
Another serious problem, which was not analysed thoroughly enough in the pa-rameterization of research units, concerns the question included in the title of this section: who contributes desired points, which will be entered in the nominator of index E, to a research unit? Th e answer to this question is not easy at all.
Th e directive mentioned in footnote 1 says that it may only be a person em-ployed in a unit for “conducting scientifi c research”. Such defi nition eliminates the following categories of people connected with a university: (a) doctoral students of a unit. It may be worth reminding that the Council for Scientifi c Research not only acknowledged PhD students whose registration and conferment procedure for a doctoral degree had been started, but it also gave them preferential weight: 0.3 (it was obviously meant to encourage units to provide doctoral studies), (b) faculty members (lecturers); (c) scientifi c-technical workers.
Only research workers and academics are taken into account (listed in art. 110, par. 1 of the Law on Higher Education revised this year). If we accept the fact that a unit may also present academic course books as the eff ects of its work, why can’t we accept it when the author is a senior lecturer, Mr Coursebook, PhD. Aft er all, it is quite natural that people employed as faculty members write valuable course books. As a matter of fact, they were awarded the same number of points as scien-tifi c monographs (12/24 points depending on whether they were written in English or in the language other than English).
What I absolutely cannot understand is the decision to eliminate the achieve-ments of a unit’s PhD students. It actually excluded them as the authors of inde-pendent publications in prestigious scientifi c journals. I suppose that in technical and natural sciences doctoral students are assigned a role of “cogs” in larger ma-chinery, which is controlled by a professor or even a doctor. In the end, such joint publication will be attributed to a given unit by virtue of the fact that a “real” re-search worker is employed there. Th is is not the case in the humanities and social sciences, though. Since a PhD student (at present, a new organization – the Na-tional Science Centre – opens an individual grant path even to people with Master’s degree, who are also allowed to supervise a research project) was allowed to par-ticipate in promoter or independent competitions organized by the Council for Science, and previously by the Council for Scientifi c Research, why can’t the pub-lications that are the eff ects of such projects be attributed to a particular unit? When a doctoral student did not want a given publication (oft en subsidized by
a unit) to be wasted, they formed a partnership with one of the unit’s employees. I would have nothing against such temporary teams if they worked on the basis of fair cooperation. Particular members of such venture, however, do no not always contribute the same amount of work. Sometimes a PhD student does almost all the work. I have also heard of cases (doctoral students complained about it) when they were refused to place independent articles in journals (highly scoring on the national list “B”) published by a unit as the unit’s employees were given precedence. It is the question of “not wasting” points. If PhD students (possibly with some preferential weight assigned to them) were acknowledged, the authorities of their unit would be motivated to take more thorough care of their scientifi c develop-ment.
Another problem which still remains to be solved concerned the issue of treat-ing works written by the employees of a unit for whom it is not the primary place of employment as part of the unit’s scientifi c achievements (points!). Th erefore, the unit could count the works of Mr Resourceful, PhD, as its own publications even if he published some of them as an employee of his home, state, university, and the rest as a worker of a private university, where he does not have to be employed on a full-time basis. Th ere have been cases when such employee was motivated with an additional bonus (which could be quite generous in rich non-state universities). Th e adoption of such solutions resulted in launching the market of points (publi-cations in journals from the ISI Master Journal list are especially highly valued).
Th e odd construction of index E evoked another controversy. Its nominator comprised all points scored by a unit’s employees (according to the key to the evaluation sheet, see attachment 2 to the above-quoted directive – in the humani-ties and social sciences mainly for publications). We may notice that points were also contributed by people for whom the unit was their additional place of employ-ment (second, third, …). What I cannot accept is the fact that the denominator included only people for whom the unit was the primary place of employment. It was easy to notice that the higher the value of the nominator and the lower the value of the denominator were, the higher the value of index E was. If a unit places points for the works of its employees who are not included in the denomina-tor, its scientifi c position is overrated. Th e unit would score a similarly overesti-mated result if it could remove the people who did not contribute any point from the denominator. However, it could not take such action. We cannot evaluate units in a reliable way (especially with regard to those units – mostly non-state universi-ties – which off er full-time jobs to people already employed by other institutions and demand that these employees should contribute a specifi ed number of points to them) if such action is legally accepted.
Conclusions:
1. Th e scientifi c achievements of a unit should also include works of PhD stu-dents (those who have already started their registration and conferment procedure for a doctoral degree) and doctoral students themselves should be included in the denominator of index E with preferential weight, e.g. 0.3-0.4.
2. Th e only correct method of calculating the value of index E should consist in counting only those employees in its nominator who have also been in-cluded in the denominator. Th erefore, if we stick to the defi nition of the denominator (it includes only those workers for whom a given unit is the primary place of employment), it is only them who can contribute points to the nominator of ratio E.
PROBLEM 4. Should the number of a research results presented by a unit be reduced?
In my view, it should be reduced and I believe that the adoption of this principle (it was not applied initially) is a step in the right direction. Why do I think so? Let us imagine that unit A has publications in highly scoring journals to its name but it does not have a lot of them – which is quite obvious (it is a lot more diffi cult to prepare a very good publication and place it in a prestigious journal with a global reach). In turn, unit B has plenty of publications (an authentic example from the times of the Council for Scientifi c Research!), most of which have been awarded 1 point (let us say there are 1000 of them). As a result, unit B is considered to be scientifi cally better in terms of points than unit A. It is obviously unfair and it distorts the scientifi c ranking.
Th us, the reduction in the number of publications that a research unit is allowed to present seems a good step, which would guarantee a reliable evaluation of the scientifi c condition of the unit. However, we should answer the question: How many publications should research units present? Th e answer is: a certain multiple of N. Th e value of the index cannot be the same for all research units, both those from the fi eld of exact sciences, and those from the area of humanities. Now we have a solution of “3N” for all of them. In my opinion, having carried out a simu-lation based on the data from last year’s parameterization, this index “xN” will take various values. What may be the implications of assuming a particular value of the index? When the scientifi c level of units in a homogeneous group is high and bal-anced, this index must take a higher value, e.g. “4N”, in order to force units to
present worse publications. In turn, when the level of publications is rather low (they are mostly domestic publications and chapters in joint works), the index value of “2N-3N” should be suffi cient. On the basis of the analysis of achievements in group N-4 which I conducted together with prof. A. Sułek, it should be enough if this index has the same value as last year, i.e. “3N”. Th e value of “2N” might even be suffi cient. Even among the publications of units which occupied the fi rst 10% of positions in group N-4 the works from the ISI Master Journal list represented a minor part, while in the range of 20-30% of units there were also low scoring chapters in joint publications.
Conclusion:
1. Index “xN” must be maintained in the parameterization procedure, but it should be diff erentiated (!) in terms of value.
PROBLEM 5. What effects of research work of a unit’s employees should be taken into consideration (and awarded points proportionally to their weight)?
Last year’s evaluation of research units was carried out according to the principles described in the above-quoted directive. Th e sum of points of a unit entered in the nominator of the index of its eff ectiveness E was calculated in accordance with the rules for our group of research units – specifi ed in the Research Unit Evaluation
Sheet for the Humanities and Social Sciences (attachment 2 to the directive of the
Minister of Science and Higher Education of May 25, 2010). We cannot discuss particular points of this tool here. Readers may become acquainted with the con-tent of the questionnaire on their own (e.g. in the public information bulletin published on the Ministry’s website). Let me only indicate its main points and comment them briefl y. Th e evaluation sheet consists of two parts: (a) the scien-tifi c one, which helps to calculate the value of the index of eff ectiveness E-I, and (b) the practical part, which makes it possible to calculate the value of the index of eff ectiveness E-II. Th e combined value of the global index of eff ectiveness E is the sum of two indexes: E-I and E-II. Let me note that the “practical” part of index E in group N-4 does not have a signifi cant infl uence on the position when it is ultimately assigned to upper categories 1 and 23.
3 It did matter, however, in technical sciences, where it aroused great controversies following the
I would not like to go deeply into the nuances connected with the number of points awarded for particular “scientifi c products”. It is not surprising that there is no agreement between humanist scholars and natural scientists as to “what” should be awarded and with “how many” points. Th ere is no reason for which the repre-sentatives of either of those two groups should persuade each other as to which scoring method takes priority. However, the very idea of parameterization of sci-entifi c journals and awarding points to them was conceived in the circles of exact, natural and technical sciences, and was later imposed on the humanities. Th e highly specifi c nature of sciences classifi ed in the group of the humanities and social science, which was only seemingly homogeneous, caused the need for divid-ing them into even smaller subgroups. Th erefore, we should aim at a consensus (“what” and “for how many points?”) within a specifi c group. Let me present an example. In language studies, it is the publications which appear in print in a given language that are important – and it is by no means English (apart from English language studies in which it plays a primary role). Th at is why the evalu-ation sheet in the humanities includes the concept of the “primary language” in the section devoted to books and chapters. For example, it is the Polish language for Polish studies. For psychology or sociology, in turn, it is English. Th erefore, if we were to compare two units, the Institute of Polish Language and the Institute of Psychology of the Polish Academy of Sciences, the latter would have to publish its entire works (monographs, academic course books and chapters in joint pub-lications) in English in order to match the former. It would obviously make no sense. Hence, it is important to distinguish a homogeneous group N-1: language studies and bibliology. Only units of a philological, linguistic and bibliological profi le would be comparable within this group. In a few cases a philological unit could obviously be a part of a faculty of a wider humanistic or social profi le. It would then be treated on the same terms as other units in this faculty (with no preference for the primary language, characteristic for given philology).
Having made these introductory remarks, let us now move on to answering the basic question mentioned in the title of this section. Th e following issues have been included in the scientifi c part of the evaluation sheet of a unit:
1. Reviewed publications; 2. Scientifi c monographs;
3. International research projects; 4. Entitlements to award degrees.
of index E equal (it assigned them the same weight). What evoked these controversies was a method of converting money (for works created for national economy) into points.
I believe that awarding points for participating in international research projects does not stir a lot of emotions. However, lack of precise defi nition of an “interna-tional research project” caused serious diffi culties both for people preparing re-ports of units for the Information Processing Centre (OPI – Ośrodek
Przetwarza-nia Informacji), and for people from ZR-1, who checked the accuracy of these
reports. Th e most frequent errors included taking into consideration such pro-grammes as cooperation projects between a Polish unit and a foreign one, domes-tic projects and uncompleted ones.
In the same way the scientifi c circles have accepted that points should be scored for units’ entitlements to award scientifi c titles – doctoral and postdoctoral degrees. I agree that these entitlements are highly scoring indeed. However, the fact that 50 points are awarded for doctoral entitlements and 150 points for postdoctoral ones does not jus-tify the separate scoring of each qualifi cation procedure conducted by a unit.
During the debate on all possible spheres of scientifi c activity that should be subject to scoring speeches delivered by a unit’s employees at international (espe-cially) and national conferences were oft en mentioned. Let me present my opinion on this issue (which is similar to the approach of the Council for Scientifi c Re-search and the Council for Science. If someone has made an important speech at a scientifi c conference, it should be published as an article in a good scientifi c journal – either international or domestic one. Th us, the same product would be awarded double points as it were. In turn, when a speech did not result in publish-ing a valuable article, it probably did not meet the highest scientifi c standards. In short, we award only those scientifi c “products” which have taken a form of pub-lication, the one fulfi ls the criteria of intersubjectivity, rather than those which are ephemeral (delivered and forgotten).
Conclusions: I am strongly in favour of the following views:
1. Let us evaluate only serious international scientifi c cooperation projects. 2. Th e parametric evaluation of research units concerns only their scientifi c
condition, whereas the level of educational services they provide should be subject to a specifi c parametric assessment, which would be exclusively fo-cused on teaching achievements. Th us, we should evaluate only those results which have been published in really good scientifi c journals – fi rst the in-ternational ones (the ISI Master Journal list and the ERIH list, which I am less enthusiastic about, though), and the domestic ones (the top part of the ministerial list “B”). Prof. A. Sułek also writes about the advantages of pub-lishing in international sociological journals (I would also add psychological ones) in this issue.
3. I do appreciate monographs, but only those which are genuinely scientifi c, and I value chapter in joint publications (at least those in group E-4!) much less. I would even go as far as to say that this type of publishing activity should not be awarded points at all. As a matter of fact, the relatively low scoring of chapters translates into a lower category (3-4) of those units which, apart from monographs, would only have chapters in joint works to their name. Th e issue of the minimum size of a monograph (now: six pub-lisher’s sheets: 240,000 characters, with spaces) still remains to be settled. I believe this size should be bigger, As far as academic course books are concerned, only those which are used nationwide (excluding local and school course books) should be taken into account. We should also seri-ously consider making a list of serious Polish publishers of scientifi c litera-ture (to put it briefl y, I would like to prevent situations in which a book is printed by a “garage” publishing house owned by its author’s relative. Th e size of academic course books should be also increased (about 15–20 pub-lisher’s sheets). In this case such list of acknowledged publishing houses would be necessary, too.
4. Finally, my most important conclusion is that we should aim at reducing the number of diff erent scientifi c “products”. I would personally recommend limiting their list to articles in scientifi c journals (but only highly scoring ones), scientifi c monographs and academic course books.
PROBLEM 6. Should one publication be taken into account several times if its authors work in different units?
My answer will be: no! Let me justify my opinion. In accordance with the principle that has been binding so far, if any publication has, say, three authors based in three diff erent units, each of them receives 100% of points. In turn, if the authors come from the same unit, each of them receives a proportional number of points, where-as their unit gets only 100%. Th us, in the former case each of the units receives 100% of points, while in the latter one only this one unit gets 100% of points. Th e reason for such treatment of joint publications was that it was meant to motivate researchers to establish international cooperation.
I am against this way of thinking. In my view, apart from genuine collaboration, some kind of quasi-cooperation occurs, in which specifi c “cooperatives” are formed. Although they provide points to the units the authors are based in, they do not actually contribute to their scientifi c achievements. If an article co-written
by three authors from diff erent research units was published in, for example,
Psy-chological Review or Nature (currently scoring 40 points on the ISI Master Journal
list), a summary overview of Polish publications in the world, based on the statis-tical list of publications collected from particular units, would include three articles instead of one. While two researchers from the same institution are awarded points for a given article or a monograph (which is assigned to the unit they are employed in) only once, if they work in diff erent units the same publication “multiplies” itself. Perhaps cooperation should be additionally rewarded, but it should not lead to “improving” reporting. I believe this issue needs further debate.
Conclusion:
1. I suggest that 100% of points should not be awarded to each co-author of one publication if they are employed in diff erent research units.
* * *
Th e procedure of evaluation of the scientifi c condition of research and research-educational units, which is regularly conducted every four years (only the last parameterization covered the period of fi ve years), has become an inseparable part of the system of evaluation procedures (apart from appraisals carried out by the State Accreditation Committee – PKA, Państwowa Komisja Akredytacyjna - and by the University Accreditation Committee – UKA, Uniwersytecka Komisja
Akredy-tacyjna) applied in Poland. Th ere is no way back. It is good that this procedure is of an objective and transparent character. Th e fact that it is not perfect yet cannot serve as the argument for abandoning it, which is unfortunately oft en abused in some circles (especially those connected with the humanities). I strongly believe that it is better to carry out the evaluation of the scientifi c level of research unit in an objective form, thus the parametric one, even if it is imperfect, than to do it on the basis of some unspecifi ed, subjective criteria. Th e fact that I am an advocate of parametric evaluation does not mean that I am unaware of its fl aws and things that remain to be done in order to improve it. As I have been involved in the process of establishing these procedures for a long time (three terms of offi ce of the Coun-cil for Scientifi c Research, and the last term of the CounCoun-cil for Science), I have a fi rm opinion on this issue. Th e problem is that my view of what I dislike may not be shared by other members of these bodies. Aft er all, there is a principle, in the highest instance, that the opinions shared by the majority of experts involved in the work are accepted (a consensus is welcome). However, it is diffi cult for me to accept the situation in which the Minister does something his or her own way against the experts’ opinion.
To sum up, I believe that the following steps must be taken as soon as possible. 1. To polish up the concept of homogeneous groups and to make it possible
to separately evaluate those research units which are smaller (e.g. institutes) than complex faculties.
2. To classify doctoral students’ works as part of a unit’s scientifi c achieve-ments.
3. To include PhD students with the weight of, say, 0.3, in N of a unit. 4. To establish the minimum N of a scientifi c unit to be appraised.
5. To reduce the number of publications presented by a unit to a certain (em-pirically checked) multiple of N – this principle does not need to be applied to all types of units (today: 3N).
6. To adopt the principle according to which points in the nominator of index E will be contributed only by those people who are counted in N of the denominator: authors cannot sell “publications’ to units other than their primary one.
7. To adopt the principle that points for publications co-authored by research-ers from diff erent units are subject to the same, proportional distribution, as in the case of all authors coming from the same institution.
8. To limit publications to monographs and course books and – fi rst of all – articles in indexed scientifi c journals.
9. To off er no points for the organization of conferences and for delivering speech-es and preparing materials by the employespeech-es and PhD students of a unit. 10. To abandon (in the humanities and social sciences) calculating partial
in-dex E-II (practical achievements): it turned out that it did not matter in “our” sciences.
11. We need to rethink the concepts of assigning categories to particular units on the basis of the value of index E. Th e present method, which is quite simple or even “mechanistic”, and consists in trimming away - according to the value of index E - consecutive fractions of units, starting from the top of an arranged sequence of units from the best to the worst ones (the top, “prestige” 30% of units are the ones in category 1) is too primitive and does not refl ect the complexity of the whole homogeneous group. To put it briefl y, it is about the fact that not each group should include, say, 30% of units considered to be the best, because it may turn out that the whole group is only at an average level. In turn, another group may contain a lot of scientifi cally very strong units and then the reduction of the number of units assigned to category 1 relegates them, quite undeservedly, to a worse category.
And one more thing. When this issue of Kultura i Edukacja appears, a new version of the principles of parameterization of research units (for the period of 2007–2010) may be made publicly available for so-called social consultation. Aft er all, a list of newly categorized, according to NEW PRINCIPLES, research units must be issued by 1 October 2012. Th e Committee for Evaluation of Research Units (KEJN – Komitet Ewaluacji Jednostek Naukowych) is working very inten-sively on the development of new principles. Will it surprise with anything? How will the new, mysterious category “A+” be established? We live in a strange country, in which a number of regulations are retroactive. In order to comply with the re-quirements of the new directive, we will have to award points “back” to all publica-tions of employees again. And how can we pursue a reasonable research policy in an institute or a faculty? How?
