The End of the RAE; and Onwards to the REF …

Stuart Croft and Andrew Russell

Around the country, colleagues have had time to digest the results of the 2008 RAE; a time of celebration for some, of satisfaction for others, and of frustration and disappointment for still others. This particular version of the RAE has been far more intensive than any previous ones; over 99% of the work submitted was read by two panel members, and that which was not, was read by one and a member of another sub-panel. As a rough average, this would mean nearly 600 pieces – books, articles, reports – read by each member of the Politics and International Studies sub-panel. For full details, do see http://www.rae.ac.uk/panels/main/j/politics/ As a result of this intensive process, it has been far easier to construct league tables. Four such league tables seem to predominate:

• the so called National Profile, or weighted GPA, which is the overall result for all those submitting by a unit of assessment;
• a league table of the 3* [‘Quality that is internationally excellent in terms of originality, significance and rigour but which nonetheless falls short of the highest standards of excellence’] and 4* pieces [‘Quality that is world-leading in terms of originality, significance and rigour’] by percentage
• a league table of 4* returns by percentage
• and a ‘research power’ measure, which counts the actual number of pieces that are in the 3* and 4* category adjusted by the number of staff submitted

Institutions, and departments, are inevitably choosing the league table that portrays them in the most favourable light.

There has been one issue that has dominated debate beyond that of departmental positions, and even the (shortly to be announced) finance that will come with that (the ‘QR’ funding): the relative position of Politics and International Studies in league tables of all disciplines. In the Times Higher table, based on model one above, the discipline ranks 62nd out of 67 submissions. This issue in particular was raised by the Chairs of PSA and BISA with the sub-panel, many of whom came to address a Heads of Department meeting in London on February 12th. It has become a key issue of debate for departments in their university settings. For example, a middle ranking in Politics and International Studies is 2.15; in sociology, 2.40; in business and management studies, 2.35; in social work, 2.50. 

In discussion, the sub-panel shared some very important insights as to how this had come about. As those charged with RAE responsibility for PSA and BISA, we thought that four implications could be drawn. First, this RAE judged each individual output; and it seems probable that there had been poor management of the ‘fourth’ piece submitted. Second, as a discipline, we had submitted a significant number of scholars to the process, with an increase from 1076 in 2001 to 1269 in 2008; perhaps greater selectivity was called for, as had apparently been the case with other disciplines. Third, the GPA or National Profile is a very poor indicator in any case, and it is one that could move significantly if any of the high volume departments had achieved only a marginally different result. It was never designed to provide a comparative tool across disciplines; the purpose had been to standardise procedures. Fourth, submissions were made by a variety of higher education institutions with different missions; perhaps a higher variability than in many other disciplinary returns.

The sub-panel was almost entirely constructed from nominees from both PSA and BISA, and the current Chairs of both Associations have thanked the sub-panel for its hard work, and for the commitment in particular of the chair, Tony Payne, to be as accessible and as transparent as the rules allow him to be.

There will clearly be ongoing processes of restructuring and reorientation in many universities as a result of the RAE rankings, and of the financial outcome that is to follow; and both PSA and BISA will together seek to monitor this very closely. However, in addition, it is exceptionally important that we begin to focus firmly on the next process: as we leave the RAE behind, we enter the brave new world of the Research Excellence Framework (see http://www.hefce.ac.uk/Research/ref/) Much is still to be defined – but the clock is running: all work since 1 January 2008 will be assessed in this new framework. Several key issues have emerged so far:

• Despite the determination to go to a metric system, politics and international studies will not be assessed using citation indexes alone; whether they form part of a basket of assessment measures, however, remains to be seen;
• However, there is a strong commitment to a ‘streamlined’ process: HEFCE have already recruited an Expert Advisory Panel to offer views on the structure of the REF based on a lighter touch system, but the exact nature of this component of the REF is still to be determined. It is not clear if that will mean greater input from a social sciences panel rather than disciplinary ones; or the submission of fewer than four publications per person (outside early career scholars); or the use of proxies to assess quality, thus reviewing the strength of the reliance on peer review;
• Esteem is likely to be dropped as an indicator, and replaced by ‘impact’. This may also be standardised to a higher level in overall judgements (from 5% for esteem this time); and may be assessed in part, or even primarily, by research users rather than scholars.

The REF will probably come to dominate many conversations in the way that the RAE has done; and together, PSA and BISA will seek to influence the debate as strongly as possible.