Excellence in Research for Australia 2015
Associate Law Deans National Meeting
The University of Western Australia
November 2015
Sarah Howard
Director, Research Evaluation
Australian Research Council
Introduction
•
•
•
•
•
ERA 2015—where are we up to?
ERA 2015 National Report
ERA 2015 methodology
Peer review in ERA
ERA 2015 outcomes and data
ERA 2015—where are we up to?
• Currently finalising evaluation
• Preparing State of Australian
University Research: ERA
National Report 2015-16
ERA 2015 National Report
WHAT?
•
State of Australian University Research: Volume 1
ERA National Report 2015–16
 Includes additional analysis of 10 year
longitudinal dataset (2003–2013)
 ERA 2010, ERA 2012, ERA 2015 comparisons
•
PLUS—additional volumes to come… applied
research, gender, open access
WHEN?
• Early December….
ERA 2015 methodology
ERA Documentation
•
Range of information about ERA development and
ERA 2015 processes
•
ERA 2015 documentation includes:




•
Background information includes:


•
Submission Guidelines
Discipline Matrix
Notifications and FAQs
2015 REC members
National Reports
ERA development documentation
www.arc.gov.au > ERA
Useful Terminology to start …
•
•
•
•
•
•
System to Evaluate Excellence of Research
(SEER)—ERA IT system
Research Evaluation Committee (REC)—
committee of experts undertaking evaluations
ANZSRC—Australia and New Zealand Standard
Research Classification
Field of Research (FoR)—ANZSRC research
classification (e.g. 2101 Archaeology)
Unit of Evaluation (UoE)—ERA unit of
assessment for an institution by FoR (e.g. ANU
2101)
Dashboard—suite of ERA indicators
ERA 2015
The ERA objectives are unchanged:
1. Establish an evaluation framework that gives
government, industry, business and the wider
community assurance of the excellence of research
conducted in Australian higher education institutions.
2. Provide a national stock-take of discipline level areas of
research strength and areas where there is opportunity
for development in Australian higher education
institutions.
3. Identify excellence across the full spectrum of research
performance.
4. Identify emerging research areas and opportunities for
further development and.
5. Allow for comparisons of research in Australia,
nationally and internationally, for all discipline areas.
ERA 2015 Reference Periods
Data type
Reference period
Research outputs
1 Jan 2008–31 Dec 2013
6
Research income
1 Jan 2011–31 Dec 2013
3
Applied measures
1 Jan 2011–31 Dec 2013
3
Esteem measures
1 Jan 2011–31 Dec 2013
3
Staff census date: 31 March 2014
Years
ERA 2015 Discipline Matrix
Discipline Matrix ERA
2015
Peer Review (30% of outputs to be nominated by output type)
Non-traditional outputs by
type
FoRCode
FoR Title
Books
Book
Chapters
Journal
articles
Research
Report for
Conference
External
Publications Other NTRO Body
Esteem Measures
HERDC
Research
Income
(Categories
1-4)
Editor
Prestigious
Works of
Reference
Membership
of learned
academy
Category 1
research
fellowships
18
Law and Legal Studies
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
1801
Law
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
1802
Maori Law
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
1899
Other Law and Legal
Studies
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Evaluation Overview
Citation Analysis or Peer Review
Volume and Activity
Research Income
Applied Measures
Esteem
Access to Repositories for
Peer Review Disciplines
Peer Reviewers
Note - There are no weightings
UoE rating
Research Evaluation Committees
by FoR
Peer Reviewer
REC Member
Stages of Evaluation
Stage 1
Stage 2A
Stage 2B
Stage 2C
Stage 3
REC Members
evaluate
assigned
material and
record
preliminary
evaluations in
SEER
REC Members
evaluate
assigned
material and
record
preliminary
evaluations in
SEER
REC Members
are given access
to the
preliminary
evaluation
outcomes for 2
digit UoEs from
other REC
members and
Peer Reviewers’
evaluations for
moderation
REC Members
are given view
only to access to
moderated 2 &
4 digit
evaluations to
prepare for
Stage 3 meeting
REC Members
meet to finalise
all UoEs
Stage 1
ERA Peer
Reviewers
evaluate
assigned
material and
record
evaluations in
SEER
ERA 2015 REC Member and Peer
Reviewer Recruitment
Research Evaluation Committee (REC) Membership
• 8 REC Chairs announced in November—to ensure ERA
evaluations are of the highest standard
• Full list of RECs members can be found on the ARC
website
Peer Reviewers
• The call for peer reviewer nominations closed on
26 November 2014
• ~ 1300 peer reviewers—even better than 2012!
• ~ 1100 of these were assigned material for evaluation
• ~ 1000 peer reviewers submitted reports
Recruitment of REC members
•
•
•
Universities invited to nominate REC members in
mid–2014
Maximum of 50 nominations per university—10%
of nominations from other countries
Nominations against published criteria:




•
•
research excellence as well as a sound understanding of
the importance of research
broad discipline expertise (interdisciplinary experience is
desirable)
professional and academic standing
interpersonal and team skills
Database of 700 researchers with broad range of
expertise
Nominees not selected were invited to participate
as peer reviewers (in the ERA peer review
disciplines)
Recruitment of Peer Reviewers
•
Peer review is conducted by REC members in
concert with external peer reviewers who are
appointed to
 extend the disciplinary expertise of the RECs and to
 share the workload
•
Peer reviewers for ERA 2015 were required for the
following 2 digit FoRs: 08, 12–16, and 18–22 (as
well as 0101, 1005 and 1006)
•
All universities were invited to nominate peer
reviewers (national and international)
•
Peer reviewers are nominated by six-digit expertise
to facilitate assignment
Peer Review in ERA 2015—
Overview

Universities nominate a ‘representative’ sample
30% of total outputs for each assessable unit (e.g.
ANU1801)

The sample is lodged in a digital repository and
made available to ERA reviewers through SEER

REC members conduct peer review of the
nominated sample

REC members also assign from a pool of peer
reviewers to extend the expertise available and
assist with workload

Peer reviewers use a structured report form

The final rating = all reviewer assessments +
dashboard information
Peer Reviewer Assignment
•
Every assessable unit is assigned three REC members
who conduct peer review
•
Every assessable unit is also assigned a minimum of two
peer reviewers
•
Assignment to peer reviewers is based on:





target maximum of 50 outputs for each reviewer
target maximum of 6 units for each reviewer
no reviewer should have only one assigned unit*
no unit has only one peer reviewer
notionally enough reviewers are assigned to mean all
outputs in the unit will be read by at least one reviewer
*This means that peer reviewers have a target of no more than 25 outputs in any
one unit. In a unit with 200 outputs a minimum of 8 reviewers have to be
assigned.
Peer Review Criteria
Criterion—Approach
•
What contextual information should the RECs know about the common practice in the
discipline and the sample of outputs reviewed?
•
Are the methodologies clearly reported?
•
Are the methodologies appropriate for the research questions or not? If not, what are
limitations/effects?
•
Are the venues/outlets appropriate for reporting the research? Again, there might be
reference to common practice for the discipline.
•
What is the quality of the venues/outlets?
Criterion—Contribution
•
Contribution of the group of outputs reviewed to the field and/or practice – nationally
and/or internationally
•
Originality and degree of innovation in the research approach or research questions
•
Theoretical and conceptual rigour
•
Depth of discussion and analysis
•
Contribution to the further development of knowledge and understanding
ERA 2015 Peer Reviewer Feedback
Survey of Peer Reviewers
• Peer reviewers invited to complete an online
survey after they had finished their assessments
• 660 of approximately 1300 reviewers returned
completed survey
• Peer reviewers were also invited to submit
comment on any aspect of their engagement in
ERA 2015
• Most additional comment was already captured
in the structured questionnaire and so there
were no additional common themes
Were the instruction were clear?
Agree
Neutral
Disagree
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Was the assigned workload manageable?
Agree
Neutral
Disagree
0%
10%
20%
30%
40%
50%
60%
70%
Main issues raised by
Peer Reviewers
•
Workload
•
Allow supplementary metrics (e.g. Google
Scholar, journal rankings etc.)
•
Universities provide contextual/research
statement for each 30% sample
•
Standardise user-interface for university
repositories
•
Recognition of the contribution of peer
reviewers (e.g. NHMRC example)
•
Feedback—especially for ‘first time’ reviewers
Can we talk about ERA 2015 results?
Not yet….
ERA 2015—some feedback
•
•
•
•
Inclusion of text books—to be included these
must meet the definition of research—we saw
a number of these
Watch out for predatory publishers which
don’t peer review manuscripts—we saw a
number of these
Make sure the coding of outputs reflects the
content of the output—know your ANZSRC
codes
Watch out for ‘recycled’ content—a revision or
subsequent version must include substantial
new research content
Know your ANZSRC codes
18 Law and Legal Studies
1801 Law
1802 Maori Law
1899 Other Law and Legal Studies
Exclusions:
a) Criminology, including policing and correctional
theory, is included in Group 1602 Criminology.
b) Legal ethics and human rights and justice issues
are included in Group 2201 Applied Ethics.
c) History and philosophy of law and justice is
included in Group 2202 History and Philosophy of
Specific Fields.
Know your ANZSRC codes
1801 LAW
This group covers law.
It includes:
•
legal institutions;
•
legal theory and practice; and
•
litigation, adjudication and dispute
resolution.
This group has twenty-seven fields:
180101 Aboriginal and Torres Strait Islander Law
180102 Access to Justice
180103 Administrative Law
180104 Civil Law and Procedure
180105 Commercial and Contract Law
180106 Comparative Law
180107 Conflict of Laws (Private International Law)
180108 Constitutional Law
180109 Corporations and Associations Law
180110 Criminal Law and Procedure
180111 Environmental and Natural Resources Law
180112 Equity and Trusts Law
180113 Family Law
180114 Human Rights Law
180115 Intellectual Property Law
180116 International Law (excl. International Trade Law)
180117 International Trade Law
180118 Labour Law
180119 Law and Society
180120 Legal Institutions (incl. Courts and Justice Systems)
180121 Legal Practice, Lawyering and the Legal Profession
180122 Legal Theory, Jurisprudence and Legal
Interpretation
180123 Litigation, Adjudication and Dispute Resolution
180124 Property Law (excl. Intellectual Property Law)
180125 Taxation Law
180126 Tort Law
180199 Law not elsewhere classified
Exclusions:
a) Criminology, including policing and correctional theory,
is included in Group 1602 Criminology.
b) Legal ethics and human rights and justice issues are
included in Group 2201 Applied Ethics.
c) History and philosophy of law and justice is included
in Group 2202 History and Philosophy of Specific Fields.
Know your ANZSRC codes
1802 MAORI LAW
This group covers Maori Law.
This group has five fields:
180201 Nga Tikanga Maori (Maori Customary Law)
180202 Te Maori Whakahaere Rauemi (Maori Resource Law))
180203 Te Tiriti o Waitangi (The Treaty of Waitangi)
180204 Te Ture Whenua (Maori Land Law)
180299 Maori Law not elsewhere classified
Know your ANZSRC codes
1899 OTHER LAW AND LEGAL STUDIES *DOES YOUR OUTPUT
REALLY BELONG HERE?
This group covers law and legal studies not elsewhere classified.
This group has one field:
189999 Law and Legal Studies not elsewhere classified
Exclusions:
a) Criminology, including policing and correctional theory, is
included in Group 1602 Criminology.
b) Legal ethics and human rights and justice issues are included
in Group 2201 Applied Ethics.
c) History and philosophy of law and justice is included
in Group 2202 History and Philosophy of Specific Fields.
ERA-HERDC Alignment Update
ERA-HERDC Alignment
Background
•
PhillipsKPA Review of Reporting Requirements for Universities
•
Department of Education and the ARC working together
•
A single research data collection
•
Part of Government’s Reducing Red Tape agenda
Aim
•
A more efficient streamlined process for the collection of data to inform RBG and
evaluate excellence
Potential Efficiencies
•
University admin staff and IT systems only need to cater for one data set
•
Reduces opportunities for error and misunderstanding—enhancing integrity,
transparency and utility of data
•
Auditing requirements in relation to the data will be streamlined
ERA-HERDC Alignment cont.
• Consultation paper on options for alignment
(8 December–13 February)
• Sector working group (3 July–21 September)
• Draft specifications completed
• Next steps – sector consultation on draft
specifications, when…?
Thank you