peer review Archives - Future Leaders Fellows Development Network

Dr Sara Shinton

This blog summarises some of the content from our Resilience, Response & Recycling session run by Sara Shinton, FLF Director and John Morison, Professor of Jurisprudence at Queen’s University Belfast. It’s aimed at anyone who has received a negative peer review response and wants some advice on how to construct an effective response.

The slides and handouts from the session are linked to below. I’ve written them to be reasonably self-explanatory so hope they help you to identify personal resilience strategies and to help you decide what is the best response to any proposal rejections. This blog will focus on the discussions we had in the workshop with John Morison, in response to four questions about responding to peer review. We were mostly talking about the grant processes, but there are parallels with publications too.

What makes a convincing response?
What pitfalls have you seen people fall into?
How do YOU write an effective response?
What are YOUR responses to reviews?

John started by saying that he’d applied for many grants and received many rejections. He echoed the early emphasis on resilience, saying that moving on from sadness and anger are vital, but not easy.

He started by reassuring our attendees about the fairness and quality of panel decisions. When he first sat on a panel he thought he had over prepared by reading everything but quickly realise that everyone else was as prepared and familiar with the proposals they were deciding on. He pointed out that panels often cover a wide range of topics, going right to the edges of each individual’s comfort zone, so they rely on the reports from the expert reviewers and the applicant response to these.

He recalled occasions where he’d seen disciplinary squabbles and what he called discipline “sectarianism” evident in reviews and that this was dismissed by the panels he had sat on. He also explained that the panel always recognises the effort each applicant has made and they are determined to safeguard the process so they are treated fairly.

What makes a convincing response?

John is impressed by responses that build on the application, but also go beyond this in their responses. They don’t take on reviewers in combative terms, but where good feedback has found a weakness will give ground graciously and admit that they haven’t presented enough evidence. Good responses are written with a confidence about their research plans and their ability to deliver on their ambitions, reassuring that they are working on the scale they have set out, have access to the resources they will need and quickly clear up any technical issues. John commented that in his field (Law) the reviewers typically question the “doability” of a programme of work rather than having issues with theoretical frameworks. In short, your response must reassure the panel that you can do what you have set out in your proposal.

What pitfalls have you seen people fall into?

John has seen problems for applicants who have indicated their proposal is interdisciplinary, when they largely sit in a single discipline with expertise and track record in that area. Their proposal will be sent to experts in the other fields and they will often identify omissions and assumptions that weaken it in their view. So he advises applicants to think carefully about who will review and to be careful about positioning work as IDR unless they are confident it is convincing to these other disciplines.

He explained that as a panel member, you draw up a checklist of the reviewer comments and review the response to ensure all have been addressed. This is a fundamental and simple piece of advice for applicants to follow. He also encourages you to state what seems obvious! Your panel is very unlikely to include people in your specific area of research and even if they are, many panels are under clear instruction NOT to re-review (FLFs who observe the sift panels in July will see this policed by chairs and conveners if it happens). So put everything in, even if you feel it’s obvious and evident in the proposal.

We also discussed situations where we had seen applicants blind-sided by an unfair review which had distracted them from comments from more reasonable reviewers. If the response only address critical challenges, but overlooks points of clarification, the panel cannot confirm the reviewers’ concerns are met and usually the proposal won’t progress.

How do YOU write an effective response?

John reflected on times where he wished he had requested an unfair review be struck out by the funder. It’s always worth asking for it to be removed before your proposal reaches the panel if you can demonstrate it isn’t a fair or valid critique, but get advice from a mentor or close colleague to help judge objectively if this is the case.

Assuming all reviews are fair, John starts by ensuring that the originality of the proposal was clear to the reviewers. He had some experience of funders who were nervous of very novel approaches, and explained that sometimes you need to prepare the ground in the year or so before you plan to apply by raising the importance of the question you want to address through media engagement.

Although the panels he has sat on do not consider track record as a criteria, so as not to disadvantage earlier career applicants, he admitted that when a proposal isn’t convincing, that sometimes you may look to see if the researcher has done something similar previously which might explain why they have taken the “doability” for granted. Again, if the reviewers question this, set out your evidence against their concerns.

What are YOUR responses to reviews and decisions?

Finally, I asked John how he felt about the reviews and funding decisions he gets. It helps him to remember that the process is a competition and not everyone can win. His response after a decision is influenced by whether his work is classified as fundable or not fundable. He’s been on panels where they’ve hoped that a particular proposal will be above the funding threshold, but seen them lose out to results from other panels. If your research is above the funding threshold then take heart and think about how to strengthen it. He warned against submitting to another funder without a fundamental reworking in line with the second funders strategy and approach – funders are used to see each other’s proposals appearing with only cosmetic changes and they rarely land well.

Finally, John encouraged our attendees to really listen to people in their field when they talk about your work. What do THEY find interesting? What are THEY not convinced by? What do THEY suggest you might do differently? These external perspectives give you an insight into the reviewer mindset, so make sure you engage with their reactions.

A huge thank you to John for sharing his experiences, including some difficult ones with such honesty. He is likely to be typical of other senior researchers around you – don’t be afraid to ask your local leaders about how they respond to review and funding decisions.

Please click to access the handouts from the session. And click here to download the session slides. Click here to see University of Edinburgh Resilience guide it’s one of many available. (Sadly the IOP one seems to have left their website.)

By Sara Shinton, Director, Future Leaders Fellows Development Network

Interdisciplinary research (IDR) runs through the UK’s research and innovation strategy, but close to a century after the term was first used we are still struggling as a sector to evaluate it. This blog summarises the key points from a recent FLFDN event on Interdisciplinary Peer Review and points to some helpful additional resources.

Although our session was focused on IDR Peer Review, we started with scene setting. Professor Barry Smith gave a whistle-stop tour of the policy landscape highlighting the “downward push” from BEIS for “a continued expansion in the conception of what research is and does” and how this underpins the UKRI Strategy. The highlights from the latest Budget show that research and innovation is a Government priority, with interdisciplinary research reflected in much of their language (see Barry’s slides for details).

Much interdisciplinary work is done by researchers with strong disciplinary bases and a clear sense of the value their IDR brings to their home discipline. However, this isn’t always an easy fit for traditional career paths and the impacts on interdisciplinary researchers have been widely reported (one example being the British Academy’s 2016 Crossing Paths project.) The Stern Review was prompted by concerns about the disadvantaging of IDR in REF2014 leading to changes to the 2021 REF with an Interdisciplinary Research Advisory Panel (IDAP), which Barry was a member of, providing expertise to ensure that “that IDR should be neither advantaged nor disadvantaged for assessment in the REF”.

To help promote better practice, REF 2021 included a definition of IDR (although it’s worth noting that IDR shouldn’t be considered to be “one thing” as it’s a term that covers many different approaches and activities). Institutional submissions also included a statement on how they were supporting IDR and you may learn something about your host by reading theirs! (The REF website includes a summary of their approach to IDR and a protocol for assessment.)

Our next speaker, Professor Patrick Haggard shared experience a panel chair and reviewer, giving us insights into the way interdisciplinarity is conveyed by effective proposal writers. He looks for

Evidence that the PI understands the added value of working with a broad team AND that it brings extra challenges (reflected in the project structure, communication and team cohesion activities)
Clear statements of added value of an IDR approach, including the value to the disciplines involved and what the partnerships will stimulate
The use of efficient signposting language to help home in on benefits (“added value” and “synergy” were two examples)
A deeper than “text book” knowledge of the disciplines in the partnership, providing evidence that the relationships are effective and have been built through genuine dialogue, listening and reframing
A compelling narrative (whilst being aware that as an IDR reviewer, you mustn’t get carried away by good writing)
Clear research questions which explain the need for IDR approaches, using schematics effectively to convey interrelationships succinctly

Underpinning much of this is respectful interaction between researchers with strong disciplinary identities. Patrick sees IDR strengthened when researchers act as guides for their collaborators, helping them to understand their expertise, being open to their perspectives and providing their own. A good IDR collaboration involves welcoming the “trespassers” whilst ensuring that there are benefits for all disciplines and partners.

As a reviewer, he warns that it takes more time to evaluate IDR as it will involve wider reading and reference checking. He has to be receptive to ideas outside his area of expertise and to invest in understanding them. He also had advice on the challenge many IDR reviewers face – feeling the limits of their knowledge don’t cover the work they evaluate. Funding bodies should ask reviewers for a statement about this and he is transparent, one example being finding it difficult to judge the degree of novelty in some fields.

Professor Catherine Lyall is an expert in IDR and brought insights from across the sector. Her publications include “Interdisciplinary Research Journeys” (2011, available on open access) and “Being an Interdisciplinary Academic” (2019). She spoke about what we mean by disciplines and how they are characterised and how this creates challenges for IDR as reviewers are often prone to pointing out disciplinary “weaknesses” rather than ID strengths. Added to this is that ID research goals are often different to disciplinary ones, requiring different research design and methods, but then they are subject to evaluation based on disciplinary models.

Catherine sees very little IDR peer review training from funders, but a motivated researcher can find a wealth of resources to help inform their approach. (Volkswagen Stiftung’s Freigest scheme and the Swiss National Science Foundation were applauded for good practice.) The EU-funded SHAPE-ID project includes Guided Pathways to help navigate resources by goal or role. There are lively communities of ID researchers and reviewers who share challenges and good practice, including:

ITD-Alliance – Global Alliance for Inter- and Transdisciplinarity

International Network for the Science of Team Science (INSciTS)

Integration and Implementation Insights – A community blog and repository of resources for improving research impact on complex real-world problems (i2insights.org)

Catherine’s key points were

Review of IDR can be biased against novelty (when judged from a disciplinary perspective)
Panels work most effectively with “integration experts” who can bring their own experiences of working across disciplinary boundaries and with different people
Funders who understand IDR will explain to reviewers what they mean by this term and how they want reviewers to approach it (as seen with the approach for REF 2021)
There is a HUGE body of work on IDR and it’s frustrating to see a lack of awareness of this leading to so much reinvention. This was a major motivator and driver for the SHAPE-ID project
The FLF community has a strong IDR base and she urged them to step forward to be evaluators and to look for opportunities to contribute to discussions about research evaluation
Most institutions will have a centre or institute which promote and foster IDR – look for this in your home institution

Finally, with more opportunities and invitations for peer review expected from funders, participants were urged to speak up for IDR and to ask “how will we ensure we don’t penalise IDR?” and to see themselves as the IDR champion in the room. The links dotted throughout this blog should equip reviewers for this role.