Campaign proposal: Write an open review for lesser known scientists

[willjharrison]

• Campaign co-creator/s: @CooperSmout

• Rationale: We all spend some of our time writing peer reviews, so why don't we start making those reviews open and equitable? This campaign has three goals: 1) increase the visibility of pre-prints written by lesser known scientists, 2) provide constructive and critical feedback (i.e. peer review) for manuscripts written by lesser known scientists, and 3) increase awareness and use of PREreview, a progressive peer-to-peer review system that aims to "bring more diversity and equity to scholarly peer review".

• Action: Pledgees will submit one publicly-signed peer-review via PREreview. The review will be written with the same time and care as would be required of a review for any journal. Importantly, reviews will be written for manuscripts of "lesser known" scientists. This is obviously hard to quantify, but one simple metric may be to review a manuscript written by authors whose names you do not recognise.

• Eligibility criteria: Anyone can sign this pledge provided they are eligible to write a review on PREreview, which, at the time of writing, only requires an ORCiD.

• Optional anonymity: Yes

• Threshold: You will have to perform this action when 10 people in your designated field of research have taken the pledge. For example, if you consider yourself a cognitive psychologist, your pledge will become active when 9 other cognitive psychologists pledge, regardless of how many physicists have pledged.

[CooperSmout]

Re: the rationale text
Perhaps we should mention the Mathew effect (Merton) somewhere in the rationale, which speaks to the type of bias we're trying to counteract? I also found an old draft of a similar campaign we (@dasaderi and @dlholf) discussed a while back, which went like this: "The current system of scientific knowledge dissemination is flawed. PREreview is a grassroots project with the mission of increasing diverse engagement of researchers in the peer review process. To do that, PREreview focuses on training, community building, and technology to support it. This XXX PREreview will launch a new, open-source platform built from the ground up with community members and their needs at the center of the design and development."

Re: where to draft the rationale
Was thinking it would be good if we can draft the campaign text as we go, rather than editing the OP in this thread. As a trial, I've created a temporary markdown file for us to start creating the campaign text. It can be edited collaboratively using HackMD (kinda like Google Docs) using this link. @willjharrison maybe take a look and let me know what you think? There are other ways to do this of course, but if we do it this way the text can then be directly uploaded to the website (it will do so automatically when I change the filename) and it will also track all the contributions people have made.

[dasaderi]

Thank you @willjharrison @CooperSmout for starting this campaign. To get more info about how PREreview preprint review platform works, folks can refer to this page.

If anyone needs assistance with creating a profile, searching for the preprint to review, and/or review it, I'm here to help.

Thanks everyone!

[nsunami]

Great campaign! Just a quick question---is PREreview open to articles outside bio/med field now? I really like the platform and the vision, but I do not seem to see papers in psychology etc.

[gavinscode]

reviews will be written for manuscripts of "lesser known" scientists. This is obviously hard to quantify, but one simple metric may be to review a manuscript written by authors whose names you do not recognise.

You could make this a bit more explicit by requiring something like: the lead or senior author is a student, postdoc or junior PI (at least at the time of pre-print submission).
I think the author's seniority should be fairly easy to determine from most researchers' public profiles (perhaps not who is a junior PI...).
Something could also be done by using the metrics of the lead/senior authors as a proxy for "lesser-known", like having an H-index less than 10, but that might also be pushing the use of metrics in an undesirable direction.

[CooperSmout]

Great campaign! Just a quick question---is PREreview open to articles outside bio/med field now? I really like the platform and the vision, but I do not seem to see papers in psychology etc.

Yep, it should be able to find any preprint now, but you need to add it to the system first. Go to the platform, click 'Add PREreview', enter the DOI you're interested in, and it should bring up the preprint. Then you just need to either request a review or add a review for the system to recognise it.

[nsunami]

Yep, it should be able to find any preprint now, but you need to add it to the system first. Go to the platform, click 'Add PREreview', enter the DOI you're interested in, and it should bring up the preprint. Then you just need to either request a review or add a review for the system to recognise it.

Thanks! So, I believe this campaign (if successful) can potentially encourage less-famous researchers to submit papers. The successful activation of this pledge means the availability of reviewers.

I agree with @gavinscode 's point on clarifying more about what "lesser-known" means. The current proposal gives an example of "not recognizing an author". Perhaps, adding more examples would clarify things a bit?

[CooperSmout]

Something could also be done by using the metrics of the lead/senior authors as a proxy for "lesser-known", like having an H-index less than 10, but that might also be pushing the use of metrics in an undesirable direction.

I like this idea, because it's (1) clearly identifiable, and (2) speaks to the type of attention bias we want to counteract. I also don't think it's pushing in an undesirable direction -- actually the opposite, because it would be serving to counteract the attention bias that high H-index authors normally benefit from.

[gavinscode]

Yeah, I suppose what I meant by undesirable direction was using quantitative metrics to evaluate research quality/prestige. But you are right @CooperSmout - aiming to be inclusive of researchers who have low scores on metrics that are typically considered important could actually work to subvert the undesirable aspects of the metric, like Mathew's effect, Goodhart's law, etc.
(incidentally, I didn't realize that my H-index was 10 when I suggested this - coincidence!)

[dasaderi]

Thanks everyone for this discussion. I was part of the team that worked on the hiddenpreprints project and always thought one day we would have included something like the "shadow index" in the search within PREreview, but we have not yet done it. It would be so great to be able to show the "visibility" of the preprint when reviewing it as a way to inform the reader and reviewer about it and potentially provide suggestions on how to boost visibility of that preprint and the PREreview as a way to bring more equity to the process.

[gavinscode]

I saw that this discussion at ASAPBio also talks about highlighting less visible preprints

https://asapbio.org/feedbackasap#1623332548545-4525cf52-291e

[willjharrison]

Sorry I have taken so long to respond, folks. Thanks for your feedback so far. I like the idea of using the heuristic that the lead and senior author both have an H-index < 10. I don't think though that it should be made concrete in the pledge per se, because a goal is to increase use of the PREreview platform. As long as people are "doing their best", then I think the general spirit of the broader goals are fulfilled. In an extreme case, if someone reviews for an author whose H-index is 100, it'll still hopefully benefit PREreview and openness of reviews (potentially more so, by raising the profile with Prof Fancy Pants' name).

[CooperSmout]

@willjharrison i agree with that -- so perhaps the only 'concrete' requirement should be that people post a PREreview (of any kind), but we can stress that people should try to focus on lesser known scientists using one of a number of criteria (beginning with the H-index idea and other ideas in the ASAPBio blog). Then when people complete their pledge, we could either check these criteria ourselves, or get people to indicate which criteria they followed when they post their review.

[CooperSmout]

(incidentally, I didn't realize that my H-index was 10 when I suggested this - coincidence!)

haha thankyou for disclosing your conflict of interest ;)

[dylangomes]

I think another potential issue with metrics like an h-index below 10, is that might be difficult to know for a lot of people. Not everyone links all of their work through things like Google Scholar or ORCID. So I guess this is just to say that making it not a 'concrete' requirement makes sense to me.