Campaign proposal: Publishing Registered Reports

[CooperSmout]

This campaign will ask researchers to pledge to submit a single Registered Report to a journal if and when a pre-specified threshold of community support has been met (e.g. X researchers have signed a pledge). Some of the parameters we will need to work out:

1. Should we target a particular field (e.g., Psychology) or make the campaign open to researchers in all fields?

2. What should be the threshold for pledge activation (e.g., pledges activate when 100 researchers have signed the pledge)? (this should be considered in conjunction with [1] above)

3. Should there be a time limit on the pledge (e.g., people have to submit the RR within 2 years of their pledge activating)?

4. Should the campaign expire within some period if it doesn't reach threshold (e.g. if we don't reach threshold within a year, take the campaign down)?

5. Any other considerations?

Note: If you're interested in joining this campaign and would like to be notified when it goes live, please add your name here

[samgrobson]

1. Should we target a particular field (e.g., Psychology) or make the campaign open to researchers in all fields?

My initial intuition on this is that it should be cross-field because the number of pledges overall will higher and seeing higher numbers might be more motivating to those involved and to people who are thinking of pledging. Plus, if it were me, I don't think I'd care too much if it was someone from my field or not who made the pledge

2. What should be the threshold for pledge activation (e.g., pledges activate when 100 researchers have signed the pledge)? (this should be considered in conjunction with [1] above)

If one field, I'd lean toward the 100 pledge threshold but it were across many fields, I'd lean more to 200 or so.

3. Should there be a time limit on the pledge (e.g., people have to submit the RR within 2 years of their pledge activating)?

I think a time limit would be good because it's a clear goal and might prompt a bit of urgency. 2 years (maybe 3) seems reasonable too

4. Should the campaign expire within some period if it doesn't reach threshold (e.g. if we don't reach threshold within a year, take the campaign down)?

Could take it down, but another option is to have the option to extend it if current pledgers want to.

[CooperSmout]

Thanks for your thoughts @samgrobson!

My initial intuition on this is that it should be cross-field because the number of pledges overall will higher and seeing higher numbers might be more motivating to those involved and to people who are thinking of pledging. Plus, if it were me, I don't think I'd care too much if it was someone from my field or not who made the pledge

Good to know! The existing OA campaigns were designed to be discipline specific, because that's the level the collective action problem is operating at -- it wouldn't do you much good if you're the only psychology researcher in the cohort when pledges go live, because you'd be supporting OA psych journals on your own. Seems to me that the same general principle applies here, but perhaps it's less of an issue because publishing a Registered Report is less likely to make or break your career (compared to boycotting all non-OA journals). Personally I'm happy to go with either direction, as these new campaigns will very much be trial and error to see what works/doesn't, but would be good to hear what others think too.

Could take it down, but another option is to have the option to extend it if current pledgers want to.

I like that idea -- get the pledgers to decide, or ask them to renew their pledge if the campaign expires. I only just started thinking about expiry dates, bc I noticed they use them elsewhere (Kickstarter and Collaction), and suspect it might help ppl sign by giving them peace of mind that their pledge won't suddenly activate many years down the track when they're in a very different situation.

[CooperSmout]

Someone raised a good point in this thread that we need to make it easy for people to see what journals they'll be able to publish in when taking this pledge. The best resource I've found so far is from the Center for Open Science (you need to click the second tab called 'Participating Journals'), but it's not currently broken down by research field so a little difficult to navigate. FOK has partnered with COS, so they might be willing to amend this list to make it more user-friendly, e.g. broken down by research field.

[Abmeister]

Should we target a particular field (e.g., Psychology) or make the campaign open to researchers in all fields?

It may be helpful to share the current journals that accept Registered Reports across fields. I think there are already helpful lists of these on some OSF pages. For example, there are about 10 neuroscience journals accepting RRs, but perhaps only 1 biology journal. That should help people decide their thresholds.

Should there be a time limit on the pledge (e.g., people have to submit the RR within 2 years of their pledge activating)?

I agree 3 years seems reasonable. Considering the review process can take much longer with RRs.

Should the campaign expire within some period if it doesn't reach threshold (e.g. if we don't reach threshold within a year, take the campaign down)?

It's good to have a fixed end time to encourage timeliness and mark a point at which time to evaluate success of the campaign. Maybe after 1 year, assess number of pledges, after 3 years, assess pledge activations, after 5 years, assess successful action and trends in RR publishing?

Any other considerations?

What are the perceived risks of an RR? I personally think it's pretty risk free, However certain studies may be geared towards an RR, such as a clinical trial or a paper where you fear null results not being published. The time cost seems to be the biggest down side, which is offset for larger studies, but perhaps not for smaller experimental designs.

Where to spread the word about the campaign? I wonder if we can start getting a list of upcoming conferences and events from the community who would be happy to share or ask if we can share the campaign?

[CooperSmout]

Maybe after 1 year, assess number of pledges, after 3 years, assess pledge activations, after 5 years, assess successful action and trends in RR publishing?

Just to clarify, there will only be one activation timepoint for this campaign (whenever we get X pledges), unlike the OA campaigns which allow user-defined thresholds (#5 #6 #7). So we will know how many pledges have activated (X) if we get there at all. But yes, it would be good to track compliance so we can assess our impact in years to come.

What are the perceived risks of an RR? I personally think it's pretty risk free, However certain studies may be geared towards an RR, such as a clinical trial or a paper where you fear null results not being published. The time cost seems to be the biggest down side, which is offset for larger studies, but perhaps not for smaller experimental designs.

Time is a factor, for sure (but this is offset somewhat by the guarantee you'll get published). Another perceived risk could be that not all journals offer RR, and so you're limiting yourself to those that do. That may be fine in fields where prestigious journals offer RR, but in others, you could be shooting yourself in the foot. But I agree with you that RRs are relatively risk-free (compared to publishing in OA journals), and so the main reasons for pledging to this campaign are probably more about joining a supportive community than solving a game-theoretic collective action problem (as per the OA campaigns). Brian Nosek suggested we need something to 'pull' people in, so I thought it could be useful if we offer some kind of support to pledgers, e.g. a webinar showing newbies how to do an RR (perhaps @evomellor could speak to whether this is possible/available?). We could also set up some kind of communication channel for the new community to share resources and discuss ideas/obstacles? This is a big part of why it would be great to have people who have already completed RRs join the campaign, so they can help others who are new to the process.

Where to spread the word about the campaign? I wonder if we can start getting a list of upcoming conferences and events from the community who would be happy to share or ask if we can share the campaign?

I've drafted a communication strategy here, which is geared mostly toward mailing lists, slack workspaces and twitter at this stage. But conferences would also be great if people are presenting their work anyway and feel comfortable adding a slide at the end of their talk. I'd be happy to make a new slide specifically for this campaign if so.

[AoifeOMahony]

I'm also in favour of it being across disciplines.

Someone raised a good point in this thread that we need to make it easy for people to see what journals they'll be able to publish in when taking this pledge. The best resource I've found so far is from the Center for Open Science (you need to click the second tab called 'Participating Journals'), but it's not currently broken down by research field so a little difficult to navigate. FOK has partnered with COS, so they might be willing to amend this list to make it more user-friendly, e.g. broken down by research field.

We're hoping to make the database of RRs that I'm creating available sometime early next year, and we've been coding journal subject areas in it, so should be able to filter by this.

[CooperSmout]

We're hoping to make the database of RRs that I'm creating available sometime early next year, and we've been coding journal subject areas in it, so should be able to filter by this

Brilliant, and great to know! Just to clarify, is this a database of journals that offer RR format, or articles that have been published under the RR format?

Also curious what subject taxonomy you're using. For FOK I'm thinking we will use the Fields of Research taxonomy, because this is what the Dimensions database uses to classify articles and so would allow us to classify articles/people automatically using the Dimensions database. If you happen to be using the same taxonomy it would be super handy for filtering according to each user, but not the end of the world if not (and sounds like you've already started coding)

[AoifeOMahony]

Brilliant, and great to know! Just to clarify, is this a database of journals that offer RR format, or articles that have been published under the RR format?

Articles published as RRs (and matched sample of regular research articles), not a list of journals.

Also curious what subject taxonomy you're using. For FOK I'm thinking we will use the Fields of Research taxonomy, because this is what the Dimensions database uses to classify articles and so would allow us to classify articles/people automatically using the Dimensions database

We don't stick 100% to that specific taxonomy but the vast majority would conform to it. Some journals kind of fit under multiple areas, and we have a journal or two that at the moment are just coded as 'Interdisciplinary' as their scope is very broad and difficult to pin down to even any one or two broad areas.

It probably is a lot more efficient to just see if journal subject areas can be added to the existing COS list of journals or similar, as our detailed database of the articles will take a while to be finished, and is limited anyway to the journals that have already published stage 2 RRs, rather than all of the journals that offer RRs.