Skip to content

README.md

Latest commit

 

History

History
173 lines (124 loc) · 11.8 KB

README.md

File metadata and controls

173 lines (124 loc) · 11.8 KB
Raw

Code and data for "Causal evidence of identity-based and institutional biases in accessing paywalled articles and scientific data"

Ibrahim, Liu, Mengal, Alshaibi, Kaufman, Zaki & Rahwan.

This repository contains the code and data needed to reproduce the quantitative results of the paper:

Study What it is Folder
Study 2 Observational analysis of citations to paywalled vs. open-access papers (OpenAlex) observational_analyses/
Study 3 Randomized audit experiment: requests for paywalled papers (18 conditions) audit_experiments/paywalled_papers_audit/
Study 4 Observational analysis of data citations to papers whose data are "available upon request" (Semantic Scholar + OpenAlex) observational_analyses/
Study 5 Randomized audit experiment: requests for "available upon request" datasets (16 conditions) audit_experiments/dataset_audit/

Study 1 (survey and interviews) is not included: aggregate survey statistics are reported in Supplementary Tables 1–5 and 26, and interview transcripts cannot be shared under the study's consent protocol (see the paper's Data Availability statement).

Repository structure

README.md                  this file
LICENSE                    MIT licence
requirements.txt           pinned Python dependencies
run_audit_demo.sh          runs all audit-experiment analyses (the demo)
expected_output/           output of run_audit_demo.sh on our machine, for comparison
audit_experiments/
  paywalled_papers_audit/  Study 3: anonymized data, regression models, Figure 3
  dataset_audit/           Study 5: anonymized data, regression models, Figure 4
observational_analyses/
  notebooks/               Jupyter notebooks for Studies 2 and 4
  src/                     Doc2Vec training and knowledge-breadth scripts (Study 2)
  data/                    small input files; large files are downloaded separately (see below)
  figures/                 figures produced by the notebooks

1. System requirements

Software

  • Python 3.12 (tested with 3.12.2 and 3.12.8; Python 3.10 or later should work)
  • The Python packages listed with exact versions in requirements.txt: numpy 1.26.4, pandas 2.2.3, scipy 1.13.1, statsmodels 0.14.4, matplotlib 3.9.4, seaborn 0.13.2, geopandas 1.0.1, gensim 4.3.3, tqdm 4.67.1, ipywidgets 8.1.5, jupyterlab 4.3.5
  • Any operating system that runs Python (macOS, Linux, Windows). run_audit_demo.sh needs a bash shell; on Windows, run the four Python scripts directly (see section 3).

Tested on

  • macOS 26.7.1 (Apple M1 Pro, 16 GB RAM), Python 3.12.2: audit-experiment analyses (Studies 3 and 5), including the demo
  • Linux high-performance computing cluster (NYU Abu Dhabi), Python 3.12.8: observational notebooks (Studies 2 and 4)

Hardware

  • The demo and the audit-experiment analyses run on a normal desktop or laptop computer. No non-standard hardware is needed.
  • The observational notebooks read input files of up to about 3 GB each, plus local snapshots of OpenAlex (several hundred GB of disk), Semantic Scholar and SciSciNet.
  • Fine-tuning ModernBERT (Study_4_classification.ipynb) needs a CUDA-capable GPU.
  • Training the Doc2Vec model from scratch (Study 2, observational_analyses/src/doc2vec.py) is optional, because its outputs are provided as downloadable files. On 42 million abstracts it needed about 430 GB of RAM and 4 days on a high-performance computing cluster.

2. Installation guide

git clone https://github.com/comnetsAD/bias_academia.git
cd bias_academia
python3 -m venv venv
source venv/bin/activate        # on Windows: venv\Scripts\activate
pip install -r requirements.txt

Typical install time on a normal desktop computer: about 2 minutes (1 min 42 s measured on the test machine above, on a fresh virtual environment, broadband connection).

3. Demo

The demo runs every analysis for both audit experiments (Studies 3 and 5) on the anonymized experimental data included in this repository.

Instructions

source venv/bin/activate
./run_audit_demo.sh

On Windows, or to run a single analysis, go to the study folder and run the scripts directly, for example cd audit_experiments/dataset_audit and then python regression_and_figures.py and python analysis_bonferroni.py.

Expected output

Each script prints its regression tables to a .log file in its own folder and saves figures as PDFs:

Folder Files written Contents
paywalled_papers_audit/ regression_and_figures.log, figure_2a.pdf, figure_2b.pdf Study 3 logistic regressions of reply and of positive reply (base, controls, interaction models); reply rate by condition (manuscript Fig. 3A) and log-odds coefficients (Fig. 3B)
paywalled_papers_audit/ analysis_bonferroni.log, figure_2a_bonferroni.pdf, figure_2b_bonferroni.pdf, log_odds_model_2_bonferroni.csv the same models with Bonferroni-adjusted p-values and confidence intervals
dataset_audit/ regression_and_figures.log, figure_4a.pdf, figure_4b.pdf Study 5 logistic regressions of reply (base, controls, interaction models); reply rate by condition (Fig. 4A) and log-odds coefficients (Fig. 4B)
dataset_audit/ analysis_bonferroni.log, figure_4b_bonferroni.pdf, logistic_regression_reply_rate_bonferroni.csv the same models with Bonferroni-adjusted p-values and confidence intervals

The script file names (figure_2*, figure_4*) follow an earlier figure numbering; they correspond to Figures 3 and 4 of the manuscript.

Reference copies of all of these outputs are in expected_output/. Your coefficients should match them to the printed precision. Matplotlib may warn that the Helvetica font is missing; this only changes the figure font.

For example, the Study 5 model with controls (dataset_audit/regression_and_figures.log, N = 9,971) should report these log-odds coefficients:

Term Coefficient p
South African sender (vs. English) −0.253 < 0.001
South African university (vs. English) −0.226 < 0.001
Low-ranked university (vs. high) −0.171 0.005

These correspond to the 22.3%, 20.2% and 15.7% lower odds of reply reported in the paper.

Expected run time on a normal desktop computer: about 20 seconds for the whole demo (19 s measured on the test machine above).

4. Instructions for use

Running the audit analyses on your own data

The scripts read one CSV per experiment, with one row per email. To analyse a new audit experiment, prepare a CSV with the same columns and replace the input file, or change the pd.read_csv(...) line at the top of the script.

Study 3 (paywalled_papers_audit_reply_data.csv):

Column Values
sender_name sender condition (Karl Muller, Olu Adeyemi, Faisal Khan)
sender_status request purpose (Journal Article, Class project)
sender_university New York University, Louisiana State University, San Diego State University
paper_domain Health Sciences, Life Sciences, Physical Sciences, Social Sciences
h_index, academic_age recipient h-index and years since first publication (from OpenAlex)
institution_continent region of the recipient's institution
matching_continent whether the sender's region matches the recipient's (No Match, Match German, Match Nigerian, Match Asian)
replied 1 if a substantive (non-automated) reply was received, else 0
is_positive_reply 1 if the reply shared the requested paper, else 0

Study 5 (dataset_audit_data.csv, one row per delivered email):

Column Values
sender, sender_country sender identity (James Whitfield / England, Kabelo Molefe / South Africa)
university_country England, South Africa
university_ranking High (QS 150–155), Low (QS 1001–1200)
request_purpose journal article, final course project
paper_domain as above
h_index, academic_age as above
replied 1 if a substantive reply was received, else 0
reply_type for replies: positive, conditional, negative (codebook: Supplementary Table 25)

Reference levels are set in the model formulas, for example Treatment(reference='England'). If your factor levels have different names, change them there.

Reproducing the observational results (Studies 2 and 4)

  1. Get the input data.
    • Small inputs are already in observational_analyses/data/.
    • Large inputs (files over 100 MB) must be downloaded from Figshare and Google Drive, and placed in observational_analyses/data/large/.
    • Some notebooks also need a local copy of the OpenAlex snapshot and of SciSciNet v1. SciSciNet v2 also works but may give slightly different numbers.
  2. Set the paths. At the top of each notebook, set OPENALEX and SCISCINET to your local copies. DIR (../data/study_2_data or ../data/study_4_data) and LARGEFILE (../data/large) work as they are.
  3. Run the notebooks. Start jupyter lab and run each notebook from top to bottom. Apart from the Study 4 order noted below the table, the notebooks are independent of each other. Figures are written to observational_analyses/figures/.
Notebook Analyses Figures written Needs OpenAlex / SciSciNet?
Study_2AB.ipynb share of references to paywalled papers by country; Global North vs. South t-tests and Cohen's d (Fig. 2A–B; Supplementary Fig. 2) OA_map_percentageNonOA.pdf, map_legend.pdf, nonOA_distribution.pdf, nonOA_distribution_Q1.pdf, nonOA_distribution_T3.pdf no (large files only)
Study_2CD.ipynb knowledge breadth of reference lists (Fig. 2D; Supplementary Fig. 3) Knowledge_breadth_all.pdf, Knowledge_breadth_all_50dim.pdf no (large files only)
Study_2EFGH.ipynb OA type and citation share, impact, novelty and disruptiveness regressions, GVIFs, regional back-of-the-envelope estimate Impact_novelty_disruptiveness_regression.pdf, Impact_paywalled_subregional_estimation.pdf yes
Study_4_merge_OpenAlex.ipynb matches Semantic Scholar papers to OpenAlex (by MAG ID, DOI, then the OpenAlex API) and assigns paper regions none (writes intermediate data files) Semantic Scholar snapshot, SciSciNet v2; queries the OpenAlex API
Study_4_classification.ipynb citation-intent classification: stage 1 labels with GPT-5.4-mini (gpt-5.4-mini-2026-03-17), stage 2 fine-tunes and applies ModernBERT (answerdotai/ModernBERT-base); data-availability status none Semantic Scholar snapshot; an OpenAI API key; a CUDA GPU
Study_4.ipynb data-citation logit models and GVIFs (Table 1; Supplementary Tables 17–18) none (tables printed in the notebook) Semantic Scholar snapshot, SciSciNet v2

For Study 4, run Study_4_merge_OpenAlex.ipynb and Study_4_classification.ipynb before Study_4.ipynb. The classification notebook needs extra packages: pip install torch transformers datasets evaluate openai.

The Doc2Vec embeddings used by Study_2CD.ipynb can be regenerated with src/doc2vec.py (training), src/ComputePaperVecDoc2vec.py (inference) and src/refListDiameter.py (knowledge breadth). These steps need a high-performance computing cluster (see Hardware).

Data

The audit-experiment files contain no names, email addresses or other identifiers of the researchers who were emailed; recipients are described only by h-index, academic age, field and institutional region. The sender names are those of the fictional requesters.

Licence

Released under the MIT licence; see LICENSE.

Citation

If you use this code or data, please cite the paper (citation to be added on publication).