An R package providing permutation-based significance testing for generalised linear models, linear mixed-effects models, and binomial logistic regression. Designed for small or non-normally distributed neuroscience datasets where standard asymptotic p-values are unreliable.
ptestR wraps glm, lme4::lmer, and binomial glm with a permutation loop: for each model, it generates a null distribution of the test statistic by randomly rearranging the outcome variable (permNum times), then computes p.perm as the proportion of permuted statistics that are at least as extreme as the observed one. This approach requires far fewer distributional assumptions than standard likelihood-ratio or Wald tests, making it well-suited to neuroimaging, EEG, and other biomedical datasets with repeated measures and small samples.
The package was developed for and is used in:
França LGS, Ciarrusta J, Gale-Grant O, Fenn-Moltu S, Fitzgibbon S, Chew A, Falconer S, Dimitrova R, Cordero-Grande L, Price AN, Hughes E, O'Muircheartaigh J, Duff E, Tuulari JJ, Deco G, Counsell SJ, Hajnal JV, Nosarti C, Arichi T, Edwards AD, McAlonan G, Batalle D (2024). Neonatal brain dynamic functional connectivity in term and preterm infants and its association with early childhood neurodevelopment. Nature Communications, 15, 16. https://doi.org/10.1038/s41467-023-44050-z
In that study, grouped_perm_glm and grouped_perm_glmm were used with 10 000 permutations (two-sided, FDR-corrected at α = 5%) to test associations between neonatal fMRI brain state features and postmenstrual age, postnatal days, preterm birth, and neurodevelopmental outcomes (Bayley-III and Q-CHAT) at 18 months of age in 390 term- and preterm-born neonates from the developing Human Connectome Project (dHCP).
It is also used in:
Ferré IBS, Corso G, dos Santos Lima GZ, Lopes SR, Leocadio-Miguel MA, França LGS, de Lima Prado T, Araújo JF (2024). Cycling reduces the entropy of neuronal activity in the human adult cortex. PLOS ONE, 19(10): e0298703. https://doi.org/10.1371/journal.pone.0298703
ptestR is not on CRAN. Install directly from GitHub:
# install.packages("remotes") # if needed
remotes::install_github("CoDe-Neuro/ptestR")The package exports three functions, one for each model class:
| Function | Model type | Underlying fit | Test statistic |
|---|---|---|---|
grouped_perm_glm() |
Generalised linear model | stats::glm |
t-statistic |
grouped_perm_glmm() |
Linear mixed-effects model | lme4::lmer |
t-statistic |
grouped_perm_binoglm() |
Binomial logistic regression | stats::glm(family = binomial) |
z-statistic |
All three share the same call signature and return a tidy tibble with one row per model term.
Permutation test for generalised linear models.
grouped_perm_glm(tbl, formla, var_to_perm, family = gaussian,
permNum = 1000, seed = 42)| Argument | Type | Description |
|---|---|---|
tbl |
data frame / tibble | Data containing all model variables |
formla |
formula | Model formula; passed to glm |
var_to_perm |
character | Name of the column to permute (typically the outcome) |
family |
family object | Error distribution and link function (default: gaussian) |
permNum |
integer | Number of permutations (default: 1000) |
seed |
integer | Random seed for reproducibility (default: 42) |
Returns a tibble with columns: term, estimate, statistic, p.value, p.perm.
Example:
library(ptestR)
counts <- sample(1:100, 9, replace = TRUE)
outcomes <- c(18, 17, 15, 20, 10, 20, 25, 13, 12)
treatment <- gl(3, 3)
TBL <- data.frame(counts, outcomes, treatment)
grouped_perm_glm(TBL, outcomes ~ counts + treatment, "outcomes")Permutation test for linear mixed-effects models. The permutation is applied only to fixed effects; the random effects structure is preserved.
grouped_perm_glmm(tbl, formla, var_to_perm, permNum = 1000, seed = 42)| Argument | Type | Description |
|---|---|---|
tbl |
data frame / tibble | Data containing all model variables |
formla |
formula | Mixed-effects formula with fixed and random parts; passed to lme4::lmer |
var_to_perm |
character | Name of the column to permute (typically the outcome) |
permNum |
integer | Number of permutations (default: 1000) |
seed |
integer | Random seed for reproducibility (default: 42) |
Returns a tibble with columns: term, effect, estimate, statistic, p.perm.
Note: p.perm replaces the conventional p.value because lme4::lmer does not compute degrees of freedom or p-values by default. The permutation p-value is a fully nonparametric alternative.
Example:
library(ptestR)
library(sdamr)
data("anchoring")
grouped_perm_glmm(
anchoring,
everest_feet ~ anchor + sex + (1 | referrer),
"everest_feet",
permNum = 1000,
seed = 42
)Typical usage pattern in a grouped analysis (applied per brain state, as in the dHCP neonatal study):
df %>%
dplyr::group_by(State) %>%
tidyr::nest() %>%
dplyr::mutate(
mk = purrr::map(data, ~ grouped_perm_glmm(
.x,
formla = feature ~ PMA + PND + Sex + motion_outliers + (1 | subject),
var_to_perm = "feature",
permNum = 10000,
seed = 42
))
) %>%
tidyr::unnest(mk) %>%
dplyr::filter(term == "PMA")Permutation test for binomial logistic regression.
grouped_perm_binoglm(tbl, formla, var_to_perm, permNum = 1000, seed = 42)| Argument | Type | Description |
|---|---|---|
tbl |
data frame / tibble | Data containing all model variables |
formla |
formula | Logistic regression formula; passed to glm(family = binomial) |
var_to_perm |
character | Name of the binary outcome column to permute |
permNum |
integer | Number of permutations (default: 1000) |
seed |
integer | Random seed for reproducibility (default: 42) |
Returns a tibble with columns: term, estimate, statistic, p.value, p.perm.
Example:
counts <- c(10, 11, 8, 9, 6, 3, 5, 1)
gender <- c(0, 0, 0, 0, 0, 1, 1, 1)
TBL <- data.frame(counts, gender)
grouped_perm_binoglm(TBL, gender ~ counts, "gender", permNum = 500, seed = 1)All three functions return a tibble. The columns common to all functions are:
| Column | Description |
|---|---|
term |
Name of the regression term (intercept or predictor) |
estimate |
Estimated coefficient from the fitted model |
statistic |
Observed test statistic (t for GLM/GLMM, z for binomial) |
p.value |
Asymptotic two-sided p-value (GLM and binoglm only; absent in GLMM) |
p.perm |
Permutation p-value: proportion of permNum permuted statistics ≥ |observed| |
effect |
"fixed" or "ran_pars" (GLMM only; results are filtered to "fixed") |
p.perm = 0 means no permuted statistic was as extreme as the observed one given permNum iterations; report as p < 1/permNum (e.g. p < 0.0001 for permNum = 10000).
| Package | Role |
|---|---|
lme4 |
Fits linear mixed-effects models (lmer) |
broom |
Tidies glm output |
broom.mixed |
Tidies lmer output |
modelr |
Generates permuted datasets (permute) |
dplyr |
Data manipulation |
purrr |
Functional iteration over permutations |
tidyr |
Reshaping outputs |
tibble |
Tidy data frames |
If you use ptestR in your research, please cite the paper it was developed for:
@article{franca2024neonatal,
title = {Neonatal brain dynamic functional connectivity in term and preterm infants
and its association with early childhood neurodevelopment},
author = {Fran{\c{c}}a, Lucas G. S. and Ciarrusta, Judit and Gale-Grant, Oliver and
Fenn-Moltu, Sunniva and Fitzgibbon, Sean and Chew, Andrew and
Falconer, Shona and Dimitrova, Ralica and Cordero-Grande, Lucilio and
Price, Anthony N. and Hughes, Emer and O'Muircheartaigh, Jonathan and
Duff, Eugene and Tuulari, Jetro J. and Deco, Gustavo and
Counsell, Serena J. and Hajnal, Joseph V. and Nosarti, Chiara and
Arichi, Tomoki and Edwards, A. David and McAlonan, Grainne and
Batalle, Dafnis},
journal = {Nature Communications},
volume = {15},
pages = {16},
year = {2024},
doi = {10.1038/s41467-023-44050-z}
}Lucas G. S. França, Yan Ge, and Dafnis Batallé — CoDe-Neuro Lab, King's College London.
MIT © 2022 Lucas G. S. França, Yan Ge, and Dafnis Batallé