Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

report.compare.loo: minor edits #427

Merged
merged 4 commits into from
May 7, 2024
Merged

report.compare.loo: minor edits #427

merged 4 commits into from
May 7, 2024

Conversation

DominiqueMakowski
Copy link
Member

After reading a bit more and hearing inputs in easystats/performance#716, I made some minor edits:

  • report IC by default as Brenton mentioned (ENP is optional as it seems like not entirely independent from the rest of the indices, so better to avoid verbosity by default I'd say - plus its interpretation remains somewhat complicated for more complex models)
  • emphasizing in the docs that the z-score/p-value is not really meaningful if the raw ELPD diff is small
> report(x)
The difference in predictive accuracy, as indexed by Expected Log Predictive Density (ELPD-LOO), suggests that 'm3' is the best model (LOOIC = 157.97), followed by 'm2'
(diff-ELPD = -13.41 +- 4.42, p = 0.002, LOOIC = 184.80) and 'm1' (diff-ELPD = -23.28 +- 4.41, p < .001, LOOIC = 204.53)

> report(x, include_IC = FALSE)
The difference in predictive accuracy, as indexed by Expected Log Predictive Density (ELPD-LOO), suggests that 'm3' is the best model (ELPD = -78.99), followed by 'm2'
(diff-ELPD = -13.41 +- 4.42, p = 0.002) and 'm1' (diff-ELPD = -23.28 +- 4.41, p < .001)

> report(x, include_ENP = TRUE)
The difference in predictive accuracy, as indexed by Expected Log Predictive Density (ELPD-LOO), suggests that 'm3' is the best model (LOOIC = 157.97, ENP = 4.92), followed
by 'm2' (diff-ELPD = -13.41 +- 4.42, p = 0.002, LOOIC = 184.80, ENP = 3.93) and 'm1' (diff-ELPD = -23.28 +- 4.41, p < .001, LOOIC = 204.53, ENP = 2.57)

@DominiqueMakowski DominiqueMakowski requested a review from rempsyc May 2, 2024 09:11
@rempsyc
Copy link
Member

rempsyc commented May 3, 2024

Thanks, but we see:

Undefined global functions or variables:
    pnorm
  Consider adding
    importFrom("stats", "pnorm")

___________________________________________

ENP   report.compare.loo.Rd:14
Error:
! Spelling errors found. Either correct them or update `inst/WORDLIST`
  using `spelling::update_wordlist()`.

@strengejacke
Copy link
Member

Thanks, but we see:

Undefined global functions or variables:
    pnorm
  Consider adding
    importFrom("stats", "pnorm")

___________________________________________

ENP   report.compare.loo.Rd:14
Error:
! Spelling errors found. Either correct them or update `inst/WORDLIST`
  using `spelling::update_wordlist()`.

And some "Place a space before left parenthesis, except in a function call." ;-)

@rempsyc
Copy link
Member

rempsyc commented May 4, 2024

(Dom I'm just secretly waiting for the meme signal to get to work 🤫)

@rempsyc rempsyc merged commit 201e979 into main May 7, 2024
22 of 27 checks passed
@rempsyc rempsyc deleted the report_loo_minor branch May 7, 2024 18:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants