Remove "it's as if people are doing MCC" wording

[xiaoyingpu]

From Jan 27 meeting

• Emphasize that the smaller FDR (can be?) driven by the incentive structure
• Remove "people do MCC equivalence" wording
• How to interpret the participant FDR values:
  • Absolute value of FDR -> 2x larger than algorithms
  • Relative magnitude of FDR -> no reversal between nregion conditions
  • Solution: report more averages over 8 and 12 regions

Sections involved:

• Intro: need a bolded sentence defining this behavior
• Intro: "do people do MCC" as the first of the two research questions
• Experimental design (3.3): ... if people do the equivalence of B-H...
• Results (4.5): reporting on RQ1 -> Now report null result?
• Discussion (5.1): generalize to "realism in EDA evaluation, which includes incentives"
• TODO:Somehwere: explain why $FDR_{8-regions}$ > $FDR_{12-regions}$ means MCC-like/risk averse? behavior

The simulation may help explain?

Possible reasoning: "if people do nothing (raw strategy), $FDR_{8-regions}$ < $FDR_{12-regions}$", but this can be confounded: for $\delta = 0.6$, if you squint hard enough, $FDR_{8-regions}$ < $FDR_{12-regions}$ is also true

[xiaoyingpu]

From meeting 01-21-2021

• run more iterations
• If FDR:12-region is lower than FDR:8-region, they must be considering something else... MCC???
  • Do they have a mental alpha in their heads?
  • Maybe use !q -> !p framing; hypothesize that people are using p-raw, and disprove it
  • double check p-raw numbers
• better name for p_raw: uncorrected

Eli:

• Is there a single visualization that's "hurting" our average compared to p-raw

[xiaoyingpu]

If FDR:8 > FDR:12, the "people doing MCC in their heads" theory won't work

vis-insight-reliability/R/analysis-final-study.Rmd

Lines 274 to 298 in 7da387a

	data.raw.fdr <- df.bh %>%
	filter(method == "uncorrected") %>%
	group_by(nregions, trial) %>%
	summarise( TP = sum(!true & reject),
	FP = sum(true & reject),
	TN = sum(true & !reject),
	FN = sum(!true & !reject),
	.groups = "drop") %>%
	mutate(fdr = ifelse(FP*TP != 0, FP / (FP + TP), 0)) %>%
	rename(ntrials = nregions) %>%
	group_by(ntrials) %>%
	summarise(fdr = mean(fdr), .groups = "drop")
	data.bh.fdr <- df.bh %>%
	filter(method == "bh") %>%
	group_by(nregions, trial) %>%
	summarise( TP = sum(!true & reject),
	FP = sum(true & reject),
	TN = sum(true & !reject),
	FN = sum(!true & !reject),
	.groups = "drop") %>%
	mutate(fdr = ifelse(FP*TP != 0, FP / (FP + TP), 0)) %>%
	rename(ntrials = nregions) %>%
	group_by(ntrials) %>%
	summarise(fdr = mean(fdr), .groups = "drop")

• check df.bh
  • df.bh is generated by data-cleaning-final-study.Rmd
• Fixed data import 5355261

~~From stimuli data, the version in analysis-final-study.Rmd and p_value_payoffs_bh_ineq.Rmd are the same in terms of uncorrected p-values, but slightly different for the B-H version. Rabbit hole here I come ~~ Fixed

[abhsarma]

If FDR:8 > FDR:12, the "people doing MCC in their heads" theory won't work

![image](https://user-images.githubusercontent.com/8374573/105580230-17efdc00-5d59-11eb-8759-9891a365895b.png

Yeah this is problematic. Since the simulations don't show that on average uncorrected FDR:8 > FDR:12, did we just manage to simulate a quirky set of data points? I do remember checking for this to make sure that number of regions with p < 0.05 is greater when 12 regions are shown vs when 8 regions are shown so not sure why this happened. Maybe I accidentally changed the seed value?

@mjskay thoughts?

[abhsarma]

Can we still use the reasoning?

[mjskay]

This makes me wonder about the data generation process... how is p_raw giving such a lower FDR with 12 graphs? Like, it doesn't make any sense.

[xiaoyingpu]

Incentives and P(null)