Resources
Tools/Demos:
False Alarm Simulator
Adding more participants after your p-value is close to being significant is tempting (for example, if you run 20 subjects and have a p-value of .15, you might be tempted to run 10 more and test for significance again), but this practice results in false-positives above the standard .05 rate. This simulator (which simulates the results of a null effect) shows you your overall false alarm rate if you supplement your original dataset.
Power-Sim Toolbox
Matlab scripts for resampling pilot data to estimate statistical power for combinations of different sample sizes and study durations, an approach that accounts for both sampling and measurement error. Useful if you have pilot data and want to optimize your design for main data collection, or have an analysis plan incompatible with other power estimation tools.
Presentations:
Statistical Power: Power's Role in the Replication Crisis, and Justifying Your Sample Size
A presentation on A) how underpowered studies can lead to failed replications, even for real effects, and B) good (and bad) ways to go about determining sample sizes for your studies.
Using Simulation to Guide Your Research
Simulation is powerful tool for research, as it can be used for guiding experimental design, generating hypotheses, understanding results, determining sample size, etc. An overview of how I've used simulation in my own research.
Tools/Demos:
False Alarm Simulator
Adding more participants after your p-value is close to being significant is tempting (for example, if you run 20 subjects and have a p-value of .15, you might be tempted to run 10 more and test for significance again), but this practice results in false-positives above the standard .05 rate. This simulator (which simulates the results of a null effect) shows you your overall false alarm rate if you supplement your original dataset.
Power-Sim Toolbox
Matlab scripts for resampling pilot data to estimate statistical power for combinations of different sample sizes and study durations, an approach that accounts for both sampling and measurement error. Useful if you have pilot data and want to optimize your design for main data collection, or have an analysis plan incompatible with other power estimation tools.
Presentations:
Statistical Power: Power's Role in the Replication Crisis, and Justifying Your Sample Size
A presentation on A) how underpowered studies can lead to failed replications, even for real effects, and B) good (and bad) ways to go about determining sample sizes for your studies.
Using Simulation to Guide Your Research
Simulation is powerful tool for research, as it can be used for guiding experimental design, generating hypotheses, understanding results, determining sample size, etc. An overview of how I've used simulation in my own research.