Skip to content

Minicourse EDSA-JG04

How to carry out a meta-analysis of NIRS studies: from extracting effect sizes to investigating sources of variability 

Jessica Gemignani

with Judit Gervain

About this course

One approach to address issues of replicability of NIRS findings is to carry out meta-analyses of NIRS studies. A meta-analysis is a set of statistical tools that allow aggregation across existing experimental studies investigating the same research question, with the goal of revealing the average effect size of a phenomenon, quantifying cross-lab and cross-study variability and identifying potential moderating factors underlying variability. It has the advantage of pooling data over a larger sample size than typically possible in single studies, therefore licensing more robust or more general conclusions.  A meta-analysis can be conducted over various types of data but its application to NIRS studies has been limited so far. In this course, participants will learn how to conduct a NIRS meta-analysis from scratch: they will learn how to select the literature, extract effect sizes from the full NIRS dataset, run the statistical analyses to estimate the average effect size, estimate its robustness and investigate sources of variability in terms of both theoretically relevant as well as methodological moderators. Finally, quantitative metrics will be illustrated that allow us to investigate the extent to which publication bias and other questionable research practices are present in the selected literature. Sample R code will be provided to reproduce the statistical analyses. At the end of the course, participants will be able to run their own meta-analysis of NIRS studies.

Learning Outcomes

Participants will learn how to

  • extract effect sizes from NIRS data, both at the individual- and study-level
  • estimate the average effect size of a phenomenon of interest, and thus evaluate its applicability
  • investigate the impact of potential moderators on the effect size
  • investigate the extent to which publication bias and other questionable research practices are present in the selected literature

Course Plan

Level: Introductory

Pre-Requisites: None
Course Duration: 3 hours

Delivery Plan

Selection of literature, organization of dataset and extraction of effect sizes (1 hour)
Statistical analyses, with practical demonstration, carried out step by step with R markdown codes (1 hour)
Interpretation of results (1 hour)

Why enrol on this course?

As the volume of NIRS research in neuroscience has increased, so have the concerns about the reproducibility of experimental findings, as, in many research areas, published studies were found not to replicate reliably. The origin of this replication crisis is likely multifactorial and includes undocumented diversification in research methodology and data analysis techniques as well as nontransparent data inclusion and exclusion criteria, difficulty publishing null results, HARKing (hypothesizing after the results are known), p-hacking, and other dubious analysis and reporting practices. Because NIRS is a relatively new technique compared to others, standardization of research practices has only started recently, e.g., with systematic comparisons of hardware performance, preprocessing methods, and statistical analyses.

One approach that can quantify the degree of reproducibly of empirical findings is to carry out meta-analyses. A meta-analysis is a set of statistical methods that allow aggregating across existing (published and/or unpublished) experimental studies that investigated the same research question, with the goal of revealing the average effect size of a specific phenomenon, quantifying cross-lab variability and identifying potential moderating factors underlying variability. It has the advantage of pooling data over a larger sample size than typically possible in single studies, therefore licensing more robust or more general conclusions. A meta-analysis can be conducted over various types of data but its application to NIRS studies has been limited so far.

The main goal of this mini course is to provide the audience with a toolkit to carry out a meta-analysis of NIRS studies. In particular, the mini-course will guide the audience through all the necessary steps: selecting the literature of interest, organizing the collected dataset, choosing the appropriate effect size based on the experimental designs, and setting up the statistical analysis. The audience will be guided through the R code relying on the metafor and lmer packages that allow to a) extract effect sizes from NIRS data, both at the individual level and at the study level b) estimate average effect size c) investigate the impact of potential moderators of the effect size d) investigate the extent to which publication bias and other questionable research practices are present in the selected literature, through ad-hoc quantitative metrics. Software options other than R will also be acknowledged. Sample R code will be provided to reproduce the statistical analyses. The methodological choices that need to be made during the analysis and the interpretation of the results will also be discussed.