Towards a foundation model for universal representation of Neuroimaging data

Proposal Abstract

A foundation model is a large deep learning model trained on a vast quantity of unlabeled data at scale resulting in a model that can be adapted to a wide range of downstream tasks (classification, regression, segmentation, etc). Such model would allow to avoid development of the new model for the task from scratch, facilitate robustness and accelerate analysis on small neuroimaging studies.

About the Project

Umbrella Project

• NeuroNeural

Emphasis:

• Research & Development & Engineering

Expected Background

• Undergraduate friendly. Familiarity with python, PyTorch and basics of Machine Learning (Supervised & Unsupervised Learning). Linear algebra and intro statistics recommended.

Primary Point of Contact

• Alex Fedorov ([email protected])

Supervisor

• Vince D. Calhoun ([email protected]), Sergey M. Plis ([email protected])

References and External Resources

• Fusion: https://github.com/Entodi/fusion (by invite only currently)
• Representation Learning: A Review and New Perspectives https://arxiv.org/abs/1206.5538
• Multi-modal learning: https://cmu-multicomp-lab.github.io/mmml-course/fall2020/readings/
• Awesome self-supervised learning: https://github.com/jason718/awesome-self-supervised-learning
• Awesome multimodal learning https://github.com/pliang279/awesome-multimodal-ml
• Self-supervised learning: The dark matter of intelligence https://ai.facebook.com/blog/self-supervised-learning-the-dark-matter-of-intelligence/ ​* ​Deep Generative Modeling https://link.springer.com/book/10.1007/978-3-030-93158-2
• On the Opportunities and Risks of Foundation Models https://arxiv.org/abs/2108.07258
• Prompt tuning https://github.com/thunlp/PromptPapers

Estimated Timelines

• 2-4 months (summer project / semester project)

Possible Deliverables

• Implementation of self-supervised and generative models (at least one model) with application to Neuroimaging data (unimodal & multimodal, mostly structural volumetric data, but can be fMRI, DTI, etc.)
• Contribution to open-source neuroimaging package (Currently, it is fusion)
• Exhaustive evaluation: comparison with baselines, data-efficiency, transfer learning, out-of-distribution generalization, fairness, interpretability, harmonization, underspecification
• Submission to Machine Learning & Deep Learning & Medical Imaging Workshops (NeurIPS, ICLR, ICML, CVPR, ECCV, ICCV, MICCAI)
• Submission to Machine Learning & Deep Learning & Medical Imaging Conferences (e.g. IEEE, NeurIPS, ICLR, ICML, MICCAI, MIDL, CVPR, ECCV, ICCV)
• Submission to Neuroimage & Nature journals