Why Multimodal?

• Multimodal interaction: A review, M. Turk
• Multimodal Data Fusion: An overview of methods, challenges, and prospects, Lahat et al.

Uniomodal Representation Learning

• Efficient Estimation of Word Representations in Vector Space, Mikolov et al.
• The Illustrated Word2vec, Jay Alammar
• Intuition and Use-cases of embeddings in NLP and beyond, Jay Alammar
• Very Deep Convolutional Networks for Large-Scale Image Recognition (VGGNet), Karen Simonyan, Andrew Zisserman
• CNN architectures for large-scale audio classification, Hershey et al.

Self-supervised Representation Learning

• Deep Clustering for Unsupervised Learning of Visual Features, Caron et al.
• A Simple Framework for Contrastive Learning of Visual Representations (SimCLR), Chen et al.
• A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends, Gui et al.
• wav2vec2.0: A framework for self-supervised learning of speech representations, Baevski et al.

Machine Learning

• Pattern Recognition and Machine Learning, C. Bishop
• Distributed Representations of Words and Phrases and their Compositionality, T. Mikolov, I. Sutskever, K. Chen, G. Corrado, J. Dean
• An Introduction to Variational Autoencoders, D. P. Kingma, M. Welling

Multimodal Learning

• Look, Listen and Learn, R. Arandjelović, A. Zisserman
• Look, Listen and Learn More: Design choices for deep audio embeddings, J. Cramer, H. Wu, J. Salamon, J. P. Bello
• Learning Transferable Visual Models From Natural Language Supervision (CLIP), Alec Radford et al.
• Learning Audio-Language Representations (CLAP), Andonian et al.
• Speech2Face: Learning the Face Behind a Voice, T. Wen et al.
• WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning, Krishna Srinivasan et al.

Speech and Audio

• Deep Clustering and Conventional Networks for Music Separation: Strong Together, J. R. Hershey et al.

Multimodal Learning: Newer domains

• Multimodal Biomedical AI, Julián N. Acosta, Guido J. Falcone, Pranav Rajpurkar, Eric J. Topol
• BIOSCAN-5M: A Multimodal Dataset for Insect Biodiversity, Zahra Gharaee, Scott C. Lowe, ZeMing Gong, Pablo Millan Arias, Nicholas Pellegrino, Austin T. Wang, Joakim Bruslund Haurum, Iuliia Zarubiieva, Lila Kari, Dirk Steinke, Graham W. Taylor, Paul Fieguth, Angel X. Chang

Datasets

• [MINST]
• [FMINST]
• [CIFAR]
• [CANDOR]
• [Coswara]
• [ImageNet]
• [WIT]
• [Google Audio Dataset]
• [Vox celeb dataset]