11/25/2025
Clinical data rarely fits into a single format. A patient’s record spans images, lab values, procedure histories, vital-sign time series, and narrative notes, each capturing a different aspect of the clinical picture. Most AI systems still handle these inputs separately, which limits their ability to model how clinicians actually synthesize information.
The Holistic AI in Medicine framework approaches the problem differently. Instead of forcing all modalities into one predefined architecture, it uses specialized feature extractors for each data type and fuses those representations into a shared embedding. This separation of concerns makes the system adaptable and allows large-scale testing without redesigning the model for each new combination of inputs.
With more than fourteen thousand model runs, researchers map out how different medical tasks draw on different information sources. Imaging drives diagnostic performance. Longitudinal measurements shape operational forecasts such as mortality or length of stay. Some modalities strengthen results only when paired with others, and a few degrade performance when the extracted signal is weak. Shapley values, derived from cooperative game theory, make these interactions interpretable and highlight where redundancy adds stability.
The broader takeaway is that multimodality is not a slogan but a measurable advantage when aligned with the structure of clinical reasoning. The ability to test and understand these combinations systematically matters more than simply increasing data volume. As healthcare organizations adopt AI, frameworks like HAIM may help them deploy systems that can evolve with new sensors, new data streams, and the increasingly complex environments in which care is delivered.
Soenksen et al., Integrated multimodal artificial intelligence framework for healthcare applications. npj Digital Medicine (2022).
https://www.nature.com/articles/s41746-022-00689-4
Artificial intelligence (AI) systems hold great promise to improve healthcare over the next decades. Specifically, AI systems leveraging multiple data sources and input modalities are poised to become a viable method to deliver more accurate results and deployable pipelines across a wide range of ap...
