ArtEvoke

Abstract

This work explores how artificial intelligence can support non-drug interventions for people living with Alzheimer's disease through the use of visual stimuli and art. We present the development and integration of ArtEvoke, a prototype platform that combines large multimodal models (LMMs), large language models (LLMs), and embedding-based vector search to connect personal stories with curated artworks. Building upon foundational research from a Master's thesis at Université Catholique de Louvain, which benchmarked the core AI models, this work focuses on integrating these components into a functional prototype and introducing a novel clinical evaluation framework. Designed to assist memory recall and cognitive engagement, ArtEvoke generates personalized visual narratives or retrieves relevant paintings based on the user's story. A key contribution of this work is the platform's ability to support structured clinical sessions, enabling practitioners to capture and "score" patient responses to provide a metric for engagement. The prototype is intended for testing in future therapeutic and research settings as a scalable support tool. Our findings highlight the platform's significant potential, suggesting that after further scientific testing and validation, it could become one of the first AI and art-based tools to offer a scalable and measurable therapeutic strategy for dementia care.

Video

Framework

ArtEvoke integrates several cutting-edge technologies to create a comprehensive platform for connecting art with personal memories:

Large Multimodal Models (LMMs) for understanding visual and textual content
Large Language Models (LLMs) for story and description generation, as well as narrative creation
Vector Database for efficient storage and retrieval of artwork embeddings to enable semantic search
Embedding Models for semantic understanding of stories and artwork descriptions
Curated Art Databases including WikiArt, SemArt, and museum collections

The platform operates in two main modes: Memory Reconstruction and Art Exploration. In Memory Reconstruction, users provide a story or memory, and the AI divides it into meaningful sections, then searches for related artworks from curated databases. In Art Exploration, users enter keywords related to themes they want to explore, and the system finds matching artworks, then generates a personalized story connecting the chosen pieces.

Results

Memory Reconstruction

Tell us a story or memory, and our AI will divide it into meaningful sections. For each section, the system searches for related artworks from our curated databases, helping you visualize and recall your memories through art.

Example

The city towers glimmered above me like steel giants. I wandered toward the quiet outskirts where the noise softened. Sunlight spilled across open fields, turning everything warm and golden. Wildflowers brushed my legs as colors replaced concrete. And at last, a still blue lake opened before me.

The AI divides your story into sections and finds related images:

Section 1

"The city towers glimmered above me like steel giants. I wandered toward the quiet outskirts where the noise softened. Sunlight spilled across open fields, turning everything warm and golden."

Section 2

"Sunlight spilled across open fields, turning everything warm and golden. Wildflowers brushed my legs as colors replaced concrete. And at last, a still blue lake opened before me."

Art Exploration

Enter keywords related to themes you want to explore. The system finds matching artworks from our curated databases, and the AI generates a personalized story connecting your chosen pieces, creating a unique narrative journey through art.

Example

Keywords Entered:

garden children beautiful city

The system finds artworks related to your keywords:

Based on the selected images, the AI generates a story:

In the vibrant city of Paris, newcomers arrived in 1934, carrying dreams and stories from their homeland. Among them was La Verja, a spirited young artist whose eyes danced with the colors of her memories. As they settled into their cozy apartment, the city buzzed with life and excitement. La Verja found solace in painting, capturing the essence of Paris in every stroke. One sunny afternoon, she painted a lively scene of the Seine River, with boats gliding past the Eiffel Tower. Her artwork soon became a cherished piece in the neighborhood, bringing joy to many.

BibTeX

@article{silvaalvarenga2024artevoke, title={ArtEvoke: Art and Memory through AI - Supporting Non-Drug Interventions for Alzheimer's Disease}, author={Silva, Augusto B. V. and Alvarenga, Vinicius}, journal={}, year={2024}, note={Under review} }

Acknowledgement

This research is being developed at the University of São Paulo. The first author completed his master's thesis in the same theme at UCLouvain, Belgium, and is now continuing the project in Brazil. We thank all collaborators and institutions that have supported this work.

For questions and feedback, please contact Augusto B. V. Silva or Vinicius Alvarenga.