Helen (Hairong) He

Computer Science + East Asian Studies & Classics @ Harvard | AI/ML Researcher committed to bridging the arts and technology

Bio

With training in both Computer Science and Classical Literature at Harvard, my research goal is to use artificial intelligence to enhance human interaction with art and culture through immersive, multisensory experiences. My work has evolved from NLP-assisted text mining and creative writing to 2D data visualizations and 3D graphics and animation powered by machine learning.

Informed by a background in classical languages and literature, my initial area of interest is text mining and visualization with NLP. My senior thesis project will be a quantitative comparative visualization of the use of masks across the corpora of Ancient Greek, Chinese, and Japanese theater. I have also done various projects experimenting with LLM co-authored creative writing.

As I developed an interest in the history of writing, art, and museums, I started to work on interactive data visualization augmented by Computer Vision. As I experimented with the Harvard Art Museum’s API of 250,000+ objects, I was fascinated by how computers can help us “look” at images at scale and spot trends and patterns, creating rich datasets for visualization and interpretation.

My most recent area of interest is machine learning in 3D data. My summer research applying ML algorithms to reimagine medieval Chinese Dunhuang dance based on static ancient mural images of dancers made me realize how multisensory and multimedia storytelling could viscerally reshape how we imagine our cultural pasts.

Resume

haironghe03

Helen He LinkedIn

My recent talk

My three years of experience on art and data at the Museum of Fine Arts, Boston and beyond

us02web.zoom.us

Research

Cave Dance - ML-Empowered Reimagination of Medieval Chinese Dance

Cave Dance 2.0

Google Slides

[In Prep for Submission End of 2025]

How shall we reimagine a 1300+ years old Dunhuang dance culture from 500+ images of murals in the Dunhuang Caves?

I am working with Professor Yilun Du from Harvard Kempner AI Institute, Professor William T. Freeman from MIT CSAIL, and Professor Eugene Wang at Harvard CAMLab to launch version 2.0 of an installation reimagining medieval Dunhuang dance culture.

I converted 2D mural images of 36 dance poses into 3D animated datasets of a Bodhisattva sculpture performing these poses, leveraging its 3D scan from Harvard Art Museums and motion capture data of professional dancers.

I refined and mapped a 3D model of a Bodhisattva sculpture onto the dancer motion capture data, working with BVH, OBJ, and FBX formats in Cinema 4D for 3D character design.

I enhanced the motion in-betweening algorithm from Slerp to deep learning-based motion in-betweening to visualize how dancers a thousand years ago could have moved their bodies between the distinct poses in the still mural images from the Dunhuang Caves, thus empowering the model to choreograph. I am currently tuning the model to accept multisensory inputs such as textual records of the dance and music.

Publication in progress

Story Ribbons - Reimagining Storyline Visualizations with LLMs

[IEEE VIS25] Best Paper Honorable Mention

As a computer scientist, how do we visualize a 500-page novel on a 2D screen? As readers, how can we better keep track of interactions between characters, locations, and themes and gain new insights into familiar stories?

In collaboration with Catherine Yeh, Prof. Martin Wattenberg, and Prof. Fernanda Viégas in the Insight + Interaction Lab at Harvard, we built an LLM-driven data parsing pipeline and narrative (fiction/theater/film/poetry) visualization system, helping readers delve into the interactions between characters, locations, and themes and gain new insights about the story.

Specifically, I suggested design improvements to enhance interactivity and interpretability, especially with the character network feature. I also built a semi-automated workflow to benchmark LLM character/theme recognition against LitCharts and SparkNotes across texts (Odyssey, Don Quixote, Pygmalion), evaluating the strengths and biases of LLM vs. traditional learning sources. I also had great fun conducting user studies with literary scholars for the system.

I am still working on expanding the data parsing pipeline into non-English texts, currently troubleshooting multilingual challenges for parsing Classical Chinese and French with The Peony Pavilion and Le Petit Prince.

This work received the Best Paper Honorable Mention at the IEEE Visualization Conference 2026.

Story Ribbons: Reimagining Storyline Visualizations with Large...

Analyzing literature involves tracking interactions between characters, locations, and themes. Visualization has the potential to facilitate the mapping and analysis of these

arxiv.org

IMLS Grant - HCI/Mixed Reality Study of 3D Data Curation

Experiment Interface. Courtesy: Mentor M.C.'s website

[iConference 2026 Submitted]

How can we make 3D data citable? Can immersive technology aid 3D data curation?

I am working on a research project with Mr. Matthew Cook from Harvard Library and Prof. Zack Lischer-Katz from the University of Arizona on a project on 3D data creation and curation in academic contexts, funded by an Institute of Museum and Library Services grant.

We have interviewed and carried out experiments with 30 participants (faculty/researchers) from 6 universities across the US who frequently generate, process, or use 3D data in their work, ranging from microbiologists to Ancient Egyptologists. I went on two of the group's research trips to The University of Oklahoma (November 2024), The University of Arizona (February 2025), and Harvard University (September 2024) to conduct interviews and AR/VR HCI experiments with more than a dozen academic professionals regarding their practices in the life cycle of 3D data.

I performed quantitative data analysis for the gathered data across 6 sites and 30+ participants, developing a program to compute the cognitive load and response accuracy for 3D model quality control shown in iPad versus VR environments. I built an automated pipeline to process video and audio recordings of these experiments.

Publication in progress in Fall 2025.

With the same group, I am also working on a project guided by Professor Peter Der Manuelian on the Giza project, building a 3D asset pipeline in Python to generate and refine models of artifacts from written archaeological records, using Meshy AI’s API, and placing the models into an Ancient Egyptian virtual environment in Unity.

Research trip to Arizona (2025/02)

LLM in low-resource data -- Harvard Kempner AI Insititute

LLM in low-resource, high-similarity languages

How well does GPT-4 perform when translating from English to Scots, Spanish to Ladino, Ancient to Modern Greek, or Mandarin to Classical Chinese?

What happens when the model struggles, losing steam and reverting to the source language, resorting to a higher-resource third language in between, or trying to imitate/caricature the language in stereotypical ways?

Advised by computer scientists Dr. David Alvarez-Melis and Dr. Naomi Saphra and in collaboration with 5+ linguists, this project evaluates and seeks to understand the performance of LLMs in translating a high-resource to a low-resource language when they are highly similar. Tasks like this, which push the limits of the model's capacity, are uniquely valuable for researching the heuristics and shortcuts the model employs.

KURE Undergraduates Celebrate Completion of Spring Research Projects - Kemp

Harvard undergraduate students discuss their research projects related to natural and artificial intelligence at KURE luncheon.

kempnerinstitute.harvard.edu

Work: AI for Museums

Palace of Versailles

AI for Collections Website

Google Slides

How can a 400-year-old palace use AI to make its collections website more engaging and inviting to a diverse range of modern audiences today?

I delivered a proposal for integrating AI (especially Computer Vision) applications into Versailles’ new collections website, which will launch in 2026, based on technical case studies of six world-top museums’ collections websites’ AI functions tailored to the collections history of the Palace.

I gave a one-hour presentation and defense of my proposals to professionals interested in AI from across the Palace.

Museum of Fine Arts, Boston

3D Imaging in Conservation

I captured and processed 3D models of three types of art objects (Japanese Noh masks, Japanese Buddhist sculptures, and an Ancient Egyptian falcon) using the techniques of photogrammetry and structured light scanning. I rendered high-fidelity models with Agisoft Metashape and Artec Studio and documented metadata. These models are used for conservation

I built a sharable web viewer for a Reflectance Transformation Imaging (RTI) model of MFA’s Donatello marble relief, the only Donatello relief in North America. Otherwise, models produced by this conservation imaging technology are impossible for the public to access because it requires highly specialized software to view. The object conservator at the MFA is presenting at the annual conference of the Renaissance Society of America.

As a member of the 3D Working Group at the MFA, I helped create a survey that aims to understand 3D imaging practices and related challenges faced by museums around the world. I independently performed data analysis and visualization of the 86 responses (39 complete and 47 incomplete), gave a presentation, and suggested best museum practices for archiving, using, and distributing 3D data based on research.

Word Cloud based on the survey

Object Stories - Harvard Art Museums

ham-object-stories-ce339cea7ce5.herokuapp.com

Object Stories Website

Want a new spin/take on traditional museum labels? Step into a world where art comes alive!

This project transforms factual data of the 250,000 objects in the museums’ API into creative and immersive object stories co-authored by humans, LLMs, and Computer Vision models analyzing eye gaze direction.

Steps:

Wrote creative story templates about artworks
Fill in the blanks in the templates with
1. Factual object data from the museums’ API
2. LLM-generated text trained on creative writing + object data
3. CV tags of the object images (including eye gaze direction)

Web Dev framework:

Node.js
Bootstrap
Handlebars

Mentor: Jeff Steward

GitHub - Object Stories

Harvard Art Museums

AI Sheet Music Cataloger - Harvard Houghton Library

It takes a Houghton Library Music Cataloger 2 hours to catalog a 19th- or 20th-century sheet music volume. It takes 2 min for an AI program.

I developed a program that takes photos of the title pages in a volume, analyzes and transcribes key cataloging information from the images, and generates a CSV file.

This efficiency would give us time and energy to do justice to hundreds of boxes of uncataloged and undigitized treasures sitting in the library.

Tools:

Images (taken by my phone in Google Drive)
Python
OpenAI API (model: GPT-4o)

Sheet Music Title Page

Code on Google Colab

Projects

Civilizations Lost: GenAI for Storytelling

Civilizations Lost

What would be the folktale of a lost civilization that was left out of the canon, with its language and records lost to time?

Adapting LLMs to craft engaging narratives in specialized domains for creative expression with RAG, this project builds a folktale storytelling engine in Python with the OpenAI Assistants API and Midjourney to create and publish an illustrated collection of AI-generated folktales from eight lost civilizations.

The civilizations included range from the Rising Star Cave people (c. 348,000 - c. 248,000 BCE) to Roanoke (c. 1600 CE).

Mentor: Dr. Jeffery Schnapp

Data Visualization

Family Reunion - Harvard Art Museums

November 16, 2024 marks 10 years of the current Harvard Art Museums’ building being open to the public. 10 years of ~8100 objects moved through the special exhibition gallery spaces on Floor 3. In a “family reunion,” what stories will the objects and exhibits tell each other?

The size of each square represents the number of objects on display in that exhibition.

Tools:

D3.js
The Museums’ API (object, exhibition description & poster data)
spaCy and LLMs (for processing exhibition descriptions)

Mentor: Jeff Steward

oppr.org

Family Reunion

GitHub - Family Reunion

Who Speaks Up - Gender Dynamics in Ancient Greek Drama

oppr.org

Who Speaks Up

How do gender dynamics in society permeate art and performance in Ancient Greece?

From a data perspective, we explore all extant Ancient Greek plays to quantify how many words in each play are spoken by female characters, and how many by males. Many plays bear the names of women—Antigone, Medea, Lysistrata … but how much is the female voice actually heard?

All plays are downloaded from Perseus Digital Library and processed in the original language with The Classical Language Toolkit.

Data mining:

Python

Visualization:

p5.js
D3.js

Roman Amphitheaters: 70BC to 280 AD

oppr.org

Roman Amphitheaters: 70 BC to 280 AD

This project aims to capture the synchrony between the empire's expansion and the construction of amphitheaters.

By tracing the distribution of newly built theaters, we are also tracing the evolving boundaries of the Roman Empire at a time of expansion.

The viewer is encouraged to cross-reference the visualization and maps of the Roman Empire such as in this video: https://www.youtube.com/watch?v=GylVIyK6voU

Dataset:

https://github.com/roman-amphitheaters/roman-amphitheaters/blob/main/README.md

Visualization:

Small multiples map created with Plotly library in Python, exported to JSON, and loaded JSON in p5.js.

roman theater.py

Data Art with p5.js

3D Modeling & Unity

Buddhist Architecture to Life: Modeling the Dragon and Tiger Pagoda

3D Pagoda

My teammate and I built a 3D model of an 8th-century Chinese pagoda "The Dragon and Tiger Pagoda" 龍虎塔 I studied in art history, and constructed an interactive historical scene around it.

The work deepened my understanding of the architectural program and provided an immersive educational tool to engage with the pagoda without visiting the physical site.

Tools:

SketchUp
Unity

Dragon and Tiger Pagoda

SketchUp & Unity

NLP-empowered Greek Lyric Poetry

How can we enable anyone, regardless of their language background, to engage with Greek lyric poetry in the original language as thoroughly as possible?

I used NLP to develop a version of Sappho’s poetry that includes the identification of dialect-specific variations, grammar annotations, treebanks, and word alignment between the original Greek and its English/Mandarin translations.

I worked with the digital classics library Perseus to incorporate new classical texts into their database Beyond Translation.

Project Files

Cognitive Neuroscience/Neuroaesthetics Research

A Dive into the Reading Brain

Research Paper: "A Delve into the Reading Brain: Multisensory Experience of the Material Book"

This cognitive neuroscience project examines the neurobiological underpinnings of the multisensory experience of reading and handling a historical manuscript – not only the visual, but also the auditory, olfactory, and tactile experience.

Multisensory Experience of the Material Book