Is That All There Is?

0:00

A recording of the finished version, from the perspective of the viewer

— PROJECT NAME

— ROLE

Story,

Direction

— DATE

March-April, 2025

For this assignment in my Experience Design class, we were given a miniature set of a graveyard inside a small coffin, and a mini projector. Our task was to tell some sort of story by projection mapping onto the buildings. Instead of projecting images and videos onto a flat screen, projection mapping allows us to wrap them around objects or buildings.

My group was interested in subverting the expectations of doing a scary story in the graveyard, but we still wanted to pick something thematically appropriate to the set.

I found the song “Is That All There Is?” by Peggy Lee, and loved the interesting contrast between the lightheartedness of the music and mood mixed with the macabre nature of the events described in the lyrics.

The coffin set, from the perspective of a viewer

The graveyard set, illuminated in purple with the moon illuminated in red

I noticed that each verse of the song corresponded nicely to different parts of the graveyard set. For the first verse, in which the singer’s house catches on fire, we could use the graveyard house. For the second verse of the circus, we could use the moon. For the third verse about falling in love, we could use the pair of statues. For the last verse about death we could use the gravestones.

Our process started by mapping out the set in the HeavyM program.

The features of the graveyard set, mapped out and represented by different colors

Once we had the set mapped out, we decided on the sorts of assets we needed for each section. The plan was to find or make the video assets we needed, then import them into HeavyM, position them in each mapped group, and warp them to look correct on the set.

Unfortunately, the version of HeavyM we were using didn’t support the audio timeline feature. We needed this feature since aligning the clips to the music mattered to our project. As a result, we came up with a work-around of editing everything as a video in Premiere. This was good because it gave us fine control over the timing of each clip. However, it had the downside that we lost the ability to easily contain videos within a region, and completely lost the ability to warp the videos to look better on the architecture. Given the short amount of time we had to complete this, this was a tradeoff we were willing to make. One of my teammates re-masked all of the regions in Premiere, and we just had to accept that the videos would look slightly warped when projected on the actual set.

We initially generated most of our assets using AI, but couldn’t quite get what we were looking for. We knew we wanted a girl dancing, and AI wasn’t consistent enough or in time with the music enough for our needs. We did keep some of the AI assets, such as the ones used in the circus scene, but chose to record an actual person dancing.

Our dancer, my friend Winnie Tsai, in an abandoned room

We didn’t have a lot of equipment, so we’re recording using a phone resting on a recycling bin. To make the dancer feel less awkward about dancing alone, we danced with her from behind the camera.

Since our plan at this point was still to bring the videos into HeavyM, we tried a few different ways to remove the background and create a transparent video. We couldn’t get the transparent video into the right format that was usable by our version of HeavyM, which was one of the reasons we switched to doing it all in Premiere. We used CapCut to remove the background and turn our dancer into a silhouette, then brought the video into Premiere for the rest of the editing.

Editing the video in Premiere

The emotional journey map of our experience

Playtesting with two guests

From playtesting, we learned a few important things:

1. Viewing angle is really important. To see everything that happens around different set pieces, it's best to be positioned directly behind the projector.

2. The projector will always get jostled, and will be a struggle to realign. The less we touch or move it, the better.

3. Some people wanted more movement and variety in the clips, some people wanted less.

As a result of this feedback, we wrote better instructions for a facilitator to guide guests into ideal viewing areas, and we adjusted the movement and variety to draw specific focus and highlight the emotional tone.

Overall, I'm happy we were able to create an experience that left our guests contemplative, pondering their own lives. I think we were successful in cultivating the mood we were going for, and in particular, I think the dancer worked really well as an emotional storyteller.

We also learned a few things that we would do differently in the future:

- Both Premiere Pro and HeavyM had their drawbacks. Using a more advanced version of HeavyM would be helpful in the future.

- We relied heavily on color changes and fades, so the lighting was not as varied and dynamic as we originally wanted. In the future, I'd like to experiment further with this.

- The projector kept becoming unaligned. We could have made everything in the mapping slightly smaller to help with this and avoid light leak, which would have made the whole thing look a bit cleaner.

A Projection-Mapped Musical Story