‘over the moon’ an xR music video

Netflix’s runaway animated success ‘Over The Moon’, commissioned an xR music video to showcase the film’s feature song, ‘Rocket to the Moon’. The music video featured a performance by Cathy Ang, the actress behind the voice of the main character Fei Fei, but needed to show the real Cathy set within the animated world of the film.

FRAY Studio was asked to join the xR team to build the multiple worlds and environments needed for this music video. We had to recreate the worlds and locations from the film in a way that would function in real-time and allow this xR shoot to be truly beautiful.

As well as the locations from the film and Cathy Ang’s performance, the video also needed a heavy dose of xR VFX magic from Savages. The technical and client process was managed by Boum Creative and produced by Trailer Park and Mirada Studios.

The time frame to achieve all of this was short. We had just 3 weeks from commissioning to filming. In that time we had to re-model the film locations for real-time, and texture, light and refine several XR scenes in collaboration with Savages, Boum Creative and the end client.

The logistics of working across multiple time zones were a continual challenge. FRAY Studio is based in the UK, Boum and Savages are on the East Coast of the US, while the client is based on the West Coast. Another challenge was that we knew we were unlikely to be able to reuse any of the 3D assets from the film due to both the nature of 3D modelling for feature animation and the time frame of getting approval for the assets to be released.

We were able to access high resolution frames from the film and some reference-3D geometry. This informed the early stages of modelling, helping us understand how the main performance areas would work within the virtual space. Objects close to the camera were created with high polygon counts and intricately detailed texturing, while objects that were in the background could be created out of just a few 100 polygons. This approach meant we were able to maintain the balance of aesthetic and real-time that xR requires.

The project was delivered in Unreal Engine, inside one Unreal project set-up with Source Control Management and carefully maintained by Boum Creative. This meant that FRAY Studio, Savages, Boum Creative and the clients could all collaborate throughout, ensuring that all the different jigsaw pieces fitted together as the project developed, rather than trying to merge the whole thing together at the end. This methodology allowed Savages to begin building and refining the project and cuing architecture from an early stage, ensuring smooth running come shoot-day.

As each model was finished in C4D, it was directly brought into Unreal to begin the xR world building, texturing and lighting, rather than waiting until all the models were finished. This parallel approach ensured that there was no bottle-neck in the process, and made sure everyone in the team had the time they needed.

We worked hard to achieve a materiality which would match the original film. To do this we had to throw away everything we normally do to try and make a scene look ‘real.’ This included things like painting over photo textures and using modified car paint material as the ‘water’ in the scene to give the solid but milky texture required.

A key part of the first scene was the creation of a real-time, dynamic weeping willow tree. Rather than buying an already modelled stock willow, we had to create it from scratch. This was because we needed to make sure we were only spending computing resources on the areas that could clearly be seen from the main camera positions. On its own the tree looked very one sided and unbalanced, but when slotted into the scene it fitted perfectly around all of the other elements. This approach also meant that we could create a gap in the willow curtain for Cathy Ang to perform in front of and trim individual branches away if they looked wrong from certain camera positions. We were also able to dynamically use the wind, on cue, to make the tree come to life.

Boum Creative worked with the creative team and the clients throughout the process to maintain the balance between creative ambition and what was achievable within the time frame. With all teams locked onto the same goal, we feel that ‘Rocket To The Moon’ beautifully showcases how much xR can accomplish.

Creative Team

Performer: Cathy Ang

xR VFX: Savages

xR World creation/sceneography: FRAY Studio

Director: Chris Riehl

Director of Photography: Pete Konczal

Producer: Trailer Park & Mirada Studios

Edit: Mirada Studios

xR Producer: Boum Creative