Handcrafting a Stylised VR Film Experience

Handcrafting a Stylised VR Film Experience

In this article, Nick Rutten discusses his technical and directorial approach to creating a sylised VR film.

Originally posted on the Substance 3D Magazine.

The VR experience Dream Droplets achieved runner-up positions in two of the Immersive Media categories of The Rookie Awards 2021, as well as gaining festival selections at the Imagine, Kaboom, Cinekid, Dutch Animation (NFF) film festivals, and more. Nick Rutten filled almost every role in the creation of the film, notably including those of director and head of all 3D creation. The team at Substance 3D were fortunate enough to discuss his motivations and processes in creating Dream Droplets.

Hi, I’m Nick Rutten. I studied 3D animation at the Utrecht School of the Arts (HKU), where I live. My passion for creating fictional worlds is what led me to pursue this education, and during this time I became really interested in stylized 3D art. The creation of Dream Droplets allowed me to funnel everything I learned during these four years, as well as many things that were still completely new to me, into this film.

A Stylised Look in 3D

For Dream Droplets, I wanted to create a very stylised look for the film, as I feel that the suspension of disbelief reaches a little further with stylised animation. Moreover, this film in particular is all about daydreaming; it concerns the internal world of the main character, Kim, and how her emotional journey is reflected – and so I felt that a stylised look would help the audience experience this journey along with her.

More and more people are interested in creating stylised artwork in 3D, I find. With graphics and creative software improving so rapidly, realistic 3D has reached an impressive height. Possibility for development and improvement exists, for sure – but that improvement will come in small gradations at this point, rather than in sweeping changes. With realistic 3D being such a proven field already, I feel that a lot of 3D creators are becoming more tempted to find out what else is possible. And this stylised kind of feel has worked really well with animation in the past.

Stylised artwork has allowed me to communicate in ways that would not have been possible if I’d worked in a realistic art style. I feel that this is a wider trend – one that I hope will continue to evolve, resulting in many great works being created, and instilling a sense of wonder in audiences!

Directing for VR

In gaming, what I love most is to explore worlds. I love the ambiance of them, I love the visual storytelling within them, and I love to feel like I’m taking part, and discovering difficult-to-reach places in that world. In my work I try to use such elements to immerse the audience as well, and VR is a medium that really allows full immersion. The audience can experience the wonders of the worlds as if they are present within them, and can relate to the stories that take place within them in very interesting ways.

As with any visual medium, you should consider how the scene itself can be indicative of the feeling or the message that you want to convey. Here, you’re experiencing the journey of the main character, Kim. The rough blue hand-painted look in the atmospheric distance is reflective of Kim’s mood – because it’s her journey; that’s how she’s experiencing these scenes. Nearer to the shrine, the stylised feel of everything is more detailed, and much more hopeful; one reason for this is to convey Kim’s change of mood. This is another advantage with stylised art; because I didn’t focus on making everything look super-realistic, I had a wider range of options in how to communicate visually with the audience.

This change of tone near the shrine is also specifically to direct the eye of the audience. In a 360-degree VR film, the viewer can look any way at any time, and so I had to make sure that people were looking at the correct places. And use of lighting is one way of doing that; here, I used my hand-painted lighting to guide the eye, while making things foggy and less detailed where the viewer shouldn’t really be looking. The scene itself can help to direct the viewer’s attention as needed, as well. In the scene with the staircase, for instance, the staircase leads up, and the two sides of the staircase converge at the focal points of the scene: the cat sitting at the shrine, and Kim approaching.

Because VR is a fairly new medium, in comparison with more conventional filmmaking, there’s still a lot to learn and explore when it comes to directing in VR. The field is just too new to learn everything about it in tutorials or books. And so I really had to discover almost everything through trial and error. I was able to assemble a large pool of teachers and students at HKU, and VR enthusiasts online, as a trial audience. I’d put the headset on them at various stages just to check, ‘Does this work, or not?’ Sometimes it didn’t, and I’d have to adjust what I was doing.

A closeup of a leaf, handpainted in Painter.

In VR films the viewer is typically some sort of floating entity; here, I really tried to get the point across that you’re seeing everything from the perspective of a drop of water, to ground you in this world. What really works well in VR is scale, so I thought it would be cool to do the ‘Alice in Wonderland’ transition, so that you experience shrinking to the size of a water droplet, and see the world around you grow. The leaves and grass around you seem like they scale to the size of a house.

Handcrafted 3D

During my Animation studies I became interested in handcrafting ways of doing 3D animation – in my time at HKU I started picking up tools that would allow me to handcraft 3D, such as Photoshop, Quill, ZBrush, and the various Substance tools.

I explored many ways of creating a handcrafted look. One study was digital claymation in ZBrush. I basically sculpted every phase of this water-creature by hand. My respect for stop-motion artist has grown significantly, since they do this process without using Ctrl-Z, haha. This technique allowed me to surpass the unconscious association people have with 3D animation. Compared to traditional 2D animation, where you can animate shapes as freely as you can draw, conventional digital 3D animation is kind of bound by rigs, and people associate that with less shape fluidity. But this technique allowed me free-flowing and controlled geometry, as I can refresh all polygon resolution per phase; this, along with the stylised V-Ray shaders I used, and hand painted textures with Adobe and Substance, causes a lot of people to ask me if I used 2D animation for this study.

It’s difficult to get a handcrafted look in 3D, you really must swim against the current of the industry standard, realistic PBR. Also, in handcrafted works you usually have loose breakup in the silhouettes – whereas in 3D you work with geometry, which is very hard.

My approach to this was to paint geometry in flowing ways with Quill. This removes you from that harsh 3D look. At the same time, I tried to utilise the graphical feel of 3D to my advantage and approach the art in the same way that digital painters or graphical artists might approach it. The works of Bastien Grivet, Zudarts Lee, Yun Ling, and Studio Ghibli were a big inspiration for me during the development of the art style.

I used Painter, formerly Substance Painter, to texture some scenes. It was really helpful, because it allowed me to load my whole scene, and to paint in all the lighting and texturing to get the exact look I needed. That is, this wasn’t the ‘classic’ PBR workflow; here, I painted on the light bounce and the shadows precisely where I needed them. I essentially painted my 3D mesh exactly as I would a 2D canvas – but a canvas that you can then freely move through in 3D space afterwards.

A tip I discovered to help with this style of hand-painting: if you have camera-based mapping, the ‘Backface culling’ setting has a default value of 90. The slider allows you to set it to a minimum of 45 – but you can also manually enter a value as absurdly low as 1. If you do this, it will create a sort of automatic mask over the geometry. You’ll find you can paint quickly with with a lot of texture, and you won’t accidentally paint onto other parts of your geometry. You can really hand-paint cleanly, that way.

I do have a request for a feature that it would be fantastic to see in Painter: it would be so great if the development team could create a lasso tool for masking. Quick lassos combined with camera-angle based masking would be a dream come true for hand-painting.

Animating the scene in Maya.

I took a similar approach with the rice field – the majority of the texturing was hand-painted in Painter, with some additions in Quill to break up the silhouette. I also hand-painted all the rocks in the scene in Painter, to achieve a dynamic feel to the sunlight creeping in through the rocks, and to the bioluminescent water which becomes more concentrated as it flows further and further through each layer of the field.

I chose to paint the whole scene at the same time, instead of painting individual assets, because this allowed me to create this dynamic feeling of light within the project’s art style. When you’re trying to ensure the art and lighting is consistent in a large environment, if your lighting isn’t uniform then it can be very annoying to check if everything matches up. Conversely, if you paint a whole scene, you can just paint light across all the assets at once. Like this, it never mismatches, and you can also see immediately if the art is consistent as well.

More about Dream Droplets

The complete VR experience is not available to the public just yet, but you can learn more about the project by heading over to the Dream Droplets webpage; you can contact me for more information or add yourself to the mailing list to get more information about screening opportunities there as well.

Nick Rutten is a graduate of the Utrecht School of the Arts (HKU), Nick has always been inspired by whimsical, animated worlds. He now uses new technologies to craft such worlds himself, most recently through his work at the 3D animation studio Colorbleed.