It is important to write down all the set-data and camera-data. In my opinion, the least you should do is write down the camera model, lens model and focal length, height of the sensor to the ground, tilt of the camera and distance from the sensor to the spot where the CGI will be inserted.
Lennart Vulto takes us through his process of shooting a shot for CGI integration, making photorealistic fish and integrating them into this live-action plate.
Some time ago, the art-director for a graduation film at The Netherlands Film Academy, approached me with the following dilemma: in the script there was a scene where the protagonist smashes a fish tank, leaving two black veiltail fishes flopping onto the ground. Because we are against animal suffering (and wanted to be able to say “no animals were harmed in the making of this film” at the end of the credits), this of course, could not be done for real.
Well, the smashing of the tank could be done without real fish, but the scripted close-up of the fish on the ground couldn’t. As the VFX supervisor for this project, it was me who had to solve this dilemma.
In this blog post I will take you through my process of shooting a shot for CGI integration, making photorealistic fish and integrating them into this live-action plate.
The shot you see above was created for the Dutch short film En zo geschiedde, 2020 (English: And So It Happened), directed by Lobke de Boer.
I was on set when we shot the plates for the fish. In pre-production the director, the cinematographer and I had discussed the blocking of the shot, and I had made an extensive plan for the plates I wanted to shoot. When it comes to integrating CGI into a live-action shot, I like to shoot the following plates:
Apart from shooting all these plates it is important to write down all the set-data and camera-data. In my opinion, the least you should do is write down the camera model, lens model and focal length (which in this case was a beautiful 35mm anamorphic), height of the sensor to the ground, tilt of the camera and distance from the sensor to the spot where the CGI will be inserted. Luckily, I got help from a camera assistant writing down all that data.
You probably have read this a thousand times, but when wanting to create something in the computer, especially something photorealistic, you always have to start researching your subject and with searching lots and lots of reference images. In my case I searched especially for shape and material references, since the lighting would be determined by the actual lighting on set.
Naturally, the shape and material also has to be oriented towards the actual living fish on set, but searching for additional references never hurts and is helpful in really trying to find detailed reference images from all angels.
All references were gathered in PureRef.
For the animation process we looked extensively at movement references of fish flopping on land as well (it was a fun afternoon), but more about that later.
After shooting, we knew exactly what the fish and the shot should look like, so it was time to start the modelling process. I started with creating a very basic blocky model and went on from there.
In my experience, you should always try to subdivide as little as possible while getting the primary shapes right, before jumping into the small details on a really high poly mesh.
Fortunately, sculpting the fish was not that hard of a task, especially because I decided against sculpting each and every scale on the fish, following the advice of Maarten Verhoeven. This decision was made under the more practical consideration that the fish wouldn’t be that big on screen, so adding the scales in the texturing stage would be sufficient and save a lot of valuable time to focus on other stuff. For a really high-end production I would advise sculpting in details and baking them, but in our case, with a pretty tight deadline, this approach was the way to go.
My general advice would be to always have a good idea of the blocking and the camera angle of a shot before investing large amounts of time into details which won’t even be visible and maybe even slow down your render later on.
Another practical consideration was to only model one fish, and to simply scale it a little to get a second variation. This would be sufficient because the second fish would be completely out of focus.
For the fins I knew in advance that I wanted to simulate them, so all I needed for this were planes in the shape of the fins. Veil tail fish aren’t named that way for no reason: their fins are extremely thin, like see-through veils. I decided to model these fins in Maya, rather than Zbrush, because in Maya I find it much faster and easier to load in a reference image, create a plane and trace the shape by manipulating points on only the x and y axis, but this is of course a personal preference.
After I was done I had a pretty high poly model of the body as well as the fins, so I loaded everything into Houdini and did a quick retopo, using the amazing Topobuild tool. I could also have used quad-draw in Maya, but since about a year, I am moving more and more towards Houdini as my main 3D software. The model is still quite high-poly, because I subdivided it again after retopo’ing, but because it was such a lightweight model, this was no problem.
As I am not very much into animation, I am glad to be surrounded by some very talented animators. I asked two classmates to help me out with the rigging and animation of the fish. For the rig all credits go to Florentijn Boot, who got into rigging in Maya especially for this project. Bram Vermaas provided us with some great tips for the rig.
The animation of the two fish was done in Maya by Winston van Graafeiland.
As I mentioned, we started by looking up reference videos of fish flopping on land. Videos like this.
At this point in the process the entire ‘fish tank throwing scene’ was unfortunately cut from the movie because it wasn’t essential for the development of the story. This meant that the fish shot we had been working on was cancelled. But since I was already quite far in the process of making the fish and had asked other artists to help, I decided to finish the shot anyway. I also felt I had the responsibility to the others to do so.
One advantage of the fact that it was no longer part of the movie, was that I had complete creative freedom over the shot. This meant the animation process was a very smooth back and forth between Winston and me, by which Winston would make a versions of the animation and I would give him feedback.
Besides, my job was to match the CG camera to the actual camera that was used for the shot, which was was quite easy, considering it was a static shot and I had written down all measurements.
During the animation process I had already started doing R&D for the fin simulation. I had never done a cloth simulation before, so I had to step out of my comfort zone and try something new. I am very glad I did, because I learned so much from it and really enjoyed doing it. I ended up using Houdini’s Vellum. Especially helpful for this task was SideFX’s Vellum Masterclass by Jeff Lait.
After a lot of tweaking of parameters (especially the mass, thickness, stretch and bend stiffness, compression stiffness and also the constraint stiffness) I finally reached a look I was satisfied with.
Once the animation of the body was done, I exported the alembic into Houdini where I ran the preconfigured vellum simulation on the fins.
This had to be tweaked again to fit the animation. If you really want to take a look at how I did the final cloth simulation just send me a message and I’m happy to share the hip-file.
Before I could dive into my texture software of preference, Substance Painter, I had to do the UVs. For this I used Maya, because in my opinion it has the best UV tools. I decided to give each “main-part” (e.g. body/eyes/top fin/side fins) an own UDIM, not only to have enough resolution, but also to increase the clarity. Maybe a bit overdoing it, but fortunately it wasn’t a very heavy scene to render.
Once my model had proper UVs, I loaded it into Substance Painter and after baking all maps I began making folders with masks for each part of the model that had a different material.
I then started with the basic colour and roughness, experimenting with some procedural smart materials as a foundation. Just as with the sculpting, I think that it is important that you first try to really get your basic materials right before delving into the small details or even the height.
Personally, I will only go more into detail and hand paint stuff like irregularities and surface imperfections once I’m satisfied with the foundation. For this part I often use real images which I take directly from Google or photobash and manipulate to my needs in Photoshop. Always remember to check your individual channels and tweak them separately.
When texturing, you have to again really look at your reference. Take your time to observe how surfaces look and how light reacts with them. Also try to think about what your model “has been through” and try to incorporate these “traces of use” in logical places in your textures.
For example, a used Second World War tank would probably have a lot of dirt on its caterpillar tracks and damaged edges (especially there where soldiers get in and out), it would probably also be paler on top because of the sun and have a lot of traces of battle, like bullet holes and so on. Luckily, my fish hasn’t fought in the Second World War, but it has its own history. Adding such details can really add a backstory to your model and bring it to life.
I decided to use Arnold as my render engine. I had access to the Arnold htoa plugin for Houdini, so I assembled everything within Houdini and plugged all exported texture maps from Substance into AiStandard Shaders. I did not change many shader parameters, apart from a few small tweaks. After four months of lighting in Houdini during my internship at BigHugFX in Munich, Germany, I was prepared for the worst, but sadly (or luckily) the HDRI I had captured on set was almost enough to light the 3D fish realistically.
I only had to add some rim lights on the two fish and a bit of bounce-fill. The plates we had shot of the reference fish, the grey ball and the Macbeth chart now came in handy for matching the lighting. For the shadows I created a separate HDRI.
When rendering I always like to render at least the following AOVs, besides the beauty:
And depending on the situation:
AOVs are, in my opinion, absolutely vital if you want to composite your CGI.
I also highly recommend rendering in ACES if you want to integrate CGI into a life-action plate. The Academy Color Encoding System (ACES) is a unifying colour management standard, which allows you to transform the colourspace of your renders and your live-action footage into a scene-referred colour science with a common (linear) starting point. Another advantage of ACES is its increased colour accuracy (due to its big range), which can help tremendously in achieving photorealism.
Lastly, but definitely not least, comes compositing. Here you really get the chance to blend your CGI into the live-action. The compositing software I used was NukeX.
One of the most important things when integrating CGI is to start with matching the colours and black values of your render with those of the hero plate. Ewoud Heidanus, our amazing compositing teacher, who was an enormous help with this project, taught me to do the following steps one after the other:
Applying these few simple nodes will make a tremendous difference in integration.
Another big help in this comp, was tweaking the specular AOV. The raw render was way too extreme, so I pulled all specular off my beauty render, matched them to the specular highlights of the plate and merged this back over my beauty. This way of working with AOVs can be an absolute lifesaver and often rescues you from having to render your entire shot again.
After the colours, blacks and specular highlights were matching, I had to defocus the right fish, to match the plate. After experimenting with a few different techniques, Ewoud Heidanus showed me a clever way of creating an anamorphic bokeh using the Convolve node, which lets you create custom filter effects by supplying your own filter image. His method was to transform a simple roto shape into the desired bokeh shape, which, in my case, was oval. Set the gain of your plate way up and then use the convolve node, with the roto shape as filter, and then merge this back over your plate.
When it comes to compositing you always have to keep the 80/20 rule in mind. This rule means that roughly 80% of your work is done in the first 20% of your time, but these last 20% of your work, to really get it to 100% takes up the remaining 80% of your time. It’s easy to be satisfied when reaching this 80%, but I think it is absolutely worth it to take your time for these last subtle fixes who elevate your final work.
Some small changes can make a big difference between photorealism and clearly CG. Some tips for achieving this goal:
Here’s a little breakdown of my comp:
Like I wrote at the beginning, I always try to incorporate interaction between something that was actually there on set and my CGI, to really merge it in there. For this shot I had the plate of the glass shards getting moved. I roto’ed one glass shard from this plate and moved and timed it just right to the jump of the fish in the foreground. I also modified the leaves of the aquarium plant on the left to slightly cover the fish in the background. This helps to enclose in your CGI into the scene, like in a sandwich.
At the end I added a subtle camera shake and vignette and then there was my final result.
Looking back on this project I learned so much. Creatively, as well as technically. There are some things I will definitely do differently next time, but that’s inherent to learning and this will probably/hopefully be the case for every project I will do in the future. So I want to end with a few takeaways, which I hope you will remember from this blog. They will probably sound like I’m a real expert, which isn’t true at all, but hey, maybe there is something that’s still new to you:
Find out more: Rookies 2020 submission | LinkedIn | Vimeo