Experiential Design and Creating an Immersive Experience

Meet Young Jeen Ren, also known as Jay. Jay is a graduate student from The One Academy of Communication Design, majoring in BA(Hons) Digital Media Design, with a specialisation in experiential design and motion design. Jay possesses a drive to push boundaries, which he applies both in his personal life and creative endeavours, generating ideas and concepts that explore new frontiers. In this article he shares details of his latest project with a focus on experiential design and creating an immersive experience.

Introduction: The Beginning of A Long Journey

A little bit of intro about my work - Caustic. This is a 10 month long self-directed project. During early ideation stages, I thought of creating an experience with water.

The Goal: to create an experience that no one has ever experience and push boundaries in experiential design.

A lot of thoughts and discussion were being thrown at the start. Jesus simulator? Walking on water? A horror experience? The concept was blurry and yet to justify the use of water element. It was a challenging process because nailing down the concept is an important step to lay down the foundation of the entire project. To figure out the concept, I experimented with the interaction between water and lights (refer Fig.1).

Before I start diving (pun intended) deep into the water project, I would like to shout out to all the lecturers and people who are involved in this project.

Lecturer: Cheang Lin Yew, Jeremy Phun, Ronald Chua
Model: Elaine Chew, Danielle Chong
Media Artist: Futuresigner

Concept & Objective : Why Water?

Using real water to communicate emotions. Water is like emotion, as it flows gently, it represents calmness; vice versa, water represent excitement when it is rough.

Caustic(refer Fig.2) - Bright filaments patterns caused by light reflection from curved surfaces (Nolte, 2021).

Caustic is the visualisation of the emotions, different shapes and forms of the caustic is formed by waves. These different shapes and forms invoke different emotions within you. This experience is about embracing the calm & chaos within yourself. Regardless of the positive or negative emotion we feel, we should learn to embrace both of them.

Project Objective: Create an experience to make people feel different emotions and embracing it, could be calm, exciting or anxiety.

In short, using real water to communicate emotions, and the caustic patterns are the visualisation of the water and the emotion. Embracing the emotions you fell in the experience regardless if positive or negative.

Interactive Experience: The Interaction

Imagine there is a dark pool of water in front of you, you took a deep step in. As your first step into the water, the whole pool lit up. You start walking around the pool, you enjoy the calm visuals that ooze out of your feet. Then you look up, mesmerised by the caustic on the wall (refer fig.4). When you walk fast, the whole visual changes, from the seemingly calm visuals to total chaos. The sound changes too, from soothing and calm to chaos and exciting.

Different visual states are triggered by walking speed. The faster you walk, the visual state changes to more "exciting" or "anxious" visuals; the slower you walk, the visual changes to a calming visual. In the idle state, pulsing lights and water drop SFX illuminates the pool to intrigue people to step in and experience.

The water drop SFX was inspired by the show Demon Slayer, where the character "Giyu" used his moves called "Dead Calm".

To sum up, the interaction is simple, it is triggered by movement speed. Instead, most of the attention is placed on the visual aesthetic. Visuals are produced using GLSL programming, which will explained deeper in the next segment.

Visual Design: Creative Coding and Challenges Faced

The process was tough, as I had only a month to learn the basics of GLSL programming. It was a steep learning curve, but it eventually paid off. There is no playbook here; I had to constantly experiment and develop the visuals throughout almost the entire project.

The visuals were crafted through GLSL programming, known as "creative coding." Yes, it was one of the biggest challenges in the project. I experimented with different styles and methods, such as "psychedelic visuals," liquid simulation, abstract patterns, fireflies, glowing orbs, and many more (refer to Fig. 4 and 5).

An initial interaction idea (refer to Fig. 6) involved having an orb circling around you, which would morph with other orbs when you stood close to someone else. This idea was eventually scrapped due to time constraints and technical difficulties.

Early experimentation included Processing and After Effects, but it wasn't working well due to my inexperience and lack of exposure. Six months passed, and the visuals and interaction still weren't up to par. My lecturers pushed me even further, encouraging me to explore shader programming such as GLSL and other plugins.

The liquid motion is created using a shader plugin, which is used to produce the liquid motion in the experience. GLSL programming was used to generate supporting visuals, ensuring that each state (calm and anxious) has at least 2-3 sets of visuals.

An important thing to note is that we should constantly test visuals in the pool. Visuals can look very different on a screen compared to projecting on water.

The next step is implementing the visuals into the program, triggering specific visuals at different states.

Technical Aspects: The Backbone of The Project

Arduino and Processing are the backbone of the interaction.

Remember the idle state you saw in the walkthrough video? How do I make the light go on and off? Arduino. Processing detects whether there is someone in the pool through LiDAR, then sends the data through serial communication to the Arduino, telling it to turn off (active mode) or turn on (idle mode).

Processing: Java-based programming software for artists and designers to do creative coding.

Arduino IDE: C++ programming software to program microcontrollers like Arduino Nano.

OpenCV: Real-time computer vision programming library.

Processing acts as the brain behind the whole operation, controlling which visuals to trigger at what time and calculating the “speed” of the movement. How? An image (refer to Fig. 11) with LiDAR datapoints is fed into OpenCV. The image constantly updates as you move around. OpenCV processes the images using the .getAverageFlow() function, which calculates  optical flow, determining the motion and speed.

Challenge No. 1: The technical process was quite tough. I worked with my technical supervisor throughout the entire duration. Setting the detection boundary was particularly challenging. The Processing software wasn't able to read the boundary accurately, causing software crashes.

Challenge No. 2: Detecting people’s movement. I tried several methods (comparing previous and current positions) and had to adapt and adopt different methods as I developed further. It was sometimes frustrating.

Next Step: Building the pool to put everything together.

The Fabrication: Building The Pool

We needed a pool for this project for obvious reasons. The idea of using a kiddie pool was definitely not an option—it looked ridiculous right from the start. We needed to build a minimalist-looking pool. Being a designer and far from a pool engineer (if that’s even a thing), I had to find a way to fabricate a pool as cheaply as possible.

My first thought was to make it using wood, but wood is generally expensive and not waterproof. If I used wood, it would be heavy and not modular for transportation at all. In the end, I decided to use heavy-duty tarps to create the base of the pool. On top, I used felt material to make it soft and pleasant to walk on. The structure of the pool was made with PVC pipes, stitched together into a 5x4m rectangle (refer to Fig. 13). Additionally, I strapped LED strips to the side of the pool using brackets and acrylic tape. One more important thing was waterproofing the electrical wiring to prevent short circuits.

LED lights
5m and 4m long LED lights were used for the pool structure. I waterproof the wiring for the LEDs, so it would not short circuit when is submerged in water (refer Fig.14).

Besides that, I used a 3D printed diffuser to diffuse the lights, making the lights looked like a seamless strip (refer Fig.15).

Challenge No.1: Designing different diffuser pattern in CAD software. My lecturer assisted in the designing the pattern, exploring different shapes. Different pattern diffuse the light differently.

The Aesthetic

I used wood to cover the sides of the pool, opting for a minimalist aesthetic. This gave it a simple and clean look, shifting attention to the visuals and caustic effects. Achieving this aesthetic required precise measurements and custom-made plywood. I made the design modular for easier transportation and management (refer to Fig. 17).

When designing an installation, it's best to make the design modular. This ensures easy installation on-site and easier management if needed.

The bricks didn't have to look polished because they would be in a dark room where no one would notice the imperfections. Due to the time constraints of the project, fully polishing the bricks would have taken too long.

Challenge No. 1: Precise measurements were needed to fit the pool size, but the outcome wasn't accurate. This was due to the flexibility of the PVC pipe; the structure bent irregularly, making it hard to fit the wood precisely.

Challenge No. 2: Sticking the plywood to the brick was another challenge. Initially, I used magnets to attach the wood to the brick, but the magnets were weak and fragile. They broke easily and didn't stick well.

I Built a slope so that people will know where to enter (refer to Fig.18). The slope ensures that people do not enter from different sides, preventing tripping.

Testing the Base: Testing materials with projector lights and focusing on walking comfort (refer to Fig.19). White materials reflect light the best, making the visuals pop even more. Darker materials absorb light, making the visuals appear dull. I tried foam and felt materials. Foam provided a more premium look and feel to the pool, while felt was comfortable for people to walk on. In the end, I chose white material due to its comfort. Close testing with my peers and supervisor also preferred the felt material.

Once everything is into place, we may start setting up the work, bringing all the elements together.

The Setup: Setting Up for Public Showcase

The setup consists of few different element; The pool, LiDAR sensors, LED lights, and short-throw projector (refer Fig.20).

The final setup (refer Fig.21) turned out well. The blue LED light casted on the wall was a surprising element at the end, making the pool look more aesthetically pleasing, However, there were some misalignment with the pool structure. It didn't affect the aesthetic as it is covered in the dark.

I noticed that you could hardly see the slope in the dark. To fix this, I added a fluorescent tape to indicate the slope. A mat was added too for people to dry their feet.

Challenge No.1: Filling the pool with water was a big hassle. I took an hour and a half to fill it up. But it took roughly 3 hours to dismantle everything. Spending time to vacuum the water out. It was physically and mentally demanding process.

The Finale: An Experience Where You Walk On Water.

We’ve come to an end of the blog, I finished rambling all my thoughts and process to you all. Through this project, I want to convey a message of embracing one’s emotions regardless, especially the negative ones. To overcome one’s anxiety or fear, is to face it straight on and embrace it.

Caustic is also a chance for me to push boundaries when it comes to designing immersive experiences and storytelling. I believe that mixing both non-digital elements(water) with digital media technology, we can create something that is out of the ordinary and rarely seen. Going back to the purpose of my creating this work is to immerse others in another realm where they can feel happy or inspired for at least 5 mins, I think this experience might just fulfill that very purpose.