“Storytelling for VR” is a hands-on concept workshop. Together with “Writing for TV and Online” and “Writing for Games”, it forms part of the “Plot Twist: Stories for the 21st Century” program supported by Film Victoria, a government agency that provides strategic leadership and assistance to the film, television and digital media sectors in the State of Victoria, Australia.
In the words of Plot Twist, the Storytelling for VR Workshop is designed for: “Anyone who has not yet worked professionally in the field of Virtual Reality and who has an idea that would be well suited to being told in this medium. You could be already working in theatre, visual arts or other narrative mediums. You just need to be new to VR.” The workshop covers “the fundamentals of working within narrative in a Virtual Reality landscape”. Participants will explore “how varying degrees of agency, perspective, immersion and the presence of the audience influence narrative development and user experience”. They will also get an idea of “how storytelling in Virtual Reality differs from traditional film/television as well as games”.
The Storytelling for VR Workshop is run by VRTOV, an award-winning artist-led studio based in Melbourne, Australia, with a strong focus on “crafting stories that put the audience at the heart of the experience”. Utilising their expertise in fields such as game design, animation, documentary, theatre, performance art, new-media art and creative technology, VRTOV specialises in creating “VR experiences that resonate with audiences around the globe”.
A few examples: VRTOV’s 2013 project Assent is an immersive documentary that transfers the viewer back to September 1973 when the military took control of Chile in a coup. The documentary won the Interactive Audience Award in the 2014 Sheffield Doc/Fest, and was an Official Selection for the 2015 Sundance New Frontier. In 2016, VRTOV collaborated with BBC Learning and Crossover Labs to present Easter Rising: Voice of a Rebel. The VR project features the 1916 uprising where thousands of men and women occupied key sites in Dublin to proclaim the Irish Republic. Also in 2016, together with BBC Research and Development, VRTOV premiered The Turning Forest at the Tribeca Film Festival. The real-time CG VR experience leads the viewer through a magical land in the footsteps of a boy. Here, in a dream-like fantasy, they get to confront a majestic creature and hear it roar. It is indeed a “virtual reality fairy tale for young and old alike”.
(All images thanks to VRTOV. Click to enlarge.)
I was selected as one of 12 participants in the one-day Storytelling for VR Workshop, which took place on May 26 in ACMI-X, a 2,000-square-metre state-of-the-art office space in the heart of Melbourne’s arts precinct. Part of the Australian Centre for the Moving Image (ACMI), ACMI-X is an eye-opening place where filmmakers, digital and visual artists, digital producers, web developers, screenwriters and designers gather to develop film, TV, video games and digital art that feature collaboration, innovation and sustainability. As a mature-age emerging writer with very limited knowledge of and experience in digital technologies, I felt overwhelmed upon arrival.
But Katy Morrison, co-founder of VRTOV and a recent VR Artist in Residence at the Royal National Theatre in London, soon put us at ease with a brief introduction on Real-Time VR. Using VRTOV’s various projects as examples, Katy explained that Real-Time VR is its own genre, involving not only the use of film, but also theatre, games and physical performance. While VR projects are produced using the game engine Unity, the story is the most important part, as the focus of any narrative form is to tell a good story. To paraphrase Katy: “VR narrative is not a screen that you wrap around your head. It is not an extended cinema.” Instead: “It is an interactive, user-initiated experience that is incredibly empowering and has a physical presence around the viewer/visitor, so they have to be able to affect that ‘reality’.” In short: “VR is an artistic medium.”
Next, Oscar Raby, award-winning multimedia artist, fellow VR Artist in Residence at London’s Royal National Theatre and VRTOV’s Creative Director, gave us an awesome talk on the concept and theoretical/philosophical background of VR. According to Oscar, the territory of experience is the body. “The body is not just a space, but concerns time as well. VR deals with extension of the body and asks how one can convey this experience to other bodies.”
We all know that artists convey what they see using visual, audio and/or textual mediums such as drawings/paintings, sounds/noise/music and/or written words. Other mediums include but not limited to physical performance, e.g. dance, theatre and installation art. But Oscar focused on practical and artistic efforts to overlap one body’s experience on another body, using “gadgets” as extensions of the body. For example, accountants of the ancient Inca Empire travelled and recorded the happenings across the land. Dangling from their belts were multiple strings, which they used to tie knots as a way to record data. As “snapshots of the land”, the data recorded by this device contained more than numbers and values, as these were “intertwined with people’s lives in order to sustain the empire”. As Oscar described it: “Numbers tell stories, always underlying stories, bringing them to human experiences and everyday lives.” (See Quipus, also known as “talking knots”.)
Oscar paid special attention to efforts to “objectify” experiences via visual devices. Since as early as Albrecht Dürer’s The Four Books of Measurement with Compass and Ruler (1525) and The Four Books of Human Proportion (1528), artists have learned to see the world with lines and numbers. But these lines and numbers are significant only because we use them to describe our experiences. As René Descartes famously put it: “I think, therefore I am.” We are the centre of the experiences. We are the starting point. Any attempt to record and refine our universe has to begin by mentally placing ourselves at the intersecting point of the x, y and z axes.
Oscar took this notion one step further. He stressed that this whole process – this mental positioning of one as the origin – can be done by anyone and then transferred to anyone else. i.e. At the central point to capture the surroundings can be any Self, any body, or even an object. Equally important is the notion that to be captured can be real, unreal and even imagined/fabricated/constructed, as seen in Scott Roth’s “Ray Casting” algorithm (1982), which renders three-dimensional scenes to two-dimensional images in computer graphics. Utilising this algorithm, id Software’s Wolfenstein 3D (1992) was a critical and commercial success, a prominent example of creating 3D illusion in a game. The player assumes the role of a spy trying to escape WWII Nazi German prison Castle Wolfenstein while carrying out a series of important missions.
To paraphrase Oscar: “We have what the camera can give us. This is dominating in storytelling, but it is not the only way. If we can share a reality – if we can pass on a reality – then we can tell the story differently. Things can be described in different ways – not just through language, not just what we can say or do, but the whole describing process as well. You get to experience it – you as the viewer/visitor get to perform it.” As if this conceptualisation is not complex enough, Oscar went even further, inviting us to imagine ourselves as the camera. That is, in the VR environment, we no longer depend on a machine to tell us what it is seeing. Rather, there is no camera, because we are the camera. We are the eye. Instead of “How does a machine tell me what it is seeing?”, we need to start considering “How do I pass on to another body what I am seeing/hearing/experiencing?”
Oscar proceeded to explain the capacities of VR technologies and how they operate from the viewer/visitor’s perspective at intellectual, emotional and psychological levels. These are crucial in designing and producing user-friendly and empowering VR experiences, shifting our focus from technologies to physical performances while still keeping us aware of what technologies can do. Oscar’s conclusion: “What is reality? Any reality can be shared.” We were left with these fascinating words from Soviet filmmaker and cinema theorist Dziga Vector: “I am an eye. A mechanical eye. I, the machine, show you a world the way only I can see it. I free myself for today and forever from human immobility… My way leads towards the creation of a fresh perception of the world. Thus, I explain in a new way the world unknown to you.”
After his talk, Oscar asked all the participants at the Writing for VR Workshop to share our story ideas and what we think VR can help with the storytelling. This is what I like about workshops, to learn from others and to be amazed at how unique each and every one of us as artists can be. Many of us are already familiar with theatre, visual arts, game design and/or other narrative mediums. Most of us have some sort of idea about using VR to help building empathy, exploring previously unknown, underestimated or unconditioned emotions, and recognising what we are and how we can be so much more. What is for sure is that all of us want to use stories and storytelling to capture our audiences!
Having received priceless one-on-one feedback from Oscar on our story ideas, we proceeded to learn the VR production structures. After that, the 12 of us were divided into four teams to brainstorm our own VR project. Each team then had two hours to work with a student from RMIT University who is already familiar with the game engine Unity, to turn our rough idea into an actual VR experience. This is where the fun began, as none of us realised how complex the process is to turn a simple and mostly verbally-composed-and-expressed idea into an real functional application on the computer screen. We wanted an image here, a background there, a segment of audio/visual material inserted at an odd spot, a colour flashing or a shape morphing, or an object moving from one corner of the screen to another. Then we had to specify the relevant requirements in order for the student programmer to understand what we needed and to locate and adjust the corresponding functions of the game engine to deliver a rough result. Then the team members took turns to wear the “gadget” – the VR headset – to experience it. Then the whole process started all over again: “Can you make this a bit bigger? Can you cut that a bit shorter? Oh, while we are here, can you also make that other thing a bit smoother/rounder/bigger/louder/whatever…???”
When I put on the VR headset, I saw a gridded dome-shaped world in black and white, much like Homer Simpson’s 3D world. I was told to move to the centre of a circle, but when I looked down to adjust my position, I could not see my feet! It was a really curious sensation, knowing I was there but unable to see myself. I suppose this is something that even Oscar could not easily explain in words, something that we had to experience by ourselves. i.e. To mentally position ourselves as the origin, at the centre of everything, we had to first “spirit away”/abandon our physical presence and concentrate instead on our existence only as something invisible – perhaps as a “ghost”. I immediately began to appreciate how tough it is for the character Kitai Raige to comprehend and master the technique “ghosting” in After Earth (2013). I also gained a better understanding of the confusion and fear felt by one of the characters in Hong Kong science fiction author Ni Kuang’s The Obscuring Point (1982). With his brainwaves altered by a mysterious power, the poor guy looks into a mirror and sees everything but himself…
Our template had a featureless grey-coloured 3D figure standing in front of us (the viewer/visitor). Behind the figure were a sphere and a cube suspending in the air. Focusing our gaze at the cube beyond the figure’s right shoulder would lead to something happening, while looking at the sphere beyond the figure’s left shoulder would trigger something else. I have to say, this was perfect for my story idea “Chance Encounter”, initially a “choose your own adventure” digital novel where readers have two options at the end of each chapter regarding how the story should develop. Anyway, when I was wearing the VR headset, I kept craning my neck and standing on my tiptoes in an attempt to look beyond the figure’s shoulders. At one stage I even tried to push the figure away, and other team members had to restrain me from moving forward and bumping into the computer table. On another occasion I jumped up and down, trying to peek out a window that was placed high above me. Sounds silly, but all of it felt really real.
Needless to day, all four teams managed to come up with their own brief, rough but highly amusing VR experience, and we all giggled like three-year-olds, delighted by our small yet amazing achievements. Perhaps we also felt that mysterious pull of something greater than us – that almighty power of cutting-edge technologies, how it can impact on us and… ahhh… how we can use it to affect the others in numerous (positive) ways. I imagine this is how Harry Potter felt when he finally accepted his magical powers, before commencing the journey to actively study the art and craft of being a wizard. All that learning in front of you – all those possibilities, all those opportunities! You just need to go for them!
To conclude my reflections on the Writing for VR Workshop, I want to apologise to those fellow participants who overheard and perhaps were bothered by my conversation with two RMIT students in Mandarin. It was a private conversation, but I recognise the fact that it might seem inappropriate and even impolite to use another language wherever the “norm” is to speak English. Still, as a writer and occasional journalist, for the purpose of interviewing people (i.e. to extract [precious] information from them), I need to make them comfortable in order for them to speak to me freely. In this case, I got to learn a fair bit about the development and general reception of VR technologies in China and Taiwan. I also gained a glimpse of how beginning-to-intermediate-level VR users might feel about the strengths and shortcomings of existing technologies, and that is quite different from the insight offered by advanced users and industrial experts in the field.
Voices under the Sun 陽光下的聲音 