{"doi":"10.1093/screen/hjaa058","title":"‘Real-time’ virtual reality and the limits of immersion","abstract":null,"journal":"Screen","year":2021,"id":679271,"datarank":0.26876392038420827,"base_score":1.791759469228055,"endowment":1.791759469228055,"self_citation_contribution":0.26876392038420827,"citation_network_contribution":0.0,"self_endowment_contribution":0.26876392038420827,"citer_contribution":0.0,"corpus_percentile":null,"corpus_rank":null,"citation_count":5,"citer_count":0,"citers_with_citation_signal":0,"citers_with_endowment":0,"datacite_reuse_total":0,"is_dataset":false,"is_dataset_confidence":null,"is_data_producer":false,"deposit_databanks":null,"is_oa":false,"file_count":0,"downloads":0,"has_version_chain":false,"published_date":null,"fair_score":null,"fair_percentile":null,"algorithm_id":"datarank_citation_only_1hop_v6","ranking_scope":"data_only","authors":[{"id":1774775,"name":"Richard Misek","orcid":null,"position":0,"is_corresponding":false}],"reference_count":0,"raw_metadata":{"has_enrichment":true,"resolved":true,"title":"‘Real-time’ virtual reality and the limits of immersion","abstract":"What does virtual reality do best? Judging by its rapid uptake in fields including architectural visualization, surgical training, crime scene reconstruction, military simulation and, of course, gaming, virtual reality (VR) has proved to be a particularly effective tool for reconstructing and simulating physical spaces, as well as enabling a haptic engagement with them. It is telling that the most common descriptor for virtual reality is ‘immersive’, a term that highlights its spatial and experiential dimension. Writing in 1999, Jay David Bolter and Richard Grusin summarized the then prevailing view as follows: virtual reality is immersive, which means that it is a medium whose purpose is to disappear […] As computer scientists themselves put it […] the viewer should forget that she is in fact wearing a computer interface and accept the graphic image that it offers as her own visual world.1 Bolter and Grusin’s extensive references to the professional and academic literature of the period highlight the degree to which spatial world-building was already underpinning assumptions about what the first generation of headset-based experiences did best. It remains so over 20 years later. Archetypal in this regard is gaming, currently the most popular VR consumer application. Of the 12 best-selling VR games on the 2018 Steam store, six were first-person shooter games and three were first-person fantasy adventures, genres that combine environmental and architectural visualization, battlefield simulation, and above all spatial exploration. In tandem with killing opponents, the goal of each of these games is to advance as far through their 3D environments as possible.2 Many notable artistic uses of VR, from the visual defamiliarization of In the Eyes of Animals (Marshmallow Laser Feast, 2015) to the dense soundscapes of Notes on Blindness: Into Darkness (Arnaud Colinart, Amaury La Burthe, Peter Middleton, James Spinney, 2016) have also exploited its ability to generate vivid spatial experiences. But what about time? How does the experience of temporality play out in VR? Virtual reality’s focus on spatial experience typically leaves the temporality of a virtual environment to default to ‘real time’: time passes within it at the same rate as it does outside it. For example, in Star Wars: Secrets of the Empire, which I discuss below, participants undertake a spatially and temporally continuous journey, as if it were a single-shot film. Any other temporality would risk compromising VR’s sense of immersion by drawing attention to its mediation. It is no coincidence that the fully interactive form of VR, the focus of this essay, is often called ‘real-time VR’. The term refers specifically to the real-time rendering of a 3D environment that allows it to update in real time – that is, instantly – in response to users’ movements.3 At the same time, the immersion in a continuously unfolding space that real-time rendering makes possible has to date almost always entailed a ‘real-time’ user experience. Doubtless this is in part a by-product of the creators’ desire to exploit VR’s spatial and haptic potential, but it is also encouraged by the software used to create VR experiences. Video editing software is structured around a timeline, which orders video clips created at different times and in different locations into a fixed duration (figure 1). Through the editing process the duration of the clips and the temporality that they collectively articulate can be readily manipulated; indeed, temporal manipulation is a key element of the editing process. By contrast, the ‘game engines’ typically used to create VR experiences are structured around a Euclidian space in which multiple assets (objects, characters, users’ avatars) are placed in a spatial relationship with each other (figure 2). Although it is easy to create spatial environments that bend the laws of physics, it is far harder to create environments whose temporality is malleable. The interface and linear editing timeline of Adobe Premiere. Image courtesy Richard Misek. The interface and 3D ‘scene’ view of the Unity game engine. Image courtesy Richard Misek. This essay emerges from a three-episode ‘virtual reality video essay’, A Machine for Viewing (2019), which I produced recently in collaboration with VR creator Oscar Raby and filmmaker Charlie Shackleton. The experience of working on this project brought into sometimes painful focus how well suited game engines are to world-building, and how ill suited they are to temporal manipulation. At a moment when uses of, and discussions around, VR remain dominated by the ideal of immersion, the explicit aim of our project was to use ‘real-time VR’ as a tool for immersion and reflection. Rather than creating an extensive space to be explored, we created a relatively small virtual cinema that would provide a familiar setting for a series of playful interactions with, and opportunities for reflecting on, ‘cinematic’ moving images. Over the course of the three episodes, participants manipulate the aspect ratios of various films shown in our ‘VR cinema’; they experience a haptic interaction with the cinema screen; and they explore the perceptual differences between 3D space and the spherical flatness of 360-degree film. All episodes feature a voice-over narration that provides a degree of guidance and context for the participant’s interactions. I cannot speak for my co-creators’ use of voice-over, but my own choice to include it was a deliberately perverse one, aimed at creating a palpable tension between a linear narrative typical of cinema and a focus on interaction that was more typical of gaming. By providing an experience that at certain times felt like a typical documentary or essay film and at other times diverged drastically from linear narrative, my aim was to foster a dialogue between linear and interactive media, and to continue in a long tradition of attempts (such as those of the MIT Open Documentary Lab and the iDocs community) to hybridize the two.4 Unfortunately I did not realize just how much Unity (the software we used) resists attempts to constrain and shape duration. For example, although recent versions of Unity include a ‘timeline’ window, it is nothing like the top-level structuring tool so familiar to video editors. Rather it is an optional function that can be added to individual assets to cause them to perform a particular series of time-bound actions, such as moving along a specific path for a specific length of time. A Unity project may thus include multiple timelines, none of which are particularly important to shaping the overall temporal experience. Misled by the timeline function, I initially scripted a linear narrative interspersed with interactive sections, without fully appreciating that Unity is designed to deliver the opposite: an interactive environment interspersed with localized time-bound activities triggered by the user’s movement through space. Through various workarounds, notably that of adding a timeline to a 3D asset and then populating it with audio clips, we were able to ensure particular lines of voiceover played at particular times. However, these workarounds also resulted in a complex network of connections between 3D assets, video clips, voice-over clips and interactive events, which meant that even the smallest alteration (such as cutting a line of voice-over) might have a ripple effect on the timing of the entire episode. Although my episode has the appearance of a linear video essay that features interactivity, under the surface its structure is stubbornly spatial. Of course temporal manipulation is often unnecessary in VR; for the various forms of professional simulation listed above, real-time spatial experience is usually sufficient. For more creative applications of VR, however, the manipulation of temporality offers many potentially exciting avenues for experimentation. William Urrichio rightly observes that at present VR ‘builds on the experiential narratives enabled by some game genres far more effectively than on the narrative structures inherited from cinema and literature’.5 But that does not mean it cannot learn from the supple temporal manipulations that have long since been exploited to such great effect in other forms, including cinema and literature. Consumer VR launches from game genres so easily at least in part because it was developed as a tool for gaming.6 As long as it remains restricted to creating the kind of immersive presence demanded by the mainstream gaming industry, it risks overlooking the creative power of temporal manipulation.7 I examine in this essay a number of VR works that engage creatively with temporality. In doing so I highlight the value of using VR to generate something like the opposite of Bolter and Grusin’s definition of immersion: a sense of spatio-temporal ‘outsideness’ that involves a reflexive experience of looking in on a constructed world; an experience that could help uncover uses of VR who are not constrained by any established ideology of what it ‘should’ be. But first I provide a brief example of VR’s default spatio-temporality. The VOID’s Star Wars: Secrets of the Empire (2018) is a touring, location-based ‘immersive virtual reality experience’ that puts participants into the role of rebels infiltrating an Imperial base.8 Participants are initially gathered into a small group and are each kitted out with a headset and backpack (in reality, a desktop computer strapped to their back that allows them to move around untethered). As they put on their headsets, the bare walls of an empty room transform into the interior of a spaceship, and the participants appear to each other as figures in stormtrooper uniforms, whose animation maps precisely onto their physical movement. As a working-through of VR’s aspiration of spatial immersion, Secrets of the Empire is a remarkable achievement. The visual environment is meticulously modelled and rendered. The constraints of the physical location are ingeniously overcome within the virtual space: mid-sized rooms become flight decks, control rooms and walkways with panoramas across vast expanses of space. Meanwhile the path physically traced by participants through the installation is surreptitiously folded back on itself so that rooms already visited become whole new virtual spaces. The experience also hybridizes VR and cinema surprisingly effectively by combining the spatial immersion of VR with the fantasy of being ‘in’ a film, as a character in the diegesis.9 Various avatars within the virtual environment speak to participants, thus strengthening their sense of being in a film while also subtly directing them through the spaces. While this work expands the cinematic experience of watching a Star Wars movie, it also reduces it. Participants are restricted to experiencing a single 20-minute scene that plays out in real time and in continuous space, and their main activities are confined to shooting and puzzle-solving. Just as the visual environment of Secrets of the Empire never escapes the iconography of the movie franchise from which it is derived, its interactivity never escapes the conventions of a first-person shooter. Nor does it need to. It is precisely the combination of familiar images and interactions that provides the most frictionless (and marketable) means of delivering the experience’s main attraction: untethered physical movement through a simulacral virtual space. Like most headset-based VR experiences, Secrets of the Empire plays out a spatial presumption that is already embedded in the commercial hardware and software used to create and experience VR. It is present, for example, in the design of the headset itself, whose lenses magnify a small rectangular LCD screen so that it extends beyond the user’s field of vision. It is present, too, in the headset’s default set-up, which even before any content has loaded translates the motion of the user’s head into an equivalent movement in the virtual environment. This spatial presumption is further reinforced by the image that appears as soon as the VR headset senses a head inside it; the Oculus Rift, for example, presents a Cartesian grid extending in all directions along the x/z axes towards a horizon. Wait a few seconds, and you find yourself in the Oculus Store, which takes the form of an expansive modernist living room with picture windows looking out towards a blue-skied Californian horizon. Paradoxically one cannot walk through this space or do anything with the objects it contains; the business of loading content and managing one’s account in fact happens on a series of flat panels that incongruously float in the middle of the living room, and whose layout looks not dissimilar from that of the Oculus website. Made to be looked at, the Oculus house fulfils no function other than to showcase the headset’s ability to display photorealist spaces. The progression from Cartesian grid to 3D simulation mentioned above in turn replicates the process by which VR experiences are created. Both Unity and Unreal, the two programs most commonly used in VR production, initially also present creators with an empty space and an infinite Cartesian grid extending along the x/z axes towards a horizon. The interface pushes creators towards the spatial environments of 3D gaming: from populating the empty space with assets (objects, characters, buildings), through to defining a ‘viewport’ that establishes a first-person point of view, and setting up ‘colliders’ that prescribe what happens when the player’s avatar hits an asset. Crucially the presumption of the software is that everything exists and happens in the same 3D space. Movement between discontinuous time-spaces, achievable so easily in cinema by means of a cut, necessitates the loading of a whole new 3D environment. As such it is typically restricted to occasional  between  of  The  of the software in this  are  by those of the  even if the software   cutting from one 3D environment to   example, by means of a top-level timeline into which  of different immersive 3D  could be  it would   rendering the new 3D environment in a  of a  This currently remains beyond the  of even the  gaming  typically used to  and experience VR  The spatial   by the game   the real time of first-person  and the  real-time rendering are thus all  As a  creating anything that  the  temporal experience so easily  by an editing timeline involves  to workarounds that  the  like those we used in A Machine for  It is  that most  uses of VR (such as those  on  and   on a sense of continuous   from a first-person point of  It is   however,  more creative uses of VR (in game design or interactive  should also  themselves to first-person experiences of continuous  – and  usually they  As if  the  constraints of   into   much recent  around VR has  not on space but on the  of users’  and on new  for  with  As the recent  of headset-based motion  and  has  the  for  interactivity to  the  of VR is  At the same time, if  with users’   no further than  them a virtual  presence and  that replicates how their  move and  in physical space, then this    VR’s spatial  Although  forms of haptic interaction may  VR to move  from the  default of   this risks  it with a new default of   in which the user’s  always forms the   of a  grid extending towards a     is also  for VR to  its  of Cartesian  In my view that    an ability for VR to provide  with  that are   to     in the  of    in other  that  for more   and even  engagement with    that   for  in headset-based  in the same  that      in cinematic  Writing about film,    that     that an     in the   if   to  in all the  which  up a being or an  with     this  by      The  of  all  the form of a flat shape – a    and so  But to  them for what they  in their  one cannot  the   with  one  to  at them from the   from outside their spatial environment. By    it is the experience of watching  flat films from the   that allows  to  and  to them in their   might  this  by  that the    as the sense of being  the   or  by a creative  plays an  role in   even   It makes possible what Richard   about   to as the  of artistic  a  attention to the  of a work and the means by which this content has been  But how can the immersive environments of VR provide the sense of an      2016) provides a   Like Secrets of the Empire it is  a first-person shooter  as  move through different   figures appear and  at  each  leaves a    its  in space. The figures  so  and from so many directions that the  can  become  in a  network of lines (figure   is, however, a    and time  The   in  their   become   the  to  the  of the   and  how to move   a  to the  of The      allows  to  back from immersive  and regard it from an   This  is as much spatial as   outside the  three  the  become    allows the  to  how far along its individual timeline a  has  and to  how long they have   it  them.    to the  of   and  with, the  of spatial immersion, by  an  from the  of  and, in  by  to   presumption of ‘real  as a    this much   from gaming     with, and plays out  precisely the kind of simulacral 3D space that the game  is designed to   as long as the user  it remains an  first-person shooter experience.  space in        is an example of the  and   degree of immersion that VR has  possible over recent  However, at a time when it is  in the process of defining itself, of  its  and its  it  to  particularly important that VR creators should also be  works that  beyond immersion and   VR  as if from the  At one  when I was  in Secrets of the Empire, I  my headset and  a  moment in which I  the physical space of the  The computer panels were   and  was a      through a  a    the  of what  just   been a   far above the   of  This was the  that the work itself was  a perceptual experience of being outside the virtual space, which  my  of the  I  with two brief  each of which in its own   into its design a sense of     are by  not  game  or experience  The first is      a ‘virtual  by its  was  using   a kind of immersive  that allows  to  in three   within the virtual 3D space that they  (figure   users’  in virtual space without any need for a     the  for VR to move beyond spatial simulation, while    of  space. For example, it  in a     created using  and  the    disappear  a few seconds, a   remains  at all times. The first part of  leaves this visual  of    within its virtual environment  is no  of the default   just a series of   In the  part of the  the  takes  their headset and  a  room in which the  is  the  then  an  of the user in  and  it into the virtual space that the  has just  This  part of the experience thus  and   the   participants to  at the virtual space on a flat  and to  on the process  its   from     It is notable that  VR experience  an element of    VR presents the user with a   and if an   to  beyond this   physical space offers an    It is no coincidence  at the  of its   A Machine for Viewing  also  a   By  the headset-based experience in a physical cinema and  an  to  the VR experience on an  cinema   with more    we folded cinema and virtual reality within each other and  them into each   As a  the virtual reality experience  a  for a series of  in cinematic  the cinema  in turn  these  from the  of the cinema space. In  at one point the  is  to   their  and they  experience the    The moment at which one takes  a headset can be as  as the moment one puts it  But is it possible to provide a reflexive   on VR from within VR?     a    does so by  spatial simulation  As the   around a  of a  a complex  of 3D    and  itself as a single  (figure     is to  to  out what it is – a house  or a  in fact the  in  is  a combination of  3D   a while it   that the image is  important than the  which involves a  relationship between one’s movement and the  the    as one  it and  as one   from it. The line traced by the participant’s physical movement  a timeline,  to a play  on a  and by   and  they  along it. The  of this     the sense that one is moving through virtual space.  one is moving through  and not just temporality in  but the specific temporality of the  that is  this  It is as if one’s  is moving  and  through a   of  and experiencing not a space but a       that the VR headset is just an  Although a user’s movement usually  a   in  within a virtual space, this is  a  It is even possible that in the  VR  may become  for interactions that  no virtual  at    observes that  allows    to become     three  that are  in  around  artistic  but almost never used with  to VR. By looking beyond immersive space,   and   VR to a space with which has  been  – the space of reflection. In the Oculus Store, for example, the   and  currently  almost  VR  and  the     a    VR can find  to  different forms of engagement beyond the  experience of spatial immersion, and  it can in particular create more space for  its  as an artistic form may be  and it may  to find a space for itself as anything other than a tool for","is_dataset_classified":null,"base_score":1.791759469228055,"endowment":1.791759469228055,"datacite_reuse_total":0,"file_count":0,"downloads":0,"views":0,"has_version_chain":false,"is_dataset":false,"is_oa":false,"pmid":"21097893","pmcid":null,"openalex_id":"https://openalex.org/W3130316430","authors":[],"funders":[],"total_grants":0,"fwci":1.0097,"citation_percentile":0.85742225,"influential_citations":0,"citation_trend":[{"year":2022,"count":1},{"year":2023,"count":2},{"year":2024,"count":1},{"year":2025,"count":1}],"oa_status":"hybrid","license":"cc-by","oa_locations":[{"url":"https://academic.oup.com/screen/article-pdf/61/4/615/36333513/hjaa058.pdf","host_type":"journal"},{"url":"https://academic.oup.com/screen/article-pdf/61/4/615/36333513/hjaa058.pdf","host_type":"publisher"},{"url":"http://academic.oup.com/screen/article-pdf/61/4/615/36333513/hjaa058.pdf","host_type":"publisher"},{"url":"https://doi.org/10.1093/screen/hjaa058","host_type":"journal"},{"url":"https://doi.org/10.1093/screen%2Fhjaa058>)","host_type":"repository"},{"url":"https://kar.kent.ac.uk/86952/1/hjaa058.pdf","host_type":"repository"}],"fields_of_study":["Digital Games and Media"],"mesh_terms":[],"keywords":["Immersion (mathematics)","Virtual reality","Computer science","Mixed reality","Computer graphics (images)","Art","Human–computer interaction","Mathematics","Geometry"],"sdg_mappings":[],"linked_datasets":[],"clinical_trials":[],"software_tools":[],"database_accessions":[],"source":"live","citation_network_status":"fetched"},"created_at":"2026-08-17T12:44:13.323513Z","pmid":null,"pmcid":null,"fwci":null,"citation_percentile":null,"influential_citations":0,"oa_status":null,"license":null,"views":0,"total_file_size_bytes":0,"version_count":0,"fair_f":null,"fair_a":null,"fair_i":null,"fair_r":null,"fair_zscore":null,"fair_rationale":null,"fair_model":null,"fair_agent_version":null,"fair_fulltext_source":null,"fair_has_llm":null,"fair_computed_at":null,"clinical_trials":[],"software_tools":[],"db_accessions":[],"linked_datasets":[],"topics":[]}