Scholarly article on topic 'First Person Experience of Body Transfer in Virtual Reality'

First Person Experience of Body Transfer in Virtual Reality Academic research paper on "Psychology"

Share paper
Academic journal
OECD Field of science

Academic research paper on topic "First Person Experience of Body Transfer in Virtual Reality"

Drumming in Immersive Virtual Reality: The Body Shapes the Way We Play

Konstantina Kilteni, llias Bergstrom, and Mel Slater

Fig. 1. The first person perspective of the participant looking down. (A) in the baseline condition participants were represented by flat shaded white hands. In the between-groups experimental conditions they either had (B) a casually dressed dark-skinned body or (C) a formally dressed light-skinned body.

Abstract—It has been shown that it is possible to generate perceptual illusions of ownership in immersive virtual reality (IVR) over a virtual body seen from first person perspective, in other words over a body that visually substitutes the person's real body. This can occur even when the virtual body is quite different in appearance from the person's real body. However, investigation of the psychological, behavioral and attitudinal consequences of such body transformations remains an interesting problem with much to be discovered. Thirty six Caucasian people participated in a between-groups experiment where they played a West-African Djembe hand drum while immersed in IVR and with a virtual body that substituted their own. The virtual hand drum was registered with a physical drum. They were alongside a virtual character that played a drum in a supporting, accompanying role. In a baseline condition participants were represented only by plainly shaded white hands, so that they were able merely to play. In the experimental condition they were represented either by a casually dressed dark-skinned virtual body (Casual Dark-Skinned - CD) or by a formal suited light-skinned body (Formal Light-Skinned - FL). Although participants of both groups experienced a strong body ownership illusion towards the virtual body, only those with the CD representation showed significant increases in their movement patterns for drumming compared to the baseline condition and compared with those embodied in the FL body. Moreover, the stronger the illusion of body ownership in the CD condition, the greater this behavioral change. A path analysis showed that the observed behavioral changes were a function of the strength of the illusion of body ownership towards the virtual body and its perceived appropriateness for the drumming task. These results demonstrate that full body ownership illusions can lead to substantial behavioral and possibly cognitive changes depending on the appearance of the virtual body. This could be important for many applications such as learning, education, training, psychotherapy and rehabilitation using IVR.

Index Terms—perception, presence, user studies, experimental methods, multimodal interaction, training, entertainment

- ♦ -

1 Introduction

perspective, when something touches the body you feel it, and it is under your full control. Would your behaviors or attitudes be affected, just because you apparently had a different body? Furthermore, would you perform some tasks better if your 'new' body seemed more appropriate for these? The present experiment addressed this question directly. We invited Caucasian people to participate in a virtual drum session and to express themselves by playing a West-African Djembe hand drum. In Immersive Virtual Reality (IVR), participants in a between-groups experimental design were represented virtually by either a male casually dressed dark-skinned body (Casual Dark-Skinned group - CD) or by a male light-skinned body wearing a formal suit (Formal Light-Skinned group -FL) (Fig. IB, C). In a baseline condition experienced by all, the virtual representation consisted only of a pair of white flat shaded hands (Fig. 1A). We investigated differences in the movement pattern between these two groups of participants while they played the drum.

Published by the IEEE Computer Society

Imagine that you wake up in the morning, you look at yourself in the mirror and suddenly, instead of your normal body appearance you see a completely different body with different hair, gender, skin color, and dressed in a very different style from normal. You then look directly towards yourself and you see that the mirror reflection is a true reflection of your changed body.

Despite its unusual appearance, you still perceive, however, that this is your body since you can see and hear from its first person

• K. Kilteni is with the Event Lab, Facultat dePsicologia. Universität de Barcelona, Spain. E-mail:

• I. Bergstrom is with the Event Lab. Facultat dePsicologia. Universität de Barcelona. Spain. E-mail:

• Mel Slater is with the Institicio Catalana de Recerca i Estudis Avançais (LCREA) at the Event Lab. Universität de Barcelona. Spain, and University College London. E-mail:

Manuscript received 13 September 2012: accepted 10 January 2013: posted

online 16March 2013: mailed on 16May 2013.

For information on obtaining reprints of this article, please send

e-mail to:

1077-2626/13/$31.00 © 2013 IEEE

1.1 Immersive Virtual Reality

Immersive virtual reality (IVR) often aims to transport people to a virtual place where they experience events and take part in activities. Such applications are mainly concerned with inducing the illusion of being in the place depicted by the IVR [1], and indeed one of the major challenges is the assessment and improvement of the sensation of 'being there' and the ability to act there, typically referred to as 'presence' [2, 3]. There are other classes of application such as scientific visualization where 'being there' is not especially important, but rather IVR is used in order to better understand a complex data set [4, 5]. The issues of 'being there' and the ability to 'act there', and of course all the technological requirements to achieve these, have generated important scientific and practical problems in IVR research.

In contrast the present study focused on using IVR to endow people with ownership over a different type of body and investigate the impact of this on their behavior in a quantifiable manner. Our hypothesis was that the form of the body would impact behavior: change the body and the resulting behavior of the participant might correspondingly change, following on from the idea of the Proteus Effect described in [6]. In order to induce such an effect, insights from cognitive neuroscience concerning body perception were applied to induce in the participants a so-called 'body ownership' illusion towards their virtual representation (or 'avatar').

1.2 Body Ownership Illusions

Body ownership refers to the feeling that the body and bodily sensations are self-attributed [7-9], e.g. 'this is my body', and it has been shown that body ownership can be experimentally induced towards inanimate objects as we now discuss. Botvinick and Cohen [10] showed that a rubber hand can be incorporated into the body representation through the use of appropriate synchronous multisensory stimulation - the Rubber Hand Illusion (RHI). In this paradigm the subject is seated at a table with a rubber arm placed on the table in an anatomically plausible position while the corresponding real arm is out of sight behind a partition. The experimenter synchronously taps the hidden real and the seen rubber hand, such that the taps and strokes are in the same place on each hand. After as little as 15s of stimulation the subject typically reports the illusion of ownership of the rubber hand. When the visual and tactile stimulation are not synchronous then the illusion does not occur.

Similarly, people can be given the illusion that a mannequin is their own body [11] when looking at it through head-mounted displays connected to video-cameras, and can be made to feel an illusion of being in a different place from their seen physical body, losing the sense of ownership of it, using a similar video-based technique [12]. An illusion of whole body ownership over a mannequin was achieved in [13] by mounting a pair of cameras at the eye positions of the mannequin and feeding the resulting video into a head-mounted display worn by the subject. The subjects thus saw the mannequin body as substituting their own when looking down towards their own body. When synchronous visual-tactile stimulation was applied to the body of the mannequin and the corresponding real location on the body of the subject, a strong illusion of ownership was induced. When the visual-tactile stimulation was not synchronous, the illusion occurred to a significantly lesser extent.

The principle of synchronous multisensory correlations for the induction of body ownership illusion has been extended to include also different modalities. For example, seeing a fake hand moving synchronously with the real one induces the illusion [14, 15]. In general, numerous studies in cognitive neuroscience and psychology have demonstrated that a fake body-part or a fake body can be illusorily experienced as the real body-part or the real body respectively, when the fake and real both receive spatiotemporally congruent multisensory and/or sensorimotor [10-17] stimulation.

1.3 Virtual Body Ownership Illusions

When someone wears a head-tracked stereo and wide field-of-view head-mounted display (HMD) his or her view of the real world is replaced by that of the virtual world. It is therefore also possible to visually substitute their real body by a virtual one. Moreover, if their body movements are tracked, then as they move their real body the virtual body can be made to move accordingly. It is also possible to include visual-tactile synchronous feedback manually (e.g. an experimenter delivering it in while watching virtual events on an external display) or through, for example, a haptic jacket. It is not too surprising to find that the same techniques of multi-sensory and sensorimotor stimulation, when applied to rubber hands and mannequin bodies, also generate the illusion of virtual body ownership.

The rubber hand illusion was shown to function in virtual reality to the same extent as in the original Botvinick and Cohen paper [18], with the difference that both the arm and the seen stimulus that was touching the virtual hand were represented only virtually. The illusion even occurs, though to a much lesser extent, with a flat video projected hand on a table top [19]. In [20] it was shown that visualmotor synchronous correlation between the real hand and a virtual hand also generated the illusion of ownership over the virtual hand. Similarly, whole body ownership illusions towards an avatar have also been demonstrated [21, 22].

In conclusion, numerous studies have shown that a virtual body-part or a virtual body can be illusorily experienced as the real body-part or the real body respectively, when the fake and real both receive spatiotemporally congruent multisensory and/or sensorimotor stimulation [21-26].

1.4 Behavioral Correlates of Ownership

Body ownership illusions have been extensively shown to have various behavioral correlates depending on the experimental setup. More specifically, the experience of body ownership towards an artificial body that differs morphologically from the real one, influences the participants' post-experiment performance in specific tasks. For example, synchronous visuotactile correlations on the faces of participants and a morphed [27] or an unfamiliar face [28], induced significant biases in their performance after the stimulation, compared to before, while executing a self-face recognition task. Additionally, in [29], an experiment was conducted using a modified version of the RHI, employing a black instead of a white rubber hand for Caucasian participants. The authors found that those participants who had experienced the body ownership illusion more strongly showed less racial bias after the illusion than before. The results of [30] are also relevant, where those participants who experienced the RHI perceived the rubber hand as being more similar morphologically to their own hand.

Analogously, it has long been known that body representation in Immersive Virtual Reality (IVR) has various behavioral consequences, as for example in enhancing presence [31] or in improving distance estimations [32]. Indeed the possibility of using IVR for body transformation was the subject of informal experimentation by Jaron Lanier and others in the very early days of virtual reality, as mentioned in [33]. Moreover, the concept that transformation of the body in IVR might lead to behavioral and attitudinal changes is not in itself new. The original ground-breaking work has been called the Proteus Effect [6]. Having a more attractive face or a taller body inside IVR affected the participants' emotional state in social interaction, an effect shown to occur in a variety of circumstances, and considered a fundamental capability, uniquely possible in IVR [34]. Observing an avatar as a self-reflection but of a different race to the participants was shown to affect their racial biases [35]. A closely related concept is that of 'virtual doppelgangers', where a participant sees an alternate self-representation carrying out activities [36-39], designed to induce attitudinal changes in the participant. Embodiment in a virtual body could also be related to the concept of self-presence, which refers to

the effect of the virtual body experience on one's self-identity [40]. In line with this, two recent studies successfully induced body ownership illusions in participants towards a virtual body that was radically different morphologically from their own: male participants were given a female virtual body [21], and thin men a fat virtual body [22]. In the latter case the participants overestimated their body size after the stimulation compared to beforehand.

1.5 An Experiment on Drumming

The purpose of the present research was to investigate whether differences between the real and virtual body have temporary consequences for participants' attitudes and behaviors under the illusory experience of body ownership. We therefore created two avatars of markedly different appearance, based on different skin color and dress style, each representing a distinctive social identity. Our goal was not to systematically explore the constituent elements of these two body types but rather address the question as to whether overall body type influences body ownership and behavior. Indeed there are so many constituent elements (e.g. skin color, race, gender, age, dress style, hair style, etc.) that identifying their separate contributions could be the subject of many additional studies.

Playing a musical instrument involves rapid and often complex motions, referred to as musical gestures [41] (p5). These include both those that directly result in the instrument producing sound, and also the ancillary movements of the performer, which while not directly producing a sound, nonetheless "have an intrinsic relationship with the music, representing a link between the music and the expressive intent of the musician" [42]. In fact, much of the expressive intent of musicians is perceived by audiences as a combination of visual and auditory percepts, and the visual aspect alone can have a great impact (for review see [43]). While performing, musicians exhibit the use of advanced embodied or enactive knowledge which can only be acquired and manifested through action [44]. Performativity [45] in the context of movement [46], refers to the notion that the pattern of overall body movement of individuals varies between situations, genders, and cultures, and is another expression of embodied knowledge. As observed by the anthropologist Marcel Mauss: the movement and positioning of people's bodies is not innate, but acquired, oftentimes tacit knowledge varying between cultures [47]. Further emphasizing the importance of visual stimuli in musical experience, it is known that the clothing and appearance of performers influence how audiences evaluate the performance [48]. More generally, dress and appearance influence the perception both of others, as well as self-perception [49].

The rationale for using a musical task in this study of body ownership and its influence on behavior was that it encourages significant motion, and therefore provides a good testing ground for the study of behavior. The use of percussion was chosen because it is a particularly visually expressive form of musical performance [43]. The West-African Djembe hand drum was chosen first because of its relatively low learning threshold compared to other instruments, requiring little particular technique for producing a sound, thus allowing participants to quickly learn to play spontaneously. Secondly, with this instrument musicians not only drum, but also simultaneously perform both choreographed and spontaneous dance movements, and oftentimes also sing. Very importantly from a technical point of view, the drumming task, even though complex, could be replicated well in IVR, without having in any way to simplify the task to circumvent limitations of current technology. The experience of playing the Djembe drum in our scenario was as rich as it would have been also in reality, which we ensured by involving an experienced Djembe drummer in the design of the scenario.

Participants were therefore invited to take part in a virtual drumming session and express themselves by playing the Djembe hand drum, while being accompanied by a neutrally dressed avatar that played a continuous supporting rhythm. In order not to limit

participants' behavior by specifying a strictly defined task, we asked them to express themselves by playing music ad lib.

In our experiment participants were represented by one of two different bodies as mentioned earlier. They saw their virtual body and the world from a first person perspective, from the eyes of that body. There were synchronous visuomotor and visuotactile correlations between the virtual and real upper body, including congruent mirror reflections. Additionally there were visual-auditory and tactile-auditory correlations caused by striking the drum with the hands. All of these correlations would be likely to maximize the sensation of body ownership over the avatar. We therefore expected that both groups would experience a body ownership illusion towards their virtual representation, but that the social cues associated with each one (through stereotyping) would influence the movement patterns of participants while playing the drum. In other words, we expected that the Formal Light-Skinned virtual body would be more detrimental to the performativity of participants, exhibited through more restrained body movements, as compared to the baseline condition. In contrast, the Casual Dark-Skinned body would encourage greater diversity in performativity.

To quantify the performativity of the participants, we recorded all their upper body movements during the whole experiment. Operationally we represented the upper body movement as the dimension of the data matrix representing the total set of positions of the body through time, using principal components analysis (PCA).

2 Materials And Methods

2.1 Recruitment and Design

Initially 38 participants were recruited from the university campus. Data for two were not useable so that the final sample was 36. They were all Caucasian (17 of them males). All participants read and signed an informed consent form. The study was performed according to institutional and national ethical standards for the protection of human participants. All participants were compensated with 10 euros ($13 at current exchange rates) after the end of the experiment.

The experiment had a between-groups design, with one factor ('Avatar Type') that had two levels: 'Casual Dark-Skinned' avatar (CD) and 'Formal Light-Skinned' avatar (FL). Nineteen participants had been arbitrarily assigned to the CD group, and 17 to the FL group. Both groups experienced the same 'white hands' baseline condition prior to embodiment into the FL or CD avatars. The baseline condition lasted 4 minutes. Then embodiment in the FL or the CD avatar lasted another 4 minutes.

Hence condition refers to 'white hands', 'Formal Light-Skin' (FL) and 'Casual Dark-Skin' (CD), and group refers to the two experimental groups assigned to experience either the FL or CD conditions after the baseline white hands. The terms 'white hands', FL and CD are further defined in Section 2.3.

2.2 Materials

A Djembe hand drum of height 64cm and diameter 33cm was used (Fig. 2 right). Participants were seated on a stool of 34 cm in diameter and fixed height of 50cm. The experiment took place in a room of (LxWxH) 340x300x287cm3.

All virtual models were designed in Autodesk 3D Studio Max 2010. A virtual drum model was created which bore a close resemblance to the actual drum used in the experiment. A virtual stool was also modeled as a replica of the real stool used by participants. The virtual scene was a neutrally furnished room of (LxWxH) 900x900x250cm3 containing the participant's avatar, an avatar representing an accompanier (the same across all conditions), a mirror of surface dimensions (WxH) 270x180cm2, wide enough to display both avatars from the vantage point of the participant, two stools where both avatars appeared sitting, each with its own Djembe drum supported between the avatar's legs (Fig. 2 left). The virtual drum and stool were carefully registered to be in the same place as

the physical drum and stool, so that when participants touched either with their virtual body they would feel the underlying corresponding real object. Both environment and accompanying avatar were designed to be neutral in relation to the drumming task, so as to avoid their appearance influencing the effect of our experimental manipulation.

Fig. 2. Left: The experimental conditions as seen in the mirror. (A) white hands condition (B) Casual Dark-Skinned avatar (C) Formal Light-Skinned avatar. In this view the participant can see the accompanying virtual drummer to the right in peripheral vision. Right: The setup for the participant, wearing the HMD, a body tracking suit, sitting on a stool playing the drum.

During all conditions, another avatar was to the right of the participant (Fig. 2 left). It was a neutrally dressed Asian avatar that played a basic accompanying rhythm for participants to improvise over. The rationale for this choice was to give a stable base rhythm to play over and possibly make the task of playing easier than if playing completely alone.

The drumming animation for this accompanying avatar was recorded using an Optitrack Motion Capture system, and manually refined using Autodesk's Motion Builder 2012 software. The movements captured were of someone with previous experience in playing the Djembe. The drumming audio used in the scenario was recorded at the same time as the motion, so that the two corresponded, and audio was played back from a physical location that closely matched the location of the accompanying avatar in VR. A Yamaha Digital Sound Projector YSP-4000 powered loudspeaker was used as the audio source. The virtual scenario was implemented using the XVR system [50], and the avatars animated using the HALCA system described in [51].

A stereo NVIS nVisor SX111 head-mounted display (HMD) was used. This has dual SXGA displays with 76°Hx64°V degrees field-of-view (FOV) per eye, totaling a wide field-of-view of 111° horizontal and 60° vertical, with a resolution of 1280x1024 per eye displayed at 60Hz. Head tracking was performed by a 6-DOF Intersense (ISENSE) IS-900 device. The body motion capture of participants was achieved using an Optitrack full body motion capture suit with 34 markers attached, captured by 12 infrared Optitrack cameras which operate at sub-millimeter precision. The laboratory cameras' calibration was of quality 'Excellent'. Head rotation data were streamed over a VRPN network to the PC running the XVR software [52]. OptiTrack Arena full body motion capture software was used to capture and stream the movement data. The equipment used is illustrated in Fig. 2 (right).

In order to map the physical movements of the participant to the avatar's bones, the avatar's spines, neck, upper arms, forearms and hands were all adjusted to the length of each of the corresponding limbs of the participant, as obtained from the Optitrack skeleton's streamed data. Then, all streamed rotations were applied hierarchically to the avatar's bones including the pelvis bone, except for the thighs, calves and feet, which were not animated. Head

rotations captured from the ISENSE tracker were mapped to animate the avatar's head. Hence, all the avatar's upper body movements were registered to the participant's physical movements, while the lower body maintained a fixed seated posture throughout the experience.

2.3 Procedures

2.3.1 Preparation

Participants had been arbitrarily assigned to one of the two groups (CD or FL). When they arrived at the laboratory for the experiment they were given an information sheet to read, with the procedures of the experiment also being explained to them verbally.

Their task was to express themselves freely through playing a drum, as the principal percussionist (soloist drummer) while a virtual player accompanied them on a second drum. To ensure that participants were aware of the cultural origins of the Djembe drum and in order to enter the right mood, they were shown a four minute medley video that included African Djembe players performing in the traditional manner and setting, as well as people from across genders, ages and nationalities, playing in constellations varying from solo drummers to an ensemble of seven. They were informed that they should later, while drumming, try to enter a mood similar to that of the drummers in the video by expressing themselves.

The experimenters then showed the participants how to support the drum with their legs, and how to perform the two most basic drum hits for playing the drum. It was demonstrated that the stronger the hit, the louder the produced sound volume. They were instructed that they were not allowed to stand up from the chair or move their legs and feet, but that they were otherwise fully free to move whichever part of the upper body they wanted. It was clarified that the accompanier would always play the same pattern, over which they were free to play as they liked.

They were reminded that they should feel free to play the drum however they wanted, without the need to imitate the other person that would accompany them (unless they wanted to) and that there were no specific rules to adhere to in their playing. Participants were also told that nobody would judge their performance and their rhythm. They were also told that the experimenters, who were seated behind a drawn curtain, would not directly see their performance.

Next, they were asked to put on the Optitrack Motion Capture suit, and the procedure for calibrating the Optitrack Arena software to their body dimensions was carried out. Then participants donned the HMD, which was calibrated so that its two screens were symmetrically placed over the participants' eyes using the method described in [53]. Subsequently, the participants were instructed to close their eyes, their hands were placed at the center of the physical drum, and two further calibrations were carried out: A precise relocation of the physical drum for its position to coincide with the virtual one, and a calibration of the avatar's position in the vertical axis to align the tracked position of the subjects virtual hands to the surface of the virtual drum.

2.3.2 White Hands Condition

As soon as the participants opened their eyes while wearing the HMD, they were instructed to look around and describe what they saw. This was to familiarize themselves with the scene and adapt to the virtual reality. In this first phase, the virtual body representing the participant consisted only of a pair of plainly shaded white hands. When looking downwards, they saw no virtual body, only white hands on the top of the drum and the empty chair (Fig. 1A). The same was seen in the mirror (Fig. 2A). In this way participants had information only about the position and orientation of their hands, providing the most minimal body representation necessary to still enable the task of drumming.

This white hands condition acted as a baseline. This condition, experienced by all participants, also served as a training phase, familiarizing participants with the virtual drum playing but without introducing potential extra difficulty or complexity to the task, as

would have arisen for example if sticks or spheres had been used instead of hands. The participant played in this white hands condition for four minutes, during which time all their motion data was recorded. After the four minutes, participants were asked to close their eyes briefly, during which time the appropriate virtual body was loaded.

2.3.3 Whole Body

The experimenters switched the virtual body representing the participant to the whole body corresponding to the group to which they were assigned. Thus, those in the CD group were provided with a male casually dressed dark-skinned virtual body, with long hair, wearing informal clothes e.g. jeans and a T-shirt (Fig. 1B, 2B) and those in the FL group a virtual body of a light-skinned man wearing a suit and tie (Fig. 1C, 2C). Participants were then left to continue playing the drum for another four minutes during which all motion data were recorded. After this time, the participants were told to close their eyes, thus concluding the experiment.

The experimental procedure and conditions are illustrated in the Supplemental Video (S1).

2.4 Variables

2.4.1 Questionnaire

Immediately after the experiment and removal of the HMD, participants were asked to complete an 8-item questionnaire about their experience. Each item was scored on a 1-7 Likert Scale where 1 means 'strongly disagree' and 7 'strongly agree' with the corresponding attribute. The questionnaire is given in Table 1 (with variable names representing the responses to the questions in parentheses).

Questions Q1, Q2 and Q8 served to evaluate the body ownership illusion and Q3 the sensation of being co-present with the accompanier avatar. Following completion of this, a short semi-structured interview was conducted and recorded.

After the interview participants were asked to complete two more questionnaires: a demographic information questionnaire, also eliciting their musical skills, and a standard personality inventory, the NEO-FFI [54]. This assesses their personality on five scales: neuroticism, extroversion, openness, agreeableness, and consciousness. The purpose of this was to check whether personality differences could have an effect on their performativity.

Table 1. Post Experiment Questionnaire

Q1 Even though the virtual body I saw did not look like me - I had the

sensation that the virtual body I saw when I looked towards myself in the mirror was mine. (memirror) Q2 Even though the virtual body I saw did not look like me - I had the

sensation that the virtual body I saw when I looked down at myself was mine. (medown)

Q3 I had the sensation that I was with another person who was playing the

drums. (otherperson) Q4 I felt myself to be more expressive than I normally am.


Q5 I felt myself to be less expressive than I normally am. (lessexpressive) Q6 My virtual body was more appropriate for playing this type of

drumming than my real body. (moreapp) Q7 My virtual body was less appropriate for playing this type of drumming

than my real body. (lessapp) Q8 Even though the virtual body I saw did not look like me - overall I had the sensation that the virtual body I saw when I looked at myself in the mirror or when I looked down at myself was my body. (mybody)

2.4.2 Movement Data

Data were collected on movements of the upper body throughout. This consisted of positional data for the following (on left and right sides where relevant): Neck (3), Head (3), Upper spine (3), Lower spine (3), Clavicles (6), Upper Arms (6), Forearms (6), Hands (6), resulting in 36 variables altogether (the figures in brackets show the

number of coordinates for each variable, 3 for one point and 6 for 2). This resulted in a N*36 data matrix (X) consisting of the positions measured in meters throughout the experiment for each participant.

We consider two 90-second intervals, the first starting 30 seconds after the beginning of the baseline (white hands), and the second 30 seconds after the start of the experimental condition (FL or CD). The 30s was allowed in order for the participants to settle in to the condition. 90s was taken to avoid getting data where the participants were becoming tired. Therefore for each participant there were two data matrices of the same size, one for the baseline condition (90s) and one for the experimental condition (90s) each recorded at 60 samples per second. Hence N = 90*60 for each matrix.

2.4.3 Performativity as Dimensionality

An important question was how to translate the complex idea of performativity into a numerical quantity for the purposes of analysis. In fact of course there is no one single measure that adequately captures this, and perhaps not even a combination of many measures - when compared with the qualitative judgments of expert observers. Here 'performativity' has been interpreted as dimensionality of the movement data. We would expect that if the different bodies had differential effects on the total movement of the participants, then this should be reflected in the fact that more dimensions would be needed to represent one experimental condition compared to the other. The operational hypothesis then becomes that the dimension needed to characterize the CD data would be greater than for the FL data, other things being equal.

Dimensionality of each condition (baseline, experiment) for each participant was determined by finding the number of eigenvalues of the covariance matrix (of each data matrix X) ordered from highest through lowest needed to account for at least 95% of the sum of all eigenvalues (i.e., a principal components analysis, PCA). We call this quantity d95. Each eigenvalue is the variance of the corresponding principal component. The sum of all eigenvalues is the total variance in the data set. Hence, the measures represent the dimensionality needed to explain 95% of the total variation in the data sets. Therefore, two values were recorded for the 95% criteria for each participant. One value was the dimension needed to represent the baseline condition (d95base), and the other was the dimension needed to represent the experimental condition (d95exp). To complement the analysis of dimensionality, which can be considered as an overall measure of performativity, we also compared the frequency of hand movements. While not as encompassing as dimensionality, it adds another more focused view of a single aspect of the behavior of participants.

3 Results

3.1 Participants and Subjective Ratings

There were no significant differences with respect to age, gender, musical skills and personal makeup between the CD and the FL groups (Supplemental Table S1). Additionally, there were no significant differences in the reported body ownership illusion towards the virtual body or the feelings of being co-present with the accompanying avatar between the two groups (Supplemental Table S2). More particularly, the three body ownership questions (Q1, Q2, Q8) were highly correlated with each other (correlation coefficients shown in Supplemental Table S3) and all three questions were rated highly (scores shown in Supplemental Table S2). Feeling co-present with the accompanier was also highly scored and it was not correlated with body ownership (Supplemental Table S3). Participants in the CD group felt that their virtual body was more appropriate for the drumming task than their real body, compared to those in the FL group (Supplemental Table S4). Responding to questions Q4 and Q5, the CD group reported that they were more expressive than they normally are, which was not the case for the FL group. However, there was no difference between the two groups

with respect to feeling less expressive than they normally are (Supplemental Table S5).

3.2 Motion Data Analysis

3.2.1 Dimensionality of the Positional Data

Table 2 shows the means and standard errors of the dimensions needed for each condition, and associated statistics. All variables are compatible with normality (Shapiro-Wilk test all P > 0.47). There was no significant difference in d95 between the two groups in the baseline condition. There was no change in the dimensionality between baseline and experimental condition for the FL group, but for the CD group a change by more than one dimension was observed. Furthermore, more than one dimension extra was needed to characterize 95% of the variation in movement in the CD group during the experimental period compared to the FL group. Similarly, t-tests of d95exp during the time on the experimental condition (CD, FL) revealed a significant difference between the condition means at P = 0.013. The rj1 show the strong effect sizes for both comparisons with d95exp in the CD condition.

Table 2. Mean ± Standard Error of d95 by Condition



P(paired t-test, 2-sided)

CD FL P (t-test, 2-sided)

7.47 ± 0.23 7.52 ± 0.38 0.90

8.53 ± 0.29 7.47 ± 0.27 0.013

0.0002 0.90

0.55 0.001

3.2.2 Dimensionality and Subjective Ratings

The relationships between the subjective ratings of body ownership (memirror, medown, mybody) and motion data dimensionality (d95exp) are shown in Supplemental Fig. S1. ANCOVA shows that for memirror there is positive slope for CD and a negative one for FL - the fitted model showed two different regression lines (F(3,32) = 6.14, R2 = 0.37, P = 0.005 rejecting the hypothesis that the slopes are the same).

3.3 An Overall Model

A path analysis was carried out using the Structural Equation Modelling software of Stata 12 in order to show in more depth the various connections between the variables (demographic data, personality characteristics, questionnaire scores, motion data dimensionality) (Supplemental Text S1). The path model is shown in Fig. 3 with corresponding details of estimates in the Supplemental Table S6.

Overall, the model suggests that motion data dimensionality was affected by both the feelings of body ownership towards the virtual body seen in the mirror and its perceived appropriateness for the drumming task. The feeling of more or less appropriateness of the virtual body was significantly associated with the experimental condition (CD versus FL). Critically, experienced body ownership was independent of the experimental condition and affected only by participants' NEO Openness scores, which is concerned with various aspects of being "open to experience". This is an excellent confirmation of consistency, that the one NEO variable that should have been related to the sensation of body ownership was so.

Fig. 3. Path analysis diagram where CD = 1, and FL = 2. 3.4 Frequency Analysis

Above we have shown that an extra dimension is needed to characterize the total upper body movement of those in the CD condition compared to those in the FL condition, and compared to the baseline. Since the amount of time in both experimental conditions was the same, this extra dimension must mean that there was more actual movement in the CD condition, and therefore that the frequency of movement must have been greater.

In order to illustrate this point further we considered the time series obtained by tracking the hands, the most active parts of the body during the drumming. From these time series we computed the number of peaks during the baseline (peaksB) and the experimental (peaksE) conditions for each individual. Then the peaks ratio, peaksR = peaksE/peaksB, gives an index of how much the movement changed from baseline to experimental conditions. Fig. 4 shows the scatter plots of this ratio for the right hand by body ownership (Q8). It can be seen that the peaks ratio is positively associated with body ownership in the CD condition and negatively associated with body ownership in the FL condition. Analysis of Covariance confirms that that the slopes of the two regression lines are significantly different (F(3,32) = 5.46, P = 0.004, R2 = 0.34) and the residual errors of the model fit are compatible with normality (Shapiro-Wilk W test, P > 0.10). For body ownership in relation to the virtual body seen in the mirror (Q1, memirror), the results are similar, with the Analysis of Covariance again showing a significant difference between the positive slope of peaksR on Q1 in the CD case, and a negative slope in the FL case (F(3,32) = 3.08, P = 0.013, R2 = 0.22, Shapiro Wilks test, P > 0.86). These results do not hold for the tracking of the left hand where there are no significant differences.

Although it is implicit in the dimensional analysis that there must have been a difference in frequency of overall movement between the CD and FL conditions, this further analysis confirms this for the most obvious movement during the drumming, at least for the right hand. This also explicitly connects the change in frequency of (right) hand movement with the strength of the body ownership illusion.


1 2 3 4 5 6 7


• CD • FL

Fig. 4. Experimental peaks over baseline peaks for the right hand movements by body ownership questions (a) Q8 (mybody): ANCOVA shows a significant difference between the slopes (F(3, 32) = 5.46, P = 0.004, R2 = 0.34) (b) Q1 (memirror): ANCOVA (F(3,32) = 3.08, P = 0.013, R2 = 0.22).

4 Discussion

Seeing a virtual body from first person perspective, and receiving spatiotemporally congruent multisensory and sensorimotor feedback with respect to the physical body entails an illusion of ownership over that virtual body. The present study extends and enriches previous studies on behavioral correlates of body ownership illusion specifically based on the appearance of the virtual body used. We found differences in the body movement patterns while drumming under a body ownership illusion, depending on whether the perceived body representation fulfilled expectations of what appearance was and was not appropriate for the context. Caucasian participants who were given a virtual body representation of a casually dressed dark-skinned avatar, exhibited higher variation and frequency of movement in a drumming task on an African drum, compared to when they were represented only by a pair of plainly shaded white hands, and compared to those participants who were given a light-skinned formally dressed avatar. Moreover, the greater the illusion of body ownership towards the CD body, the more the variation and frequency of movement while the opposite was the case for those embodied in the FL Body. These results provide the first piece of evidence that full virtual body ownership illusions can lead to substantial behavioral changes in the context of musical performativity, depending on the appearance the 'new body representation' disposes.

There are various possible interpretations of the present results. First, it could be argued that participants were behaving according to how they thought the experimenters would expect them to behave. However, this is highly unlikely since the experimental design was a between-groups one and participants therefore, were not aware of the other experimental condition, nor of the experiment's purpose. Second, it could be stated that expectation could have played a role in the translation from the 'white hands' baseline condition to a

virtual whole-body experimental condition. If this was true, then participants of both groups should have expressed the same behavioral changes independently of the body appearance, and the results clearly indicate that this was not the case. Another explanation could be that the results were simply the outcome of a learning effect in drumming from baseline to experimental condition, but then similar behavioral patterns should have been observed in both groups. Alternatively, it could be argued that the usual dress style of participants might have influenced the present results. If participants normally dress in a casual style similar to the clothes of the casual avatar, they may have self-identified more with their avatar (although the skin color similarity was absent). If this were the case, we would expect significantly higher body ownership scores on the post-experiment questionnaire for the CD group than for the FL group and this was not found. Critically, the path analysis model revealed independence between the illusion of body ownership and the condition (CD, FL).

Finally, it could be argued that the presence of the neutrally dressed accompanying avatar affected the observed behavioral changes. According to such an explanation, participants in the two groups behaved differently not because they perceived their body to be different but because they considered that the accompanying avatar expected them to behave like this. Such an explanation could be considered to be reasonable since participants reported strong feelings of being with another person in the virtual room. If participants indeed behaved only in order satisfy the expectations that they thought the accompanying avatar might have, there would have been no contribution of the body ownership illusion to the behavioral patterns. However, the results show that the behavioral changes of participants were significantly related to the experienced body ownership illusion towards the virtual body representation.

Having considered these alternative interpretations, we argue that it is not only the body representation that is updated during a virtual body ownership illusion, but that under specific circumstances, one's self-representation in terms of attitudes and behaviors can be temporarily affected [40]. The results suggest that body ownership can drive behavioral changes while executing a task when the new body representation is considered as more appropriate for the task than the real one. More than a mere body identification ("this is my body"), we speculate that participants were probably engaged in higher-level cognitive processes including a self-identification with the perceived social group to which the new body belongs, and adapted temporarily some aspects of their cognition to the new body representation ("my new body is more appropriate for drumming"). Since participants were given the clear instruction to feel free to act and express themselves through drumming without imitating any specific pattern, one could even argue for an ideated transfer of skills under a body ownership illusion in the form of motivation, e.g. "My new body knows how to play". And indeed, there were participants who reported spontaneously such a conditioning in the open-ended interview after the experiment.

Another fundamental result of the present study concerns the induction of strong body ownership illusions notwithstanding demographic differences between the participants' body and their given virtual one. The body ownership scores were very high and independent of the virtual body appearance, suggesting that people can experience the illusion of body ownership with respect to avatars of different skin color, supporting the results of [29]. Moreover, although almost half of the participants were female, all of them were given a male virtual body representation. However, there was no effect of gender on the body ownership illusion, replicating the results of [21] in which males had the illusion of ownership with respect to a female avatar. Therefore, the present results suggest that rich multisensory and sensorimotor correlations can induce strong body ownership illusion regardless of any demographic differences between the real and the virtual body representation. Once the illusion is established, it may be that the social identity associated with the new body drives the behavior.

The current approach differs conceptually from the Proteus [6] and Doppelganger [36] effects in several respects, although it is clearly located in that domain of work. First, the participant is the self-generator of his or her own behaviors, i.e., the body is not shown to be doing anything other than what the participant is actually doing. Second, in those studies the participant engaged in substantial social interaction with others (typically a confederate represented by an avatar in the VR), and therefore the effect of body transformation could be argued to be a self-reflection through the eyes of the other. In other words the other sees me as different and therefore I feel able to act accordingly. In contrast, our results consider the transformation in behavior as a function of how the participant experiences his or her virtual body and this was clearly illustrated in the path model. Critically, the stronger the body ownership towards the virtual body, the more the observed behavioral change.

Virtual reality offers an advantageous platform which can be used to systematically explore the role of body representation in cognition and behavior. Here, we have applied the principles of cognitive neuroscience to a highly novel situation, the creation of music, specifically drumming, in IVR and have explored how the form of the body impacts the way that the music is played. There are many possible applications of the present results. Earlier a range of studies was mentioned, where the similar Proteus Effect has been shown to be effective. In general, any situation where it could be advantageous to experience what it is like to be someone else with different characteristics would be a candidate. From a psychological point of view, this could serve as a type of empathy machine - not only can you be embodied in a different body but it is likely that the form of your behavior and maybe even attitudes will (at least temporarily) change. If having a different body can affect performativity in music, a careful selection of avatars for different tasks might also provide a useful means towards training for those tasks. For educational applications, embodiment in a suitable avatar could promote the learning process. Imagine a virtual philosophy class on the work of Socrates, taking place in a virtual Ancient Greece setting where students are represented by avatars dressed as Ancient Greeks wearing tunics.

Furthermore, when people are more willing to experiment with new possibilities for themselves they should also be more open to new knowledge and skills. This also emphasizes the fact that one of the important areas of application of IVR is to psychotherapy [55] and more generally to rehabilitation. Here, we consider that the utility of IVR rehabilitation procedures could be significantly enhanced through an embodiment approach. Learning how it is to be someone else, or more to the point, another version of oneself, could turn out to be an important first step in many different types of rehabilitation. Finally, the present experiment may serve as an example of how to approach the replication of a complex real-world task in VR, without over-simplifying the task to circumvent limitations of current VR Technology.

5 Conclusion

The results of the present study provide evidence that body ownership illusions towards virtual body representations can invoke substantial behavioral and possibly cognitive consequences when participants are embodied in avatars that differ morphologically from their physical bodies.


This research was supported by the European Research Council project TRAVERSE (#227985), the European Union FP7 AAT project VR-HYPERSPACE (#285681), and the Spanish Ministerio de Ciencia i Innoviacion project ResPIVE (#PSI2011-27930). We thank Ariadna Belles and Elias Giannopoulos for helping with the experiments and the video capture.


[1] F. Brooks Jr, "What's real about virtual reality?," Computer Graphics and Applications, IEEE, vol. 19, pp. 16-27, 1999.

[2] T. B. Sheridan, "Musings on Telepresence and Virtual Presence," Presence: Teleoperators and Virtual Environments, vol. 1, pp. 120-126, 1992.

[3] M. V. Sanchez-Vives and M. Slater, "From presence to consciousness through virtual reality," Nat Rev Neurosci, vol. 6, pp. 332-9, Apr 2005.

[4] S. Bryson, "Virtual reality in scientific visualization," Communications of the Acm, vol. 39, pp. 62-71, 1996.

[5] Prabhat, et al., "A comparative study of Desktop, Fishtank, and Cave systems for the exploration of volume rendered confocal data sets," IEEE Trans Vis Comput Graph, vol. 14, pp. 551-63, May-Jun 2008.

[6] N. Yee and J. N. Bailenson, "The Proteus effect: Self transformations in virtual reality," Human Communication Research, vol. 33, pp. 271-90, 2007.

[7] S. Gallagher, "Philosophical conceptions of the self: implications for cognitive science," Trends Cogn Sci, vol. 4, pp. 14-21, Jan 2000.

[8] M. Tsakiris, "My body in the brain: a neurocognitive model of body-ownership," Neuropsychologia, vol. 48, pp. 703-12, Feb 2010.

[9] O. Blanke, "Multisensory brain mechanisms of bodily self-consciousness," Nat Rev Neurosci, vol. 13, pp. 556-71, Aug 2012.

[10] M. Botvinick and J. Cohen, "Rubber hands 'feel' touch that eyes see," Nature, vol. 391, p. 756, Feb 19 1998.

[11] B. Lenggenhager, et al., "Video ergo sum: manipulating bodily self-consciousness," Science, vol. 317, pp. 1096-9, Aug 24 2007.

[12] H. H. Ehrsson, "The experimental induction of out-of-body experiences," Science, vol. 317, p. 1048, Aug 24 2007.

[13] V. I. Petkova and H. H. Ehrsson, "If I were you: perceptual illusion of body swapping," PLoS One, vol. 3, p. e3832, 2008.

[14] A. Kalckert and H. H. Ehrsson, "Moving a rubber hand that feels like your own: dissociation of ownership and agency," Frontiers in Human Neuroscience, vol. 6, 2012-March-14 2012.

[15] T. Dummer, et al., "Movement and the rubber hand illusion," Perception, vol. 38, pp. 271-80, 2009.

[16] M. Tsakiris, et al., "Having a body versus moving your body: How agency structures body-ownership," Conscious Cogn, vol. 15, pp. 42332, Jun 2006.

[17] L. D. Walsh, et al., "Proprioceptive signals contribute to the sense of body ownership," J Physiol, vol. 589, pp. 3009-21, Jun 15 2011.

[18] M. Slater, et al., "Towards a digital body: the virtual arm illusion," Front Hum Neurosci, vol. 2, p. 6, 2008.

[19] W. Ijsselsteijn, et al., "Is this my hand I see before me? The rubber hand illusion in Reality, Virtual Reality and Mixed Reality," Presence-Teleoperators and Virtual Environments, vol. 15, pp. 455-464, 2006.

[20] M. V. Sanchez-Vives, et al., "Virtual hand illusion induced by visuomotor correlations," PLoS One, vol. 5, p. e10381, 2010.

[21] M. Slater, et al., "First person experience of body transfer in virtual reality," PLoS ONE, p. e10564. doi:10.1371/journal.pone.0010564, 2010.

[22] J. M. Normand, et al., "Multisensory stimulation can induce an illusion of larger belly size in immersive virtual reality," PLoS One, vol. 6, p. e16128, 2011.

[23] D. Perez-Marcos, et al., "Is my hand connected to my body? The impact of body continuity and arm alignment on the virtual hand illusion," Cognitive Neurodynamics, pp. 1-11, 2011.

[24] K. Kilteni, et al., "Extending Body Space in Immersive Virtual Reality: A Very Long Arm Illusion," PLoS One, vol. 7, p. e40867, 2012.

[25] Y. Yuan and A. Steed, "Is the rubber hand illusion induced by immersive virtual reality?," in Virtual Reality Conference (VR), 2010 IEEE, 2010, pp. 95-102.

[26] M. González-Franco, et al., "The contribution of real-time mirror reflections of motor actions on virtual body ownership in an immersive virtual environment," 2010, pp. 111-114.

[27] M. Tsakiris, "Looking for myself: current multisensory input alters self-face recognition," PLoS One, vol. 3, p. e4040, 2008.

[28] A. Tajadura-Jiménez, et al., "The Other in Me: Interpersonal Multisensory Stimulation Changes the Mental Representation of the Self," PLoS One, vol. 7, p. e40682, 2012.

[29] H. Farmer, et al., "Beyond the colour of my skin: How skin colour affects the sense of body-ownership," Conscious Cogn, Jun 1 2012.

[30] M. R. Longo, et al., "Self awareness and the body image," Acta Psychol (Amst), vol. 132, pp. 166-72, Oct 2009.

[31] M. Slater, et al., "Simulating virtual environments within virtual environments as the basis for a psychophysics of presence," ACM Transactions on Graphics (TOG), vol. 29, p. 92, 2010.

[32] B. J. Mohler, et al., "The Effect of Viewing a Self-Avatar on Distance Judgments in an HMD-Based Virtual Environment," Presence: Teleoperators and Virtual Environments, vol. 19, pp. 230-242, 2010.

[33] L. Jaron, "You Are Not a Gadget: A Manifesto," ed: New York: Knopf, 2010.

[34] J. Bailenson, et al., "Transformed social interaction in mediated interpersonal communication," Mediated Interpersonal Communication, p. 77, 2008.

[35] V. Groom, et al., "The influence of racial embodiment on racial bias in immersive virtual environments," Social Influence, vol. 4, pp. 231-248, 2009.

[36] J. Fox and J. N. Bailenson, "Virtual self-modeling: The effects of vicarious reinforcement and identification on exercise behaviors," Media Psychology, vol. 12, pp. 1-25, 2009.

[37] H. E. Hershfield, et al., "Increasing saving behavior through age-progressed renderings of the future self," Journal of Marketing Research, vol. 48, pp. 23-37.

[38] J. Fox, et al., "Virtual experiences, physical behaviors: The effect of presence on imitation of an eating avatar," Presence: Teleoperators and Virtual Environments, vol. 18, pp. 294-303, 2009.

[39] S. J. Ahn, "Embodied Experiences in Immersive Virtual Environments: Effects on Pro-Environmental Attitude and Behavior," PhD, Stanford University, 2011.

[40] K. M. Lee, "Presence, explicated," Communication Theory, vol. 14, pp. 27-50, 2004.

[41] E. R. Miranda and M. M. Wanderley, New digital musical instruments: control and interaction beyond the keyboard vol. 21: AR Editions, Inc., 2006.

[42] M. Nusseck and M. M. Wanderley, "Music and motion-how music-related ancillary body movements contribute to the experience of music," Music Perception, vol. 26, pp. 335-353, 2009.

[43] M. Schutz, "Seeing music? What musicians need to know about vision," 2008.

[44] F. J. Varela, et al., The embodied mind: Cognitive science and human experience: MIT press, 1999.

[45] K. Hall, "Performativity," Journal of Linguistic Anthropology, vol. 9, pp. 184-187, 1999.

[46] R. Eves, "Engendering Gesture: Gender Performativity and Bodily Regimes from New Ireland," The Asia Pacific Journal of Anthropology, vol. 11, pp. 1-16.

[47] M. Mauss, "Les techniques du corps," Journal de psychologie, vol. 32, pp. 365-86, 1936.

[48] N. K. Griffiths, "'Posh music should equal posh dress': an investigation into the concert dress and physical appearance of female soloists," Psychology of Music, vol. 38, pp. 159-177, 2010.

[49] Y. H. Kwon, "The influence of appropriateness of dress and gender on the self-perception of occupational attributes," Clothing and Textiles Research Journal, vol. 12, pp. 33-39, 1994.

[50] F. Tecchia, et al., "A flexible framework for wide-spectrum vr development," Presence: Teleoperators and Virtual Environments, vol. 19, pp. 302-312, 2010.

[51] M. Gillies and B. Spanlang, "Comparing and evaluating real-time character engines for virtual environments," PRESENCE -Teleoperators and Virtual Environments, vol. 19, pp. 95-117, 2010.

[52] T. C. Hudson, et al., "VRPN: a device-independent, network-transparent VR peripheral system," 2001, pp. 55-61.

[53] J. A. Jones, et al., "The effects of virtual reality, augmented reality, and motion parallax on egocentric depth perception," 2008, pp. 9-14.

[54] P. T. Costa, et al., Revised neo personality inventory (neo pi-r) and neo five-factor inventory (neo-ffi): Psychological Assessment Resources Odessa, FL, 1992.

[55] C. J. Bohil, et al., "Virtual reality in neuroscience research and therapy," Nature Reviews Neuroscience, 2011.