Multimodality & Information Experience Design

Dylan Yamada-Rice

Outline
Information Experience Design (IED) work is multimodal. This workshop showed participants  how an understanding of multimodal theory can be used to expand their IED practice. There was a focus on:

  • The meaning of a mode
  • What it means to transduct information from one mode to another
  • The ways in which modes relate to culture and history
  • Information that is lost and gained when one mode is used over another

Slides 

The workshop started with an outline of multimodal theory similar to the last session (see here).

In the last multimodality workshop students were asked to observe one mode across the ground floor of the RCA White City campus and find a way to represent it, i.e. to see how the building smells and find a way of representing it. Having done this with groups of social science students in the past I was surprised when none of the IED students choose to represent the mode they were representing in writing.

Much of the literature on multimodal analysis within the social sciences uses a form of transcription that breaks down the multimodal whole by transcribing each mode into words and placing them in a table (such as the example below by Mavers, 2005):

Screen Shot 2018-11-15 at 15.58.20
An example of a multimodal transcript by Mavers 2005

Instead, the IED students transcribed their mode of focus into another but no one used writing. For example, sound in the ground floor was represented by colour or texture transducted into sound etc. This can be seen in the following example:

Screen Shot 2018-11-15 at 16.00.08.png

Since that workshop last year I have been thinking more about IED as a process of experimentation, whereby information is reduced to representation in a collection of modes a process that often involves investigating how the design is altered when some parts of the information are transducted from one mode into a range of others. In the process of doing this there is also a great deal of experimentation with materials too. As well as a constant shifting from obvious modal choices and materials to more abstract ones to see how it affects the experience of the information being conveyed.

The work of some IED graduates was shared as a means of showing the different modes they had used in order to convey information on a specific topic. One example was  Sylvana Lautier work Their Voices on climate change.

“My project visualizes the impact of global warming in Uummannaq, a small and remote island off of Northern Greenland. It is a pristine and awe-inspiring landscape of striking beauty and purity. The local population is much affected by melting ice, as well as by ‘black winds’ which bring pollution from all over the world. 

Their Voices immerses you in the world of Inuit children, with their unspoilt emotion, unfettered intuition, and purity of intention. This new generation aspires to a Western way of life, and thus the Inuits may lose their traditions and culture.” (Sylvana Lautier)

Sylvana’s work took on many forms over the course of her final year on the MA in IED. For example, she travelled to Uummannaq in Greenland to interview children about their views on climate change. She turned these interviews into a photographic book:

Screen Shot 2018-11-15 at 17.21.35.png
Book by Sylvana Lautier

She also produced a documentary film. Following these two pieces of work she began to explore how to enable her audience to understand the importance of the children in her work. To do this, Sylvana built a traditional Greenlandic house in children’s proportions in which she installed her documentary film. The size of the door forced the audience  to bend down to enter into her installation thus loosely embodying the proportions of the children in her study as they did so:

Picture1
Their Voices installation by Sylvana Lautier

Inside the house Sylvana projected her film through a mirror on which ice was melting. The audience was invited to touch the ices and move it across the mirror. As the audience did so ice melted and the  image of her documentary film became distorted by the pattern of water:

Screen Shot 2018-11-15 at 17.27.31.png
Their Voices installation by Sylvana Lautier

In this way the materiality of ice and water a reoccurring theme from Sylvana’s research findings were introduced into her final IED installation.

I elaborated on this by explaining how in my work in games design it is important to think about how narratives or toys that have physical versions can be adapted to digital platforms. To do so I often explore how specific elements of toys’ physical materials can be recreated in the digital For example, what is the one aspect of the material of Lego bricks that needs to remain in a digital game about the blocks for the user to recognise the brand?

In conducting research I also analyse multimodal data in a range of modes in order to see how they afford meaning, i.e. by reducing video footage of a child playing in VR to a line drawing in order to understand movement:

Screen Shot 2018-11-15 at 17.39.42.png
Children’s movement when using VR reduced to a line

Striping away all other details from the video, such as the background and the colour and texture of the child’s clothing, brought about a different way of understanding their movement when using VR.

Finally, the workshop participants were asked to think about modes in relation to senses:

Screen Shot 2018-11-15 at 17.29.57

There are connections between this and Sensory Ethnography, which is the focus of a future research workshop.

Practical workshop
The practical element of the workshop asked participants to use an IED graduate’s final  project as a starting point. Then to dissect the different modes used, and the extent to which they dominate the piece. From which they were asked to use practical means to explore what would happen to the information being conveyed by the work if different modes were foregrounded than the ones used.

Further Reading
Jewitt, C., Bezemer, J. & O’Halloran, K. (2016) Introducing Multimodality. Routledge.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s