Give Me A Sign: Developing The Narrative

We are continuing to develop our script for the character of Shunya. When an audience member selects and presents a mudra, one out of three outputs will appear on a screen in the form of a short animated video, accompanied by sound and text. Each of these clips will address a certain theme which branches from the original meaning of the Indian dance mudra. Shunya will respond with a statement, philosophy or a leading question which we are scripting in parts collaborating with ChatGPT3.

Mudras & the Theme of Climate Action

We have been researching and interpreting the meaning of the mudras further by using them as tools to explore narratives and urgent conversations around possible futures and the climate crisis.

Similar to our first prototype we are using 6 mudras to converse with Shunya:

• Mookoola: This sign is interpreted as the bud of a flower. It can also mean to demonstrate the action of eating, or used to show the navel of a human, animal or God. Here the navel is meant to represent the connection of life — the umbilical cord. 
• Prithvi or Mayura: Mayura means ‘Peacock’ in indian dance. And in yoga the same mudra is called Prithvi, which denotes the Earth.
• Shikara: Shikhara is the human element.  It translates into the word ‘peak’. The peak of a mountain, the peak of civilization, the top of the food chain, man or the masculine, male Gods, Shiva, leader of the tribe/ pack / society, a phallic symbol.
• Soochi: Soochi literally means ‘needle’, it can be used to indicate or point to something or symbolise a tool or technology. 
• Mushthi: This sign takes the form of a fist, which can represent power, grasping at something stolen or holding a weapon. 

Pataka: The Pataka mudra means’ flag’, it can be used as a sign to stop,  or as a blessing. In dance depending on the way it is performed it can denote the sea, forest, clouds, river, wind, shield, among many others.

Interaction

This project is a successor from an earlier prototype where we used Google’s Teachable Machine. The key problem with the first version that we wanted to address was the issue of running the project successfully no matter who is sitting in front of the camera, what they are wearing, and their surroundings. 

For this new iteration of Give Me a Sign we have been learning and experimenting with more robust gesture recognition Machine Learning techniques. We are exploring the possibility of hand pose recognition, where instead of the computer classifying images based on matching pixels, it can detect the position of a hand. 

The hand is broken up into ‘landmark’ points which the Machine Learning model can be trained on, mapping the arrangement of these points to our desired triggered outputs such as sound or animation. We began using libraries such as the Tensorflow/MediaPipe Handpose Model (which we found to be a better version of the ML5 Handpose model), p5.js, and MI5 Neural Network. The project needed to function fully offline, while running video and real-time skeletal hand recognition. 

Instead of the computer classifying images based on matching pixels, it can detect the position of a hand.

We want the user to have an experience beyond the basic human-computer interaction, generating the perception of communication and collaboration with the machine. By feeding it a gestural meaning and receiving a unique response, that builds a conversation between the user and the machine; a story that can be narrated depending on which gesture is presented and in which order.

User Interface: Holding Page

The holding page is a key element of the project, acting as a User Interface (UI) for interaction. Our speculative/fantastical character Shunya, is one that is learning from us and sharing their knowledge back with the user.

Currently we are aiming for this holding page to have an animated hand that cyclically runs through our six selected gestures, until an audience member is prompted to choose a gesture to hold up to the webcam, thus activating Shunya.

Visual Language

Visual language is the key to communicating the key themes of each of the selected Mudras meanings in relation to the climate change narrative.

The elements we are designing:

• 3D Character 
• Mask 
• Environment – stable diffusion, void, fog, after effects
• Video transitions

Masks

We are designing 3D mask models for each of gesture, reflecting on the meaning and message we wish to convey with the gesture.

For instance, Prithvi (Earth) can correspond to ecological awareness, which corresponds to entanglement, vines, roots, and mycelium.

Sketches

Early sketch of a mask which could be modelled and used for the gesture, Prithvi.

Instead of a human head – deconstructing our theme into elements that evoke them and using those to construct Shunya. 

Character Design

Early experiments creating our bespoke character within Unreal Engine’s Metahuman. We have decided not to go in this direction.

Layout Mockup

Instead of a full humanoid figure we will have an invisible body and head, with 6 alternate masks and corresponding floating hands. Text will also appear on the screen, which matches the statements, philosophy and questions which will be audible. 

Hands

For the holding page we are planning on using a wireframe single hand. Shunya however will have hands which are textured to correspond to the aesthetic of their mask. 

Deepmotion

We are using Deepmotion to animate our character. Shunya will perform the mudra that the audience member has presented, and the way the mudra is performed takes inspiration from Indian dance and other more contemporary uses.

Sound

We are working with a sound designer, bringing together a mixture of traditional indian instruments with a more ambient synth spacey soundscape, and layered spoken word elements.

Challenges

Having such a small team has been a challenge, it has been difficult to balance the technical aspects of the project alongside the more artistic and design elements. We now have Hasan on board to help us complete the programming, which has allowed us to really get sucked into the visuals, narrative and storytelling.

Give Me A Sign is the product of the BeFantastic Fellowship, an online programme fostering international collaborations between creatives in the UK and India, exploring AI technologies and creating provocative performance pieces amplified with creative AI.

As one of the BeFantastic winning commissions, Give Me A Sign will be presented at HOME in Manchester, February 25th – 26th and in March 2023 at the FutureFantastic Festival, an exciting and ambitious new AI+Art festival as part of the British Council’s India Together UK Season of Culture, celebrating the remarkable bond between two countries and exploring our cultures, our shared planet and our relationship with digital technologies that will shape our future together. Conceptualized by Jaaga’s BeFantastic (India), in association with FutureEverything.