Sensory Room

Overview

Through this project, we set out to answer the question: 'How can we help the hearing impaired experience music?' This question is particularly significant, as over 466 million people worldwide have some form of hearing impairment—a number projected to rise to 900 million by 2050, meaning 1 in 10 people will be affected.

Our research revealed that individuals with hearing impairments primarily experience music through vibrations. Currently, they rely on expensive and bulky vibration devices worn on their bodies to perceive music in daily life. Some even attend concerts barefoot to fully immerse themselves in the vibrations of the sound.

Solution

We came up of making the experience to be multi-sensory, not only using the visual component of the VR but also the tactile component made with Arduino. Audio component must be involved because having hearing impairment doesn’t necessarily mean that they hear nothing.

Design Process

UX Design

There were 3 main points to consider throughout the design process. The design was done by actively communcating with developers and strategists.

1. What status should the environment be in in order to give the most appropriate visual feedback to the user? We answered that the environment should be in a same line as a wearable vibrator does. Therefore, using visual sources that look like vibration and making them to change and move (vibrate) will do its work the most for the appropriate feedback.

2. Can we alternate the default controllers of the VR headesets to an actual hand? For this question, we chose to use Leap Motion, which can detect the hand motions of the users and also proved to be compatible with VR so that the users can control the game with their own hands.

3. What shape should the sound come from? Should it be a new, creative instrument with new shape or a typical keyboard with a same sound as a keyboard? We assumed that using a shape of a familiar shape and functions of a keyboard because it is intuitive and educational.

Environment Design

I used Maya to do the modeling and exported to Cinema 4D (C4D) for prototyping basic animation and rendering.

Music clouds, made up of vertex and rough faces so that the vibration can be visualized clearly. It changes the color and the movement according to the combination of the background music and the music that the user plays.

Keyboard table, modeled with beautiful, soft curves like a music that the user will feel.

Music balls. Our team chose to use small balls to make the vibration on the ground rather than transforming the ground itself to move, so that the users can experience the fully immersive environment but prevent them from having VR sickness.

Outcome

Demo Video

We designed and developed a working prototype, as shown in this video.

Achievement

Our team won the 1st place and $2500 in MIT Hacking arts 2019. 7 strangers from all over the world with same set of mind became a team like a destiny. We were a team of 3 developers, 2 designers, and 2 business strategists from MIT, Harvard University, North Eastern University, and Sogang University.