Musical Internet of Things Connects Music Lovers in an Age of DistancingSeptember 29, 2020 by Teja Sai Marellapudi
At a time of physical distancing, the applications of Musical Internet of Things are ever more relevant.
Previously, we discussed the merging fields of IoT and the performing arts, a phenomenon known as the Musical Internet of Things or "IoMusT." This technology has only become more prevalent during the age of COVID-19 where many live musical events, like concerts, are no longer possible in a physical sense.
Here are a few ways we see IoMusT appear in real-world applications.
Musical performances connected over networks enable musicians to reach both co-located as well as remote audiences.
IoMusT network block diagram.
One example is a networked music performance (NMP) system called ReacTable, which consists of a touchable interface of a table that can track how objects are moved on the surface to subsequently control musical output. This interface allows musicians located in different locations to collaborate with items on the table and perform in real-time at the same concert.
ReacTable interface. Image used courtesy of ReacTable
Networked performances also allow connections between various Things, like smart instruments and wearables, which provide more comprehensive services and musical collaboration to both musicians and audience members. Networked music systems can take place over Local Area Networks (LAN), Wide Area Networks (WAN), or Wireless Sensor Networks (WSN).
An interactive performance usually consists of sensor-equipped live performers and musicians with real-time control over media elements, like lights, using physical gestures. Digital musical instruments are a key part of live performances, mapping gestures to control signals of the instrument.
Singer-songwriter Imogen Heap is a pioneer in interactive technology for musical performances. She is known for developing MI-MU Gloves, which allow her to introduce auditory elements to her performance through hand and arm movements.
MI-MU Gloves rely on motion sensors embedded in the fabric. Image used courtesy of MI-MU
Wall Street Journal writer John Jurgensen explains, "Instead of fiddling with a laptop on stage, as many electronic artists do, Ms. Heap controls her gear with the WiFi-connected gloves and custom movements, such as spreading her arms to add a cavernous echo to her vocals or pinching her fingers to brighten a synthesizer tone."
Virtual and Augmented Reality Concerts
VR and AR interfaces, including head-mounted displays, can immerse users in a multisensory musical experience. Recently, that has even extended to remote concerts and collaborations that connect people across the globe.
These virtual environments sometimes include a number of virtual instruments that the users themselves can play. Immersive music performances include 3D widgets and graphical elements that allow users to control and visualize music. For example, Melody VR is an app that allows you to "watch a concert from your couch."
Melody VR is an app that allows users to experience live concert performances using VR technology. Image used courtesy of Melody VR
Augmented reality has led to enhanced live performances by superimposing the live performance stage with a virtual environment filled with various digital elements and widgets to control the musical process.
Coachella, one of the world’s biggest music festivals, is known for using such immersive tech. In 2017, the concert planners released an AR/VR app for festival-goers. The app allowed attendees to access an AR experience using the fan's "Welcome Box" number as a tracking marker.
Musical Haptic Wearables
The last few years have seen a tremendous rise in wearables as IoT technology has become more sophisticated. A number of smart devices are now available as wearables that can sense, store, analyze data, and send it back to an application.
The wearables available to the IoMusT are called musical haptic wearables.
This armband prototype is an example of a haptic wearable that allows a group of music performers to non-verbally exchange musical directions during an improvisation. Screenshot used courtesy of Internet of Musical Things
These devices include haptic simulation, gesture tracking, and wireless connectivity. Musical haptic devices facilitate an added layer of communication between musicians in both co-located and remote locations.
Audience Participation Through Technology
In recent times, the trend of actively engaging audiences through various connected technologies has increased. This is known as technology-mediated audience participation (TMAP). In TMAP performances, the audience has control over the mood and visual content being delivered in the concert via their smartphones.
An Open Symphony performance. Image used courtesy of Mathieu Barthet
Examples of this mechanism are Mood Conductor, dubbed an "emotion-driven interactive music performance," and Open Symphony, a combination of sensors and apps that allows audiences to influence the music played in a live environment.
Connecting Music Lovers Everywhere
While engineers are continually forging new innovations in the IoT space, IoMusT has become increasingly important at a time where physical gatherings—including musician rehearsals, concerts, and other musical gatherings—are limited.