What Is the Internet of Musical Things (IoMusT)?

Most people in today’s connected world are well aware of the term the Internet of Things (IoT). But what about the Internet of Musical Things?

According to Luca Turchet et. al, researchers studying the Internet of Musical Things, a musical thing (MusT) is a “computing device capable of sensing and exchanging data to serve a musical purpose."

The Internet of Musical Things ("IoMusT") appears in a number of applications, including augmented and immersive concerts with VR experience, audience participation, haptic devices, and smart studio production. These IoMusT experiences are attracting both artists and audiences around the world.

For example, in Taylor Swift's 2015 music tour for her 1989 album, fans were given LED bracelets upon entering the venue. These bracelets, designed by PixMob, included infrared transmitters and an RFID chip to sync to sensory aspects of the concert experience: the bracelet lit up in sync with the music, lighting, and movement of the concert-goer for a more immersive experience.

 

These wristbands have also been used at Super Bowl halftime shows. Image used courtesy of PixMob

 

But what's the magic behind IoMusT? And what are the design challenges associated with this musical branch of IoT? This article aims to establish the possibilities as well as the limitations of the IoMusT.

 

The Parts That Make Up the IoMusT

To understand the IoMusT we must explore the components that make up the system. The IoMusT consists of hardware and software components that fall into three categories: 1) the musical things themselves, 2) connectivity, and 3) apps and services.

 

1. Musical Things

Musical Things are devices that have sensing, actuating, connectivity, and software capabilities, which can be used by either artists or audiences. Musical Things can include a smart instrument, a musical haptic wearable, a networked speaker system, and smart mixing consoles.

There are a number of these devices on the market, like Elk Audio's Sensus Smart Guitar, a guitar augmented with wireless sensors and network technologies. Teenage Engineering, a Swedish electronics company, recently unveiled Rumble, a silent haptic subwoofer module for the synthesizer OP-Z. It recreates the natural bass vibrations experienced on acoustic instruments and translates them via digital gear.

 

A Rumble module, which allows users to "feel the beat." Image used courtesy of Teenage Engineering

 

Another device for musicians is a wearable from Soundbrenner that vibrates on the player's wrist as a metronome while also acting as a contact tuner and dB meter.

And then, of course, there are the ubiquitous smart speakers, like Amazon Alexa and Google Home, which allows users to cast their music onto devices connected by the same Wi-Fi network.

 

2. Connectivity

As with any IoT device, connectivity is a key factor.

The IoMusT supports wired and wireless communication in both local and remote communications—for example, to facilitate local and remote concert experiences. IoMusT devices connect over standard protocols to ensure interoperability and compatibility between different interface layers and musical devices.

 

Elk Audio says the Sensus guitar is the first smart guitar that allows users to add any number of modulations and effects without additional amplifiers, accessories, or computers. Image used courtesy of Elk Audio

 

However, the main challenges in connectivity involve real-time use cases. In order to achieve high-quality audio, an IoMusT ecosystem requires low latency, high reliability, and tight synchronization.

 

3. Apps and Services

More interactive experiences can be built and customized for IoMusT using web tools and APIs, providing meaningful experiences for artists, producers, and audiences.

For example, letting users connect to smart musical instruments will allow fans to follow how a musician is playing. This might extend to direct audience participation in concerts with the IoMusT devices delivering content from musicians’ smart instruments to audience members’ smartphones and vice versa. 

One application of this appeared when Eventbrite acquired an RFID chipmaker Scintilla Technologies to design "wearable tickets" that will quickly approve concert-goers as they pass under an RFID antenna.

Those learning to play music could also benefit from e-learning platforms connected to the IoMusT, which can collect and analyze data about how the student is playing. 

 

Challenges and Solutions of the IoMusT

As Luca Turchet et. al explain, the IoMusT ecosystem experiences many of the same challenges and solutions as any other IoT device, but some of these limitations are more exaggerated because of IoMusT's dependence on synchronization, device compatibility, and certain design constraints.

Designers of IoMusT devices face the issue of battery and energy consumption in the overall system. Designers may also grapple with the issue of miniaturizing computing units to support low-latency, sensing, actuating, and RF communication. Currently, wearables and VR technology are making steady progress on this front.

 

Lag-Time Can Kill an IoMusT Experience

Real-time music entails reliable, audio transmission over both wired and wireless networks—the key being low-latency. The requirement of real-time music can put high pressure on designers building communication networks that aim for near-perfect transmission quality. To this end, IoMusT devices must transfer audio signals at a stable reception rate with satisfactory synchronization between two smart devices.

Communication networks can fail to receive messages, which can lead to jitters and errors in data reception. Synchronization becomes an even bigger issue in the audio streams of devices that use various clock frequencies.

 

Diagram of a proposed IoMusT ecosystem. Image used courtesy of Luca Turchet et. al

 

There are a couple of ways to solve this problem. One of the solutions is to shape the communication waveform so that both music and synchronization information can be transferred via the same wave. Another approach adapts the parameters of communication protocols. The rise in edge computing can also play an important role in reducing network latency by offloading cloud computing.

 

Standardization Is Key

The success of an IoMusT experience also hinges on the compatibility between different elements of a network. This compatibility becomes crucial for latency and synchronized signal transmissions over channels. Users must establish a common communication protocol with a range of devices.

This calls for developing ad-hoc protocols and interchange formats that are relevant to different Musical Things. Designers might also develop separate APIs specifically for IoMusT end applications. Another aspect will involve defining new formats for storing files of Musical Things.

 

Technology That Opens Artistic Potential

The IoMusT holds great promise for more immersive musical performances. It allows users to easily share data for audio production, e-learning, and music design. 

The IoMusT yields a number of new interfaces for expressing music, networking remote music performances, and developing inspired art through human-computer interaction and artificial intelligence. While designers of this branch of IoT must confront issues of low-latency, synchronization, and standardization, the solutions to these barriers will surely open more interesting and creative ways of expressing music.

 

---

 

Have you ever worked on an IoMusT device? What was your experience like? Share your thoughts in the comments below.