Philip Sheppard has recorded solo cello albums, composed more than 60 soundtracks, and adapted 206 national anthems for Olympic Games medal ceremonies. He knows every corner of the famed Abbey Road Studios. His work is the process of creating music, but sometimes he just wants to listen, preferably while walking in the woods.
During one of those walks, in 2016, he didn’t want to be bothered choosing what music to play. Instead he imagined some sort of magical accompaniment piped into his earphones that would dynamically reflect his surroundings and his mood, a literal soundtrack for his saunter. Violins might soar with vibrato at the moment when the sun broke through the trees. Flutes might trill along with the songbirds.
“Surely, it's possible that the music should follow me rather than me trying something from a playlist,” he says. “Moreover, if I change direction or the sun is setting, I'd love the music to change as well.”
In order to do that, the system would not only have to be able to process multiple inputs—sound, sight, biometrics, speed, weather information—but to understand music well enough to translate those inputs into musical changes that sounded like a coherent composition. Sheppard tried to outline a means of doing this—basically annotating a score with the way potential inputs might change it—but to really implement the idea he needed an expert in artificial intelligence.
Fortunately, Sheppard knew the perfect candidate: Tom Gruber, the co-creator of the Siri intelligent assistant and later one of Apple’s top AI executives. Gruber had even given a TED talk called “Humanistic AI,” urging collaboration between people and machines. The two had met at a conference in Monterey in 2010 and become friends. So in 2017, Sheppard pitched Gruber on the idea during a walk on the Santa Cruz coast, sans soundtrack. They became CEO and CTO of a new company. (Gruber left Apple in 2018.)
Three years later, the result is LifeScore, which commissions music, records it with top-notch artists, and license it to businesses ranging from video games to automakers. Listeners hear unique combinations of those scores, remixed on the fly. While soundtrack composers synch their work to a script, a LifeScore composition is directed by an AI conductor. The AI matches arpeggios and pianissimos to the experience of listeners in real time, often recombining phrases and instrumentation in ways that the composer never imagined.
The company’s public debut will come this Thursday, as a LifeScore soundtrack accompanies the new season of Artificial, a scripted live-action series on the Twitch platform. Artificial is streamed live, with chose-your-adventure-style plot developments shaped by responses from the audience. With LifeScore, viewer reaction will affect the music as well.
But that’s only a prelude to the company’s ultimate ambition: providing meaningful musical experiences. Sheppard says that his biggest thrills as a composer come when the score that previously existed only in his mind and in musical notation are animated by musicians into sound. “As a creator that's where the goosebumps live,” he says. Now he wants to provide that for people in the context of their everyday activities. “I would love that to happen every time someone listens to music, if possible,” he says. “So they get the pleasure of almost being able to be the composer in that equation.” (Though the “composition” might come from traditionally non-creative sources like one’s pulse rate.)
But none of that can happen without another uncredited collaborator—AI. Gruber and his small team of engineers were charged with creating an engine that could accept inputs to reconstruct a composition on the fly—altering tone, tempo, and instrumentation—while somehow sounding as if it were composed that way from the start. It required teaching the LifeScore engine how to be a maestro. “There's musicology, there's theory behind it,” he says.
“It seems bizarre,” says Sheppard. “But at the same time, it's doable.”
So how does LifeScore do it? It starts, like any music, with a human creator. Both Sheppard and Gruber are emphatic that they do not want an algorithm that writes the score itself, as some previous efforts have attempted to do. “My whole career has been the idea—let's augment instead of automate,” says Gruber. “There's plenty of music in the world and plenty of musicians. But what we don't have is that experience where you're interacting with your music and helping you create it.”
Composers of a LifeScore theme must understand that they are not auteurs but collaborators, creating a substrate that will be animated by an AI engine and inputs from the listeners themselves. These creators aren’t writing self-contained symphonies or pop tunes, but pieces that can be endlessly recombined. Sheppard says it’s a musical equivalent to a Lego-brick kit. Groups of instruments might play certain measures in a way that could stand alone or be weaved into a more elaborate orchestration. A few hours of music in the studio could be stretched to thousands of hours of playback, with no duplication.
Sheppard demonstrates the end product to me via Zoom from his living room in London. In the background, I can see his cello leaning against a stuffed chair. He’s wearing a grey hoodie with neon yellow drawstrings. He begins a composition that would be appropriate for one of his forest walks. The neo-classical music playing on his iPhone is brisk and uplifting. He swings the phone to his left and the music responds to the gyroscope’s report. (Right now, the system doesn’t process biometric input but adding that doing so will be “trivial,” the company says.)
“The cello is coming over the top—it wasn’t written to go like this,” he says. We listen a bit more as the strings reach for the heavens. “I’ve not heard this music before,” says the composer.
"Artificial" is the perfect vehicle for testing LifeScore. The series is a Pygmalion-esque tale of an android and her creator in which the actors perform live and which already uses feedback from viewers to steer the plotlines. Executive producer Bernie Su, looking for a twist in the new season, says he was blown away by Sheppard’s LifeScore demo.
“Each character has a certain theme of music, like Peter and the Wolf,” explains Su. Based on what emotions people explicitly express in the chat channel, the musical themes move among four emotional classes: happy, sad, mysterious, and intense. (This blunt form of input will become more sophisticated over time—eventually, with increased AI recognition skills, LifeScore might be able to interpret the images, sounds and language of a scene well enough understand by itself which musical emotions should be expressed.) The AI, which understands how to translate those emotions into the language of music—slowing down at intense moments, for instance—changes the score accordingly. And each of those categories have three levels of intensity—a lot of feedback can ramp up the music by volume or the choice of instruments. Just like a composer, the AI engine understands what musical grammar evokes emotion from the audience. “What an elegant, sophisticated, non-invasive way to build the audience into the experience,” Su says. “Okay, we haven't done an episode yet, but I’m incredibly excited.”
The other big entertainment use case for LifeScore will be video games. Sheppard has composed many a soundtrack for the genre, and thinks that the LifeScore’s ability to stretch a few hours of studio time into a near infinite variety of musical moods and tunes will give it an edge over the current practice of endlessly looping parts of the score as players explore virtual worlds.
Creating aural spaces for business customers is the big market to break into. “Companies now are thinking of music as part of an environment that they're selling as a whole product,” says Gruber. “And so we're into that side of things."
The granddaddy of that business has been Muzak, which has become a pejorative term for anodyne, syrupy cover versions of vaguely popular tunes—more tranquilizer than energizer. Maybe that’s why the Muzak company changed its name in 2017 to Mood Media. It is a licensing behemoth, measuring its success by metrics like increased sales in stores that use one of its thousands of intricately programmed playlists of custom recorded tunes.
It would seem that LifeScore is in direct competition with Mood Music. But Sheppard and Gruber argue that the products exist in different universes. The Muzak experience is subliminal; LifeScore wants to transform the ambient music world into a goosebump machine. Sheppard enthusiastically recounts the time that he gave a demo to “an austere business person,” who burst into tears when moving the phone led to a certain unique musical sequence. “It was very sweet and, I think, quite embarrassing for him as well,” he says. (For the record, I remained dry-eyed throughout the demo.)
The company thinks that driving is an ideal LifeScore scenario, and says it’s negotiating with automakers. Inputs could include speed, road conditions, weather, and proximity to one’s destination. Another big market it hope to crack is wellness, with heart rate and other biometrics helping shape compositions. It has already recorded hours of its Lego-brick musician building kits with music appropriate for yoga, cardio workouts, and meditation. The company says it is working on analyzing huge data sets of biometrics and behavior to see how people react to musical signals, so eventually LifeScore will be able to learn how to adapt compositions “to optimize measures of well being such as relaxation, level of physical effort, or mental focus.”
Another scenario is that employers could pump in LifeScore as ambient music in offices, if we ever return to them. Gruber says that the music could, for instance, energize us when it gets input that our body posture is sagging. Unlike more blunt means of goosing productivity, he says, a LifeScore cue would be welcome. “If you're dealing with music," he says, "you can use things that are more humane than just straight out cattle-prod feedback.” I suspect, however, that the millions who now listen to their own music in the office via headphones might not find this so friendly. Nor will they necessarily welcome the cameras or other sensors that took note of their posture.
Sheppard says when he first started out in Abbey Road, he used Studio Two, where the ghosts of the Beatles linger. (The microphones that George Martin supplied to the group are still around and Sheppard used them whenever he could.) He also made forays to the cavernous Studio One, at one point gathering 90 musicians to record the background music for a video game. “But actually I’ve fallen in love with Studio Three,” he says. That’s where Pink Floyd recorded “The Dark Side of the Moon.” The musicians laid down the soundtrack for Artificial there.
Unlike the tunes that have emerged from these studios over the decades, these will never climb the charts. They won’t even be finished, until they are streamed into the speakers and headphones of listeners in living rooms, cars, yoga studios, and forests. Will LifeScore’s AI-augmented compositions be hits in their own way? Only if they deliver those goosebumps. Not to mention office productivity, in a friendly way.
- How a Chinese AI giant made chatting—and surveillance—easy
- The confessions of Marcus Hutchins, the hacker who saved the internet
- We’ll learn to sing together when we’re far apart
- 27 days in Tokyo Bay: What happened on the Diamond Princess
- Tips and tools for cutting your hair at home
- 👁 AI uncovers a potential Covid-19 treatment. Plus: Get the latest AI news
- 🏃🏽♀️ Want the best tools to get healthy? Check out our Gear team’s picks for the best fitness trackers, running gear (including shoes and socks), and best headphones
