Home / News & Analysis / AI Has Been Creating Music and the Results Are…Weird

AI Has Been Creating Music and the Results Are…Weird

In late May, a small crowd at St. Dunstan's church in East London's Stepney district gathered for two hours of traditional Irish music. But this event was different; the tunes it featured were composed, in part, by an artificial intelligence (AI) algorithm, dubbed folk-rnn, a stark reminder of how cutting-edge AI is gradually permeating every aspect of human life and culture—even creativity.

Developed by researchers at Kingston University and Queen Mary University of London, folk-rnn is one of numerous projects exploring the intersection of artificial intelligence and creative arts. Folk-rnn's performance was met with a mixture of fascination, awe, and consternation at seeing soulless machines conquering something widely considered to be the exclusive domain of human intelligence. But these expeditions are discovering new ways that man and machine can cooperate.

[youtube https://www.youtube.com/watch?v=HOPz71Bx714&w=560&h=315]

How Does AI Create Art?

Like many other AI products, folk-rnn uses machine learning algorithms, a subset of artificial intelligence. Instead of relying on predefined rules, machine learning ingests large data sets and creates mathematical representations of the patterns and correlations it finds, which it then uses to accomplish tasks.

Folk-rnn was trained with a crowd-sourced repertoire of 23,000 Irish music transcripts before starting to crank out its own tunes. Since its inception in 2015, folk-rnn has undergone three iterations and has produced more than 100,000 songs, many of which have been compiled in an 14-volume online compendium.

Flow Machines, a five-year project funded by the European Research Council and coordinated by Sony's Computer Science Labs, also applied AI algorithms to music. Its most notable—and bizarre—achievement is "Daddy's Car," a song generated by an algorithm that was trained with lead sheets from 40 of The Beatles' hit songs.

[youtube https://www.youtube.com/watch?v=LSHZ_b05W7o&w=560&h=315]

Welcome Mistakes

Algorithms can mimic the style and feel of a musical genre, but they often make basic mistakes a human composer would not. In fact, most of the pieces played at folk-rnn's debut were tweaked by human musicians.

"Art is not a well-defined problem, because you never know exactly what you want," says Francois Pachet, who served as the lead researcher at Flow Machines and is now director of Spotify's Creator Technology Research Lab. But, he adds cheerfully, "it's good actually that art is not well defined. Otherwise, it would not be art."

The generated lead sheet for "Daddy's Car" was also edited by a human musician, and some tracks were added by hand. "There was pretty much a lot of AI in there, but not everything," Pachet says, "including voice lyrics and structure, and of course the whole mix and production."

"The real benefit is coming up with sequences that aren't expected, and that lead to musically interesting ideas," says Bob Sturm, a lecturer in digital media at Queen Mary, University of London who worked on folk-rnn. "We want the system to create mistakes, but the right kind of mistakes."

[youtube https://www.youtube.com/watch?v=lZKc363886Y&w=560&h=315]

Daren Banarsë, an Irish musician who examined and played some of the tunes generated by folk-rnn, attested to the benefits of interesting mistakes. "There was one reel which intrigued me," he says. "The melody kept oscillating between major and minor, in a somewhat random fashion. Stylistically, it was incorrect, but it was quirky, something I wouldn't have thought of myself."

Spotify's Pachet explains that these unexpected twists can actually help improve the quality of pop music. "Take the 30 or 50 most popular songs on YouTube. If you look at the melody, the harmony, the rhythm and the structure, they are extremely conventional, which is quite depressing. You have only three or four chords, and they're always the same. Creative AI is very interesting, not only because it's fun, but also because it brings hope. I hope that we could change or impact the quality of the most popular songs today."

No Right Answers

"The thing that makes art wonderful for humanity is that there is no right answer—it's entirely subjective," says Drew Silverstein, CEO and co-founder of Amper Music, an AI startup based in New York. "You and I might listen to the exact same piece of music, and you might like it, and I might hate it, and neither of us is right or wrong. It's just different.

"The challenge in the modern world is to build an AI that is capable of reflecting that subjectivity," he adds. "Interestingly, sometimes, neural networks and purely data-driven approaches are not the right answer."

Oded Ben-Tal, senior lecturer in music technology at Kingston University and a researcher for folk-rnn, points out another challenge AI faces in respect to creating music: Data does not represent everything.

"In some ways, you can say music is information. We listen to a lot of music, and as a composer, I get inspired by what I hear to make new music," Ben-Tal says. "But the translation into data is a big stumbling block and a big problem in that analogy. Because no data actually captures all the music."

To put it simply, an AI algorithm's interpretation and understanding of music and arts is very different from that of humans.

"In the case of our system, it's far too easy to fall into the trap of saying it's learning the style or it's learning aspects of Irish music, when in fact it's not doing that," says Sturm. "It's learning very abstract representations of this kind of music. And these abstract representations have very little to do with how you experience the music, how a composer puts them together in the context of this music within the tradition.

"Humans are necessary in the pursuit because, at the end of the day, we have to make decisions on whether to incorporate certain things produced by the computer that we curate from this output and create new music," Sturm says.

Google's DeepDream

In visual arts, the divide between the perception of humans and machines is even more accentuated. For instance, take DeepDream, an inside-out version of Google's super-efficient image-classification algorithm. When you give it a photo, it looks for familiar patterns and modifies the image to look more like the things it has identified. This can be useful to turn rough sketches into more enhanced drawings, but it yields unexpected results when left to its own devices. If you provide DeepDream with an image of your face and it finds a pattern that looks like a dog, it'll turn a part of your face into a dog.

"It's almost like the neural net is hallucinating," an artist who interned at Google's DeepMind AI lab said about the software in an interview with Wired last year. "It sees dogs everywhere!"

But AI-generated art often looks stunning and can rake in thousands of dollars at auctions. At a San Francisco art show held last year, paintings created with the help of Google's DeepDream sold for up to $8,000.

The Business of Creative AI

While researchers and scientists continue to explore creative AI, a handful of startups have already moved into the space and are offering products that solve specific business use cases. One is Silverstein's Amper Music, which he describes as a "composer, producer, performer that creates unique professional music tailored to any content in a matter of seconds."

To create music with Amper, you specify the desired mood, length, and genre. The AI produces a basic composition in a few seconds that you can tweak and adjust. Amper also offers an application programming interface (API), so developers can incorporate the platform's creative power into their software.

[youtube https://www.youtube.com/watch?v=lyXrU_Qo6UQ&w=560&h=315]

Jukedeck, a London-based startup created by two former Cambridge University students, provides a similar service. Like Amper, users provide Jukedeck with basic parameters, and it provides them an original musical track.

The main customers of both companies are businesses that require "functional music," the type used in ads, video games, presentations, and YouTube videos. Jukedeck has created more than 500,000 tracks for customers including Coca-Cola, Google, and London's Natural History Museum. Composers are also learning to use the tools to enhance the music they create for their customers.

A third startup, Australia-based Popgun, is building an AI musician that can play music with humans. Named Alice, the AI listens to what you play and then responds instantly with a unique creation that fits with what you played.

[youtube https://www.youtube.com/watch?v=y_zUtY05TuM&w=560&h=315]

In the visual arts industry, business use cases are gradually emerging. Last year, Adobe introduced Sensei, an AI platform aimed at improving human creativity. Sensei assists artists in a number of ways, such as automatically removing the background of photos or finding stock images based on the context of a poster or sketch.

Collaboration Between AI and Human Artists

Perhaps not surprisingly, these startups are founded and managed by people who have strong backgrounds as artists. Amper's Silverstein studied music composition and theory at Vanderbilt University and composed music for TV, films, and video games. Ed Newton-Rex, founder and CEO of Jukedeck, is also a practiced music composer.

But not everyone is convinced of the positive role of artificial intelligence in arts. Some of the attendees at folk_rnn's event described the AI-generated pieces as lacking in "spirit, emotion and passion." Others expressed concerned for the "cultural impact and the loss of the human beauty and understanding of music."

"I haven't met one musician that I've told about this who hasn't reacted with something close to the negative side of things," said Úna Monaghan, a composer and researcher involved in folk-rnn who spoke to Inverse. "Their reaction has been from slightly negative, to outright 'why are you doing this?'"

The developers of creative AI algorithms do not generally share these concerns. "I don't think humans will become redundant in music-making," says Newton-Rex. "For a start, we as listeners care about much more than just the music we're listening to; we care about the artist, and about their story. That will always be the case."

Automation and IoT Predictions

"We think of functional music as music that is valued for its use case and not for the creativity or collaboration that went into making it," Silverstein says. But artistic music, Silverstein explains, "is much more about the process than the use case. Steven Spielberg and John Williams writing the score of Star Wars, that's about a human collaboration."

"The key use-cases we see lie in collaboration with musicians," says Jack Nolan, co-founder of Popgun. "Artists can use Alice as a source of creative inspiration or to help them come up with melodies and chord progressions in their music. We don't think people will ever stop wanting to create their own sounds. We think AI will help them do this, rather than replace them."

Daren Banarsë agrees on the benefits of collaboration. "I always find it daunting when I have to start a large-scale composition. Maybe I could give the computer a few parameters: the number of players, the mood, even the names of some of my favorite composers, and it could generate a basic structure for me," he says. "I wouldn't expect it to work out of the box, but it would be a starting point. Or it could output a selection of melodic ideas or chord progressions for me to look through. And somewhere in there, there's going to be a computer glitch or random quirk, which could take me in a completely unexpected direction."

Ben-Tal admits that some jobs might be affected. "Working musicians will have to adapt," he says. "I show this to my students and say, 'You need to up your game.' This will mean some of the entry-level jobs into the music industry will not be there in five or ten years, or you'll need to do things differently or have a different set of skills."

'Democratizing Creativity'

AI creativity can also help people without inherent talent or hard-earned skills express themselves artistically. Take Vincent's AI drawing platform, which helps transform rough sketches into professional-looking paintings, and the AI music platforms that create decent music with minimal input.

Related

Jukedeck's Newton-Rex describes this as "democratizing" creativity. "People with less formal musical education can get to grips with the basics of music and use AI to help them make music," he says.

Pachet concurs. He draws an analogy between recent AI developments and the arrival of the first digital synthesizers in the 80s, followed by digital samplers. At the time, there was a similar fear that musicians would lose their jobs to computers. "But what happened was the exact opposite, in a sense that everyone took these new machines and hardware with them and learned how to use them productively," he says. "The music industry exploded in some sense."

"There will be more people doing music, and hopefully more interesting music," he adds, reflecting back on AI creativity. "I cannot predict the future, but I'm not worried about AI replacing artists. I'm worried about all the other things, the well-defined problems, like automated healthcare and autonomous vehicles. These things are really going to destroy jobs. But for the creative domains, I don't think it's going to happen."

Read more

Check Also

Amazon’s Show Mode Dock makes the Echo Show mostly unnecessary

Google gave us a hardware blitzkrieg at CES. Among other things, the company announced a new smart display category, aimed at taking on the Echo Show through sheer, brute force. The new Show Mode Dock isn’t a direct response, but it’s a clever one. Two years ago, Amazon introduced Alexa for the Fire tablet line. Last year, the feature went hands-free. In June, all of those additions finally paid off with the addition of Show Mode for the Fire HD 8 and 10, along with the dock, which effectively turns the tablets into an Echo Show. It’s a perfect bit of stream-crossing synergy for the company. When I met with Amazon prior to release, I asked if the company was afraid of cannibalizing the Show. They seemed unconcerned. Not surprising, really. Hardware has always been secondary to its strategy. The more Alexa devices in the world, the better. That’s really the bottom line here. For consumers, the form factor makes sense. You can pick up the 8- and 10-inch bundle for $110 and $190, respectively, putting it considerably below the Show’s $230 MSRP (though Amazon sale prices do tend to fluctuate a fair bit). That cost is getting you not only a smart display, but a Fire tablet that can be unhooked and used in all of the standard tablet ways. In fact, the more I talk about it, the less compelling the Show becomes. It was never a particularly attractive piece of hardware for one that’s meant to be displayed in your home at all times. In fact, it’s got a bit of an unintentional retro RadioShack vibe. It’s also unnecessarily big and bulky — that’s part of what made the much smaller Spot that much more appealing. Given the new product category and some of the deep discounts it’s been getting in recent months, I wouldn’t be too surprised to see a new Show on the way in the not so distant future. In the meantime, however, the device does have a few things going for it versus the tablet/dock combo. Chief among them are better mics and speakers. Of course, you can always connect the tablet to a Bluetooth speaker (through the app, not over voice yet) to address the latter issue. But for now, if you’re looking for a screen-enabled device that can also double as a small entertainment hub, the Show is probably still a better bet. It’s worth pointing out, too, that neither the Fire Tablet nor the dock are what anyone would classify as premium devices. Amazon’s efforts to compete on the high end of the tablet market evaporated years ago. The new Fires have decent screens, but otherwise mostly fit the bill of content delivery devices. It’s a strategy that has worked quite well for Amazon, as much of the rest of the tablet category has dried up. There isn’t a lot to the dock itself. It’s a small bit of plastic with a kickstand that swivels out. There’s a plastic tablet case with two metal pads on the back that snap onto the dock with magnets. A small micro-USB module plugs into the tablet’s port, connecting the two, for data transfer and power, so it can charge while docked. The key to the whole thing is the addition of Show Mode to the tablet, bringing the same UI you get on the smart display. You can enable it manually on the device by swiping down on the home menu (strangely, this doesn’t seem to be enabled through voice yet). The Mode does away with all of the details of the standard Fire OS, instead defaulting to a large, card-based system. The Mode is also enabled when the tablet is docked. When you remove it, it reverts back to the standard tablet. Simple. It all works as advertised. Though again, the speakers aren’t great, and it’s not as good at picking up sounds across the room. Although $40 and $55 for the 8- and 10-inch dock, respectively, is a bit steep, taken together, it’s ultimately a better deal than the Show — and either way, you’re getting a screen larger than the smart display’s 7-inch. The Show Mode Dock/Fire Tablet combo is really just the all-around better deal. It also starts shipping next week — no word yet on when those Google displays are finally arriving.

Leave a Reply

Your email address will not be published. Required fields are marked *

Disclaimer: Trading in bitcoins or other digital currencies carries a high level of risk and can result in the total loss of the invested capital. theonlinetech.org does not provide investment advice, but only reflects its own opinion. Please ensure that if you trade or invest in bitcoins or other digital currencies (for example, investing in cloud mining services) you fully understand the risks involved! Please also note that some external links are affiliate links.