![\"Image\"](\"https:\/\/www.artefact.com\/\/wp-content\/uploads\/2024\/11\/Gabrielle-Chou-NYU-Shanghai.png\")
![\"Image\"](\"https:\/\/www.artefact.com\/\/wp-content\/uploads\/2024\/11\/Nicolas-Lang.jpg\")
From Studio to Streaming: AI Influence in Shaping Music and Its Market Impact<\/h2><\/div>The first documented live performance blending generative AI and a musician took place in November 2022, when pianist David Dolan (Guildhall School of Music and Drama) improvised in dialogue with a semi-autonomous AI system designed by composer Oded Ben-Tal (Kingston University). This groundbreaking concert demonstrated the creative possibilities of AI as a collaborative partner in music. Since then, musicians and even non-musicians have embraced generative AI, producing full compositions with vocals, instruments, and even sounds from non-existent instruments. This marks the beginning of a new era where AI is reshaping music creation and production<\/a>. As artificial intelligence continues to reshape industries, the music world is experiencing its own revolution. From hobbyists to major record labels, stakeholders across the industry are grappling with the implications of this technology.<\/p>\n<\/div>The music industry, historically shaped by technological advancements, is entering a new phase of transformation. With the music industry\u2019s broad impact across entertainment, media, and streaming platforms hosting millions of tracks, the integration of AI promises to redefine how music is created, produced, and consumed.<\/p>\n<\/div>
Generative AI, the technology behind this seismic shift, has rapidly evolved from academic research to practical applications. Models like Music Transformer and MusicLM are pushing the boundaries of what’s possible, translating abstract concepts into harmonious compositions. These AI systems, trained on vast datasets of musical pieces, can now generate original melodies, harmonies, and even entire songs with astonishing proficiency.<\/p>\n<\/div>
However, as with any technological revolution, the rise of AI in music brings both excitement and anxiety. Artists and industry professionals are divided on its potential impact. Some see it as a powerful tool for creativity and democratization, while others worry about the implications for human artistry and job security.<\/p>\n<\/div>
The landscape of the music industry is already changing. Streaming platforms are leveraging AI for personalized playlists, while production software is incorporating AI-assisted mixing and mastering tools. Even major record labels are exploring AI’s potential for talent scouting and trend prediction.<\/p>\n<\/div>
As we delve into this new era, questions abound. How will AI reshape the roles of artists, producers, and engineers? What legal and ethical considerations must we address? And perhaps most importantly, how will this technology affect the emotional connection between musicians and their audience?<\/p>\n<\/div>
In this article, we’ll explore the current state of generative AI in music, its applications across different sectors of the industry, and the potential future it heralds. From the technology behind these AI models to the practical challenges of bringing them to market, we’ll examine how this innovation is poised to transform one of the world’s most beloved and influential creative industries.<\/p>\n<\/div>
As the music industry enters this AI-driven era, it’s evident that future composers may not be solely human, marking a significant shift as the industry navigates this new frontier.<\/p>\n<\/div>
1. The Technology Behind Generative AI for Music<\/h2><\/div>From Composition to Code: The Breakthroughs of Generative AI in Music<\/strong><\/h3><\/div>The journey from traditional composition to AI-generated music has been nothing short of ground breaking. We are familiar with text generation using chatGPT and image generation with Stable diffusion and MidJourney. Music generation major breakthroughs lie in the adaptation of those powerful models to the specificities of music data.<\/p>\n<\/div>
I. GPT for Music (discrete generation)<\/span><\/h3><\/div>Just as GPT models process language, similar principles can be applied to musical elements. The key lies in tokenization\u2014breaking down music into discrete, manageable units. The process of applying GPT models to music through tokenization is a crucial step in enabling AI to understand and generate music. Here’s a more detailed explanation:<\/p>\n<\/div>
1. Tokenization in music: Just as GPT models break down text into tokens (words or subwords), music needs to be broken down into discrete units that the AI can process. We explore below two tokenization methods:<\/p>\n<\/div>
- <\/i><\/span>\n
Music Sheet: A note-level representation of music<\/p>\n<\/div><\/li><\/ul>
![\"Music](\"https:\/\/www.artefact.com\/\/wp-content\/uploads\/2024\/11\/image-1-article-Gen-AI-in-Music.png\" "\\"Music")
<\/span><\/div>i. Notes can be seen as discrete element containing multiple informations :
\na) Pitch: The specific note being played (e.g., C, D, E, etc.)
\nb) Duration: How long the note is held (e.g., quarter note, half note)
\nc) Velocity: How loudly or softly the note is played
\nd) Instrument: Which instrument is playing the note<\/p>\n
ii. Example : a C4 note played for a quarter beat might be represented with tokens : ; ; , while a G3 note played for a half beat might be tokenized into ; ;
\n( we can add tokens for velocities, bar structures, instruments, genres,…)<\/p>\n
iii. Numerical Sequence creation:
\nThese series of notes-tokens are then concatenated and arranged in sequences, similar to how words form sentences in language models. Each token is assigned a unique numerical ID, enabling the GPT model to receive a similar input as for text generation.<\/p>\n
iv. Example : A musical phrase looks like: [123; 8; 456; 118; 12; 451;…] where each number represents a specific musical event.<\/p>\n<\/div>
- <\/i><\/span>\n
Music as continuous waves (MusicLM)<\/p>\n<\/div><\/li><\/ul>
![\"Graph](\"https:\/\/www.artefact.com\/\/wp-content\/uploads\/2024\/11\/image-2-article-Gen-AI-in-Music.png\" "\\"Graph")
<\/span><\/div>i. Audio files are stored in computers in a wave format, representing the amplitude of the signal over time. Which provides a richer representation than simple music sheets.<\/p>\n
ii. This wave can be divided into segments representing the \u201cmeaning of music\u201d using a trained model (similar to BERT for word vectors) which creates a tokenized version of the audio input.<\/p>\n
iii. The resulting vectors can be conditioned by a joint audio and text embedding, creating tokens representing both audio characteristics and textual description, enabling the user to use a text-prompt as input during inference.<\/p>\n<\/div>
2. AI training:<\/p>\n<\/div>
- <\/i><\/span>\n
Our two tokenization methods give a sense of musical grammar and vocabulary. This will allow the model to understand the structure, harmony, rhythm, and other aspects of music.<\/p>\n<\/div><\/li>
- <\/i><\/span>\n
The AI model, similar to GPT, can then process these sequences of numbers, learning patterns and relationships between different musical elements by trying to reconstruct a song, exactly as it would do for text generation.<\/p>\n<\/div><\/li><\/ul>
3. Generation:<\/p>\n<\/div>
- <\/i><\/span>\n
When generating new songs, the AI predicts the next likely token in a sequence, similar to how it predicts the next word in a sentence for text generation.<\/p>\n<\/div><\/li>
- <\/i><\/span>\n
These predicted tokens can then be converted back into musical notation or audio.<\/p>\n<\/div><\/li><\/ul>
Music Transformer, developed by Google’s Magenta project, marked a significant milestone in this field. By being among the first to apply the Transformer\u2019s Attention mechanism\u2014a cornerstone of language models\u2014to musical sequences, it achieved remarkable coherence in long-form compositions. Other models followed suit, each pushing the boundaries of what AI could accomplish in music creation.<\/p>\n<\/div>
II. Diffusion models for music (Continuous generation)<\/span><\/h3><\/div>Diffusion models, such as MidJourney or Stable Diffusion, have been largely adopted and used for image generation. The generated images are getting more and more detailed and customizable showing great consistency while adding style or resolution details in a prompt. This personalization is an interesting feature for music generation as well. How can we turn music into an image?<\/p>\n<\/div>
Music as a picture:<\/strong><\/p>\n<\/div>![\"Graph:](\"https:\/\/www.artefact.com\/\/wp-content\/uploads\/2024\/11\/image-3-article-Gen-AI-in-Music.png\" "\\"Graph:")
<\/span><\/div>- <\/i><\/span>\n
A spectrogram is a visual representation of the spectrum of frequencies in a sound or musical pice as they vary with time. It uses a color scale to represent the amplitude of a frequency at a certain time. It is similar to a 2D image with color patterns.<\/p>\n<\/div><\/li>
- <\/i><\/span>\n
With an added text description, we have the same pairs of image & description used to train diffusion models. We can directly use them to generate pictures of spectrograms conditioned by a text description. The generated spectrograms can later be converted back into audio.<\/p>\n<\/div><\/li>
- <\/i><\/span>\n
Diffusion models for pictures proved to re-use existing image assets during generation, posing an ownership rights problem. Such a problem is crucial to music which is why MusicLDM (Latent Diffusion Model)\u00a0authors proposed to tackle this by adding synthetic data generated by other AI models to diversify their training music dataset.<\/p>\n<\/div><\/li><\/ul>
By considering spectrograms as pictures and aligning them with a text description, it is possible to have a powerful control over the generation, enabling users to prompt their desired music as they would do for pictures with MidJourney.<\/p>\n<\/div>
Bridging the Gap from Innovation to Industry: Challenges in AI Music Generation:<\/strong>
\nWe have explored 2 different ways to generate music using AI. All of these technologies led to impressive and entertaining demos. However, the path from laboratory experiments to market-ready products is fraught with challenges. Technical hurdles abound, from ensuring consistent quality across various musical styles to managing the computational demands of real-time generation. Market challenges are equally daunting, as developers grapple with finding the right audience and use cases for their AI composers.
\nManagement and legal considerations add further complexity. Building and leading teams at the intersection of music and AI requires a unique blend of skills. Meanwhile, the ethical and legal landscape remains largely uncharted, raising questions about copyright, authorship, and the very nature of creativity in the age of AI.
\nAs we delve deeper into the current applications and products emerging from this technological revolution, it becomes clear that the impact of AI on the music industry will be far-reaching, touching every stakeholder from listeners to major record labels.<\/p>\n<\/div>2. Current Applications and Products<\/h2><\/div>As generative AI continues to develop, a variety of platforms and tools have emerged in the music industry. These innovations are moving from experimental concepts to practical applications, each offering different capabilities and potential uses.<\/p>\n<\/div>
I. The most used today: Suno.AI<\/span><\/h3><\/div>In the landscape of AI music generation, Suno.AI has gained significant attention. The platform allows users to create music from text prompts, generating full songs including vocals, instrumentation, and lyrics in various genres and styles.<\/p>\n<\/div>
Suno.AI has reported a user base of 12 million and completed successful funding rounds, reaching a valuation of $500 million. The platform offers both free and paid tiers, with the free version allowing users to create a limited number of songs per day. Its applications range from film trailer mockups to video game soundtracks.
\nA filmmaker working on a short film might use Suno.AI to generate a custom soundtrack when unable to hire a composer. Advertising agencies are exploring the platform to quickly create jingles and background music for commercials, potentially altering their workflow for music in advertising.<\/p>\n<\/div>
II. Other AI Music Platforms<\/span><\/h3><\/div>While Suno.AI has garnered attention, it’s part of a growing ecosystem of AI music platforms.<\/p>\n<\/div>
- <\/i><\/span>\n
Boomy AI:<\/strong> This platform focuses on enabling users to create and monetize AI-generated music. Users can upload their AI-assisted creations to streaming platforms, potentially earning royalties.<\/p>\n<\/div><\/li>- <\/i><\/span>\n
AIVA:<\/strong> Recognized as an official composer by France’s music rights society SACEM, AIVA represents a step towards legetimizing AI-composed music. Some established composers are using AIVA to expedite their workflow, generating initial themes that they then elaborate on and orchestrate.<\/p>\n<\/div><\/li>- <\/i><\/span>\n
Endel:<\/strong> This platform takes a different approach by creating personalized, adaptive soundscapes for relaxation and focus. It generates custom background music that responds to factors such as heart rate, time of day, and weather.<\/p>\n<\/div><\/li>- <\/i><\/span>\n
MuseNet and Google’s Magenta Project<\/strong> are research-focused initiatives exploring the boundaries of machine-generated compositions. These projects are investigating fundamental questions about creativity, style, and musical intelligence.<\/p>\n<\/div><\/li>- <\/i><\/span>\n
Amper Music<\/strong> caters to content creators who need quick, rights-cleared music for their projects. It allows for the generation of background tracks for various media content, addressing time constraints and copyright concerns.<\/p>\n<\/div><\/li><\/ul>As these platforms evolve and new ones emerge, the applications of AI in music continue to diversify. From individual creators to major corporations, the potential uses span a wide range, indicating a future where AI could play an increasingly significant role in music creation and production.<\/p>\n<\/div>
3. Impact on Different Stakeholders<\/h2><\/div>As generative AI continues to evolve, its impact reverberates through every facet of the music industry. From listeners to major record labels, each group is experiencing a shift in their relationship with music creation and consumption. While some view these changes with anxieties, others see them as the natural progression of an industry that has always been shaped by technological advancements.<\/p>\n<\/div>
I. Listeners<\/span><\/h3><\/div>The journey for music consumers has been one of increasing personalization and discovery. Before AI, listeners relied on radio DJs, music magazines, and word-of-mouth to discover new music. Playlists were painstakingly curated by hand, often limited by personal music libraries.<\/p>\n<\/div>
With the advent of AI, platforms like Spotify introduced features such as Discover Weekly in July 2015<\/a>, using algorithms to analyze listening habits and suggest new tracks. This marked a significant leap in personalized music discovery, exposing listeners to a broader range of artists and genres.<\/p>\n<\/div>Now, with generative AI, we’re on the verge of an even more tailored experience. LifeScore<\/a> is a system that not only recommends existing music but creates new compositions based on a listener’s mood, activity, or even biometric data. A jogger might have a custom soundtrack generated in real-time, adapting to their pace and heart rate, blending their favorite genres into something entirely new.<\/p>\n<\/div>II. Artists<\/span><\/h3><\/div>For musicians, the creative process has always been influenced by available tools and technologies. Before AI, composition relied heavily on traditional instruments and recording techniques. The introduction of synthesizers and digital audio workstations (DAWs) in the late 20th century already began to reshape how music was created. As noted by Jean-Michel Jarre<\/a>, an early electronic music pioneer :”Technology has always dictated styles and not the other way around. It\u2019s because we invented the violin that Vivaldi made the music he made… For me, AI is not necessarily a danger if it is used well”.<\/p>\n<\/div>Composition software can suggest chord progressions or melodies, serving as a collaborative partner in the creative process.<\/p>\n<\/div>
With generative AI, we’re seeing the emergence of AI co-writing. Artists can now input a basic melody or lyrical idea and have an AI system generate complementary elements, potentially sparking new creative directions. Some musicians are even experimenting with real-time style adaptation during live performances<\/a>, using AI to morph their sound on the fly based on audience reactions.<\/p>\n<\/div>For amateur artists, multiple training apps are making good use of AI to propose real-time, interactive and personalized learning experiences. With other tools enabling you to simulate arrangements and bands, the learning cost of creativity is reduced, enabling more people to play and enjoy music.<\/p>\n<\/div>
III. Engineers<\/span><\/h3><\/div>The role of audio engineers has been one of constant adaptation to new technologies. Traditionally, mixing and mastering were entirely manual processes, requiring a keen ear and years of experience. Interrogated for this article, Marius Blanchard, Data Scientist at Spotify’s mentioned: “AI is not here to replace musicians, but to empower them. It’s a tool that can help artists explore new creative territories and push the boundaries of music creation.”<\/p>\n<\/div>
AI has already made inroads in this domain, with tools like iZotope’s Neutron<\/a> offering intelligent mixing assistance. These systems can analyze a track and suggest EQ settings, compression levels, and other parameters, streamlining the mixing process.<\/p>\n<\/div>As generative AI becomes more sophisticated, we may see fully automated mixing and mastering systems that can process an entire album with minimal human intervention. However, many in the industry argue that the human touch will always be necessary to capture the emotional nuances of a performance.<\/p>\n<\/div>
IV. Music & Platform Companies<\/span><\/h3><\/div>Record labels and music publishers have long relied on human intuition and market research to spot trends and discover new talent. A&R (Artists and Repertoire) executives would scour clubs and demo tapes, looking for the next big thing.<\/p>\n<\/div>
The integration of AI has shifted this process towards data-driven decision making. Platforms analyze streaming data, social media engagement, and other metrics to identify rising artists and predict hit potential.<\/p>\n<\/div>
With generative AI, music companies are exploring even more radical possibilities. Some are experimenting with AI systems that can generate music tailored to specific market segments or demographics. Others are using predictive models to forecast which song structures or lyrical themes are likely to resonate with audiences in the coming months.<\/p>\n<\/div>
As the industry grapples with these changes, the key question remains: How will the balance between human creativity and AI assistance evolve? While some fear the loss of human touch in music creation, others see AI as a tool that will ultimately enhance human creativity, opening new avenues for expression and collaboration.<\/p>\n<\/div>
4. Future Visions and Opportunities<\/h2><\/div>As generative AI continues to reshape the music industry, stakeholders are grappling with a complex landscape of legal, economic, and ethical considerations. This transformative technology promises to democratize music creation while simultaneously challenging traditional notions of artistry and copyright.<\/p>\n<\/div>
I. Legal & Market Considerations<\/span><\/h3><\/div>1. Copyright in the Age of AI<\/p>\n<\/div>
Before AI’s emergence, copyright laws in music were relatively straightforward. Composers and performers held rights to their original works, with clear guidelines on sampling and fair use. However, the introduction of AI-generated music has blurred these lines significantly.<\/p>\n<\/div>
In the U.S., copyright protection extends to original works of authorship, while European doctrine emphasizes the creator’s personal touch. This distinction becomes crucial when considering AI-generated music. The U.S. Copyright Office has stated that it will not register works produced by a machine or mere mechanical process that operates randomly or automatically without creative input or intervention from a human author.<\/p>\n<\/div>
In 2023, at least one high-profile case highlighted copyright concerns around AI-generated music mimicking popular artists. A track called “Heart on My Sleeve” that used AI to imitate the voices of Drake and The Weeknd was uploaded to streaming services and then promptly removed. Universal Music filed a claim on YouTube, and the track’s video was taken down. This incident raised issues around personality rights and the unauthorized use of artists’ voices<\/a>, even if created through AI This action set a precedent for how the industry might handle AI-created content that closely resembles existing artists’ work.<\/p>\n<\/div>For artists, AI presents a double-edged sword. Some embrace the technology as a collaborative tool, while others fear it may devalue their craft.<\/p>\n<\/div>
Music companies face their own set of challenges.<\/p>\n<\/div>
2. Market Projections<\/p>\n<\/div>
AI in the music market is poised for significant growth. Industry analysts project the sector<\/a> to reach $38.7 billion by 2033, up from $3.9 billion in 2023.This tenfold increase reflects the anticipated widespread adoption of AI tools across various aspects of music production, distribution, and consumption. Interestingly, artists are more looking for AI powered tools<\/a> in music production and music mastering (66%) rather than for AI music generation (47%). Creativity-enabling tools are seen as more important than pure AI-generation tools, even if we have to prepare ourselves to listen to more and more AI-generated music extracts.<\/p>\n<\/div>II. Potential Transformations<\/span><\/h3><\/div>1. Democratization of Music Production<\/p>\n<\/div>
AI is lowering the barriers to entry for music creation. Tools that once required expensive studios and years of training are now accessible through user-friendly apps. This democratization could lead to an explosion of new musical styles and voices.<\/p>\n<\/div>
A music tech startup founder, Alex Mitchell, CEO of Boomy, envisions a future where anyone can be a composer. “Now, we’re seeing people all over the world create instant songs with Boomy, release them, and even earn royalty share income. For the first time, musical expression is available to an entirely new type of creator and audience<\/a>” . This accessibility could fundamentally alter the music landscape, potentially unearthing talent that might otherwise have gone undiscovered.<\/p>\n<\/div>2. New Artist-Audience Interactions<\/p>\n<\/div>
AI is indeed reshaping how artists connect with their audience. Some artists and music platforms are already experimenting with AI-powered chatbots that allow fans to interact with their music catalog in novel ways. For example, Spotify introduced an AI-driven chatbot within its application that interacts with users to understand their music preferences, moods, and listening habits. This allows the chatbot to curate customized playlists and suggest new tracks tailored to each user’s unique taste.<\/p>\n<\/div>
These AI chatbots are being used to enhance fan engagement by providing personalized recommendations, real-time updates, and interactive experiences. They can offer artist updates, recommend playlists, facilitate ticket purchases, and manage fan inquiries, fostering deeper connections between artists and their audience.<\/p>\n<\/div>
A forward-thinking artist, Holly Herndon, recently launched an AI version of her voice that fans can collaborate with to create personalized tracks. This blending of human creativity and AI capabilities points to a future where the line between artist and audience becomes increasingly fluid.<\/p>\n<\/div>
A Multifaceted Conclusion<\/h2><\/div>I. AI’s Role in Creativity<\/span><\/h3><\/div>The integration of AI in music creation raises profound questions about the nature of artistry. While some argue that AI is merely a tool, akin to a new instrument, others fear it may diminish the human element that makes music emotionally resonant.<\/p>\n<\/div>
Marius Blanchard, Data Scientist from Columbia University and working in the music industry offers a balanced perspective: “I do not believe AI will ever replace artists; […] the audience-artist connection lives on many different levels beyond the music. Instead, I hope that AI will democratize access to quality sound production and be a creativity boost for more and more artists to express and share their creations.”<\/p>\n<\/div>
II. Impact on Industry Jobs<\/span><\/h3><\/div>As AI capabilities grow, certain roles within the music industry may evolve or become obsolete. Sound engineers, for instance, may need to adapt their skills to work alongside AI mixing and mastering tools. However, new job categories are likely to emerge, such as AI music programmers or ethical AI consultants for the creative industries.<\/p>\n<\/div>
III. Authenticity and Emotional Connection<\/span><\/h3><\/div>Perhaps the most significant question facing the industry is whether AI-generated music can forge the same emotional connections with listeners as human-created works. While AI can analyze patterns and create technically proficient compositions, the ineffable quality of human experience that often infuses great music remains a challenge to replicate.<\/p>\n<\/div>
Electronic musician and AI researcher Holly Herndon offers an optimistic view<\/a>: “I think the best way forward is for artists to lean into developments with machine learning,” she said, suggesting that they “think of ways to conditionally invite others to experiment with them.\u201d<\/p>\n<\/div>References<\/h2><\/div>Jeff Ens and Philippe Pasquier, (2020). MMM : Exploring Conditional Multi-Track
\nMusic Generation with the Transformer. arXiv preprint arXiv:2008.06048 https:\/\/arxiv.org\/abs\/2008.06048<\/a><\/p>\nAndrea Agostinelli et al, (2023). MusicLM: Generating Music From Text. arXiv preprint arXiv:2301.11325 https:\/\/arxiv.org\/abs\/2301.11325<\/a><\/p>\nHuang et al, (2018). Music Transformer: Generating Music with Long-Term Structure. arXiv preprint arXiv:1809.04281 https:\/\/arxiv.org\/abs\/1809.04281<\/a><\/p>\n
The first documented live performance blending generative AI and a musician took place in November 2022, when pianist David Dolan (Guildhall School of Music and Drama) improvised in dialogue with a semi-autonomous AI system designed by composer Oded Ben-Tal (Kingston University). This groundbreaking concert demonstrated the creative possibilities of AI as a collaborative partner in music. Since then, musicians and even non-musicians have embraced generative AI, producing full compositions with vocals, instruments, and even sounds from non-existent instruments. This marks the beginning of a new era where AI is reshaping music creation and production<\/a>. As artificial intelligence continues to reshape industries, the music world is experiencing its own revolution. From hobbyists to major record labels, stakeholders across the industry are grappling with the implications of this technology.<\/p>\n<\/div> The music industry, historically shaped by technological advancements, is entering a new phase of transformation. With the music industry\u2019s broad impact across entertainment, media, and streaming platforms hosting millions of tracks, the integration of AI promises to redefine how music is created, produced, and consumed.<\/p>\n<\/div> Generative AI, the technology behind this seismic shift, has rapidly evolved from academic research to practical applications. Models like Music Transformer and MusicLM are pushing the boundaries of what’s possible, translating abstract concepts into harmonious compositions. These AI systems, trained on vast datasets of musical pieces, can now generate original melodies, harmonies, and even entire songs with astonishing proficiency.<\/p>\n<\/div> However, as with any technological revolution, the rise of AI in music brings both excitement and anxiety. Artists and industry professionals are divided on its potential impact. Some see it as a powerful tool for creativity and democratization, while others worry about the implications for human artistry and job security.<\/p>\n<\/div> The landscape of the music industry is already changing. Streaming platforms are leveraging AI for personalized playlists, while production software is incorporating AI-assisted mixing and mastering tools. Even major record labels are exploring AI’s potential for talent scouting and trend prediction.<\/p>\n<\/div> As we delve into this new era, questions abound. How will AI reshape the roles of artists, producers, and engineers? What legal and ethical considerations must we address? And perhaps most importantly, how will this technology affect the emotional connection between musicians and their audience?<\/p>\n<\/div> In this article, we’ll explore the current state of generative AI in music, its applications across different sectors of the industry, and the potential future it heralds. From the technology behind these AI models to the practical challenges of bringing them to market, we’ll examine how this innovation is poised to transform one of the world’s most beloved and influential creative industries.<\/p>\n<\/div> As the music industry enters this AI-driven era, it’s evident that future composers may not be solely human, marking a significant shift as the industry navigates this new frontier.<\/p>\n<\/div> The journey from traditional composition to AI-generated music has been nothing short of ground breaking. We are familiar with text generation using chatGPT and image generation with Stable diffusion and MidJourney. Music generation major breakthroughs lie in the adaptation of those powerful models to the specificities of music data.<\/p>\n<\/div> Just as GPT models process language, similar principles can be applied to musical elements. The key lies in tokenization\u2014breaking down music into discrete, manageable units. The process of applying GPT models to music through tokenization is a crucial step in enabling AI to understand and generate music. Here’s a more detailed explanation:<\/p>\n<\/div> 1. Tokenization in music: Just as GPT models break down text into tokens (words or subwords), music needs to be broken down into discrete units that the AI can process. We explore below two tokenization methods:<\/p>\n<\/div> Music Sheet: A note-level representation of music<\/p>\n<\/div><\/li><\/ul> i. Notes can be seen as discrete element containing multiple informations : ii. Example : a C4 note played for a quarter beat might be represented with tokens : ; ; , while a G3 note played for a half beat might be tokenized into ; ; iii. Numerical Sequence creation: iv. Example : A musical phrase looks like: [123; 8; 456; 118; 12; 451;…] where each number represents a specific musical event.<\/p>\n<\/div> Music as continuous waves (MusicLM)<\/p>\n<\/div><\/li><\/ul> i. Audio files are stored in computers in a wave format, representing the amplitude of the signal over time. Which provides a richer representation than simple music sheets.<\/p>\n ii. This wave can be divided into segments representing the \u201cmeaning of music\u201d using a trained model (similar to BERT for word vectors) which creates a tokenized version of the audio input.<\/p>\n iii. The resulting vectors can be conditioned by a joint audio and text embedding, creating tokens representing both audio characteristics and textual description, enabling the user to use a text-prompt as input during inference.<\/p>\n<\/div> 2. AI training:<\/p>\n<\/div> Our two tokenization methods give a sense of musical grammar and vocabulary. This will allow the model to understand the structure, harmony, rhythm, and other aspects of music.<\/p>\n<\/div><\/li> The AI model, similar to GPT, can then process these sequences of numbers, learning patterns and relationships between different musical elements by trying to reconstruct a song, exactly as it would do for text generation.<\/p>\n<\/div><\/li><\/ul> 3. Generation:<\/p>\n<\/div> When generating new songs, the AI predicts the next likely token in a sequence, similar to how it predicts the next word in a sentence for text generation.<\/p>\n<\/div><\/li> These predicted tokens can then be converted back into musical notation or audio.<\/p>\n<\/div><\/li><\/ul> Music Transformer, developed by Google’s Magenta project, marked a significant milestone in this field. By being among the first to apply the Transformer\u2019s Attention mechanism\u2014a cornerstone of language models\u2014to musical sequences, it achieved remarkable coherence in long-form compositions. Other models followed suit, each pushing the boundaries of what AI could accomplish in music creation.<\/p>\n<\/div> Diffusion models, such as MidJourney or Stable Diffusion, have been largely adopted and used for image generation. The generated images are getting more and more detailed and customizable showing great consistency while adding style or resolution details in a prompt. This personalization is an interesting feature for music generation as well. How can we turn music into an image?<\/p>\n<\/div> Music as a picture:<\/strong><\/p>\n<\/div> A spectrogram is a visual representation of the spectrum of frequencies in a sound or musical pice as they vary with time. It uses a color scale to represent the amplitude of a frequency at a certain time. It is similar to a 2D image with color patterns.<\/p>\n<\/div><\/li> With an added text description, we have the same pairs of image & description used to train diffusion models. We can directly use them to generate pictures of spectrograms conditioned by a text description. The generated spectrograms can later be converted back into audio.<\/p>\n<\/div><\/li> Diffusion models for pictures proved to re-use existing image assets during generation, posing an ownership rights problem. Such a problem is crucial to music which is why MusicLDM (Latent Diffusion Model)\u00a0authors proposed to tackle this by adding synthetic data generated by other AI models to diversify their training music dataset.<\/p>\n<\/div><\/li><\/ul> By considering spectrograms as pictures and aligning them with a text description, it is possible to have a powerful control over the generation, enabling users to prompt their desired music as they would do for pictures with MidJourney.<\/p>\n<\/div> Bridging the Gap from Innovation to Industry: Challenges in AI Music Generation:<\/strong> As generative AI continues to develop, a variety of platforms and tools have emerged in the music industry. These innovations are moving from experimental concepts to practical applications, each offering different capabilities and potential uses.<\/p>\n<\/div> In the landscape of AI music generation, Suno.AI has gained significant attention. The platform allows users to create music from text prompts, generating full songs including vocals, instrumentation, and lyrics in various genres and styles.<\/p>\n<\/div> Suno.AI has reported a user base of 12 million and completed successful funding rounds, reaching a valuation of $500 million. The platform offers both free and paid tiers, with the free version allowing users to create a limited number of songs per day. Its applications range from film trailer mockups to video game soundtracks. While Suno.AI has garnered attention, it’s part of a growing ecosystem of AI music platforms.<\/p>\n<\/div> Boomy AI:<\/strong> This platform focuses on enabling users to create and monetize AI-generated music. Users can upload their AI-assisted creations to streaming platforms, potentially earning royalties.<\/p>\n<\/div><\/li> AIVA:<\/strong> Recognized as an official composer by France’s music rights society SACEM, AIVA represents a step towards legetimizing AI-composed music. Some established composers are using AIVA to expedite their workflow, generating initial themes that they then elaborate on and orchestrate.<\/p>\n<\/div><\/li> Endel:<\/strong> This platform takes a different approach by creating personalized, adaptive soundscapes for relaxation and focus. It generates custom background music that responds to factors such as heart rate, time of day, and weather.<\/p>\n<\/div><\/li> MuseNet and Google’s Magenta Project<\/strong> are research-focused initiatives exploring the boundaries of machine-generated compositions. These projects are investigating fundamental questions about creativity, style, and musical intelligence.<\/p>\n<\/div><\/li> Amper Music<\/strong> caters to content creators who need quick, rights-cleared music for their projects. It allows for the generation of background tracks for various media content, addressing time constraints and copyright concerns.<\/p>\n<\/div><\/li><\/ul> As these platforms evolve and new ones emerge, the applications of AI in music continue to diversify. From individual creators to major corporations, the potential uses span a wide range, indicating a future where AI could play an increasingly significant role in music creation and production.<\/p>\n<\/div> As generative AI continues to evolve, its impact reverberates through every facet of the music industry. From listeners to major record labels, each group is experiencing a shift in their relationship with music creation and consumption. While some view these changes with anxieties, others see them as the natural progression of an industry that has always been shaped by technological advancements.<\/p>\n<\/div> The journey for music consumers has been one of increasing personalization and discovery. Before AI, listeners relied on radio DJs, music magazines, and word-of-mouth to discover new music. Playlists were painstakingly curated by hand, often limited by personal music libraries.<\/p>\n<\/div> With the advent of AI, platforms like Spotify introduced features such as Discover Weekly in July 2015<\/a>, using algorithms to analyze listening habits and suggest new tracks. This marked a significant leap in personalized music discovery, exposing listeners to a broader range of artists and genres.<\/p>\n<\/div> Now, with generative AI, we’re on the verge of an even more tailored experience. LifeScore<\/a> is a system that not only recommends existing music but creates new compositions based on a listener’s mood, activity, or even biometric data. A jogger might have a custom soundtrack generated in real-time, adapting to their pace and heart rate, blending their favorite genres into something entirely new.<\/p>\n<\/div> For musicians, the creative process has always been influenced by available tools and technologies. Before AI, composition relied heavily on traditional instruments and recording techniques. The introduction of synthesizers and digital audio workstations (DAWs) in the late 20th century already began to reshape how music was created. As noted by Jean-Michel Jarre<\/a>, an early electronic music pioneer :”Technology has always dictated styles and not the other way around. It\u2019s because we invented the violin that Vivaldi made the music he made… For me, AI is not necessarily a danger if it is used well”.<\/p>\n<\/div> Composition software can suggest chord progressions or melodies, serving as a collaborative partner in the creative process.<\/p>\n<\/div> With generative AI, we’re seeing the emergence of AI co-writing. Artists can now input a basic melody or lyrical idea and have an AI system generate complementary elements, potentially sparking new creative directions. Some musicians are even experimenting with real-time style adaptation during live performances<\/a>, using AI to morph their sound on the fly based on audience reactions.<\/p>\n<\/div> For amateur artists, multiple training apps are making good use of AI to propose real-time, interactive and personalized learning experiences. With other tools enabling you to simulate arrangements and bands, the learning cost of creativity is reduced, enabling more people to play and enjoy music.<\/p>\n<\/div> The role of audio engineers has been one of constant adaptation to new technologies. Traditionally, mixing and mastering were entirely manual processes, requiring a keen ear and years of experience. Interrogated for this article, Marius Blanchard, Data Scientist at Spotify’s mentioned: “AI is not here to replace musicians, but to empower them. It’s a tool that can help artists explore new creative territories and push the boundaries of music creation.”<\/p>\n<\/div> AI has already made inroads in this domain, with tools like iZotope’s Neutron<\/a> offering intelligent mixing assistance. These systems can analyze a track and suggest EQ settings, compression levels, and other parameters, streamlining the mixing process.<\/p>\n<\/div> As generative AI becomes more sophisticated, we may see fully automated mixing and mastering systems that can process an entire album with minimal human intervention. However, many in the industry argue that the human touch will always be necessary to capture the emotional nuances of a performance.<\/p>\n<\/div> Record labels and music publishers have long relied on human intuition and market research to spot trends and discover new talent. A&R (Artists and Repertoire) executives would scour clubs and demo tapes, looking for the next big thing.<\/p>\n<\/div> The integration of AI has shifted this process towards data-driven decision making. Platforms analyze streaming data, social media engagement, and other metrics to identify rising artists and predict hit potential.<\/p>\n<\/div> With generative AI, music companies are exploring even more radical possibilities. Some are experimenting with AI systems that can generate music tailored to specific market segments or demographics. Others are using predictive models to forecast which song structures or lyrical themes are likely to resonate with audiences in the coming months.<\/p>\n<\/div> As the industry grapples with these changes, the key question remains: How will the balance between human creativity and AI assistance evolve? While some fear the loss of human touch in music creation, others see AI as a tool that will ultimately enhance human creativity, opening new avenues for expression and collaboration.<\/p>\n<\/div> As generative AI continues to reshape the music industry, stakeholders are grappling with a complex landscape of legal, economic, and ethical considerations. This transformative technology promises to democratize music creation while simultaneously challenging traditional notions of artistry and copyright.<\/p>\n<\/div> 1. Copyright in the Age of AI<\/p>\n<\/div> Before AI’s emergence, copyright laws in music were relatively straightforward. Composers and performers held rights to their original works, with clear guidelines on sampling and fair use. However, the introduction of AI-generated music has blurred these lines significantly.<\/p>\n<\/div> In the U.S., copyright protection extends to original works of authorship, while European doctrine emphasizes the creator’s personal touch. This distinction becomes crucial when considering AI-generated music. The U.S. Copyright Office has stated that it will not register works produced by a machine or mere mechanical process that operates randomly or automatically without creative input or intervention from a human author.<\/p>\n<\/div> In 2023, at least one high-profile case highlighted copyright concerns around AI-generated music mimicking popular artists. A track called “Heart on My Sleeve” that used AI to imitate the voices of Drake and The Weeknd was uploaded to streaming services and then promptly removed. Universal Music filed a claim on YouTube, and the track’s video was taken down. This incident raised issues around personality rights and the unauthorized use of artists’ voices<\/a>, even if created through AI This action set a precedent for how the industry might handle AI-created content that closely resembles existing artists’ work.<\/p>\n<\/div> For artists, AI presents a double-edged sword. Some embrace the technology as a collaborative tool, while others fear it may devalue their craft.<\/p>\n<\/div> Music companies face their own set of challenges.<\/p>\n<\/div> 2. Market Projections<\/p>\n<\/div> AI in the music market is poised for significant growth. Industry analysts project the sector<\/a> to reach $38.7 billion by 2033, up from $3.9 billion in 2023.This tenfold increase reflects the anticipated widespread adoption of AI tools across various aspects of music production, distribution, and consumption. Interestingly, artists are more looking for AI powered tools<\/a> in music production and music mastering (66%) rather than for AI music generation (47%). Creativity-enabling tools are seen as more important than pure AI-generation tools, even if we have to prepare ourselves to listen to more and more AI-generated music extracts.<\/p>\n<\/div> 1. Democratization of Music Production<\/p>\n<\/div> AI is lowering the barriers to entry for music creation. Tools that once required expensive studios and years of training are now accessible through user-friendly apps. This democratization could lead to an explosion of new musical styles and voices.<\/p>\n<\/div> A music tech startup founder, Alex Mitchell, CEO of Boomy, envisions a future where anyone can be a composer. “Now, we’re seeing people all over the world create instant songs with Boomy, release them, and even earn royalty share income. For the first time, musical expression is available to an entirely new type of creator and audience<\/a>” . This accessibility could fundamentally alter the music landscape, potentially unearthing talent that might otherwise have gone undiscovered.<\/p>\n<\/div> 2. New Artist-Audience Interactions<\/p>\n<\/div> AI is indeed reshaping how artists connect with their audience. Some artists and music platforms are already experimenting with AI-powered chatbots that allow fans to interact with their music catalog in novel ways. For example, Spotify introduced an AI-driven chatbot within its application that interacts with users to understand their music preferences, moods, and listening habits. This allows the chatbot to curate customized playlists and suggest new tracks tailored to each user’s unique taste.<\/p>\n<\/div> These AI chatbots are being used to enhance fan engagement by providing personalized recommendations, real-time updates, and interactive experiences. They can offer artist updates, recommend playlists, facilitate ticket purchases, and manage fan inquiries, fostering deeper connections between artists and their audience.<\/p>\n<\/div> A forward-thinking artist, Holly Herndon, recently launched an AI version of her voice that fans can collaborate with to create personalized tracks. This blending of human creativity and AI capabilities points to a future where the line between artist and audience becomes increasingly fluid.<\/p>\n<\/div> The integration of AI in music creation raises profound questions about the nature of artistry. While some argue that AI is merely a tool, akin to a new instrument, others fear it may diminish the human element that makes music emotionally resonant.<\/p>\n<\/div> Marius Blanchard, Data Scientist from Columbia University and working in the music industry offers a balanced perspective: “I do not believe AI will ever replace artists; […] the audience-artist connection lives on many different levels beyond the music. Instead, I hope that AI will democratize access to quality sound production and be a creativity boost for more and more artists to express and share their creations.”<\/p>\n<\/div> As AI capabilities grow, certain roles within the music industry may evolve or become obsolete. Sound engineers, for instance, may need to adapt their skills to work alongside AI mixing and mastering tools. However, new job categories are likely to emerge, such as AI music programmers or ethical AI consultants for the creative industries.<\/p>\n<\/div> Perhaps the most significant question facing the industry is whether AI-generated music can forge the same emotional connections with listeners as human-created works. While AI can analyze patterns and create technically proficient compositions, the ineffable quality of human experience that often infuses great music remains a challenge to replicate.<\/p>\n<\/div> Electronic musician and AI researcher Holly Herndon offers an optimistic view<\/a>: “I think the best way forward is for artists to lean into developments with machine learning,” she said, suggesting that they “think of ways to conditionally invite others to experiment with them.\u201d<\/p>\n<\/div> Jeff Ens and Philippe Pasquier, (2020). MMM : Exploring Conditional Multi-Track Andrea Agostinelli et al, (2023). MusicLM: Generating Music From Text. arXiv preprint arXiv:2301.11325 https:\/\/arxiv.org\/abs\/2301.11325<\/a><\/p>\n Huang et al, (2018). Music Transformer: Generating Music with Long-Term Structure. arXiv preprint arXiv:1809.04281 https:\/\/arxiv.org\/abs\/1809.04281<\/a><\/p>\n1. The Technology Behind Generative AI for Music<\/h2><\/div>
From Composition to Code: The Breakthroughs of Generative AI in Music<\/strong><\/h3><\/div>
I. GPT for Music (discrete generation)<\/span><\/h3><\/div>
<\/span><\/div>
\na) Pitch: The specific note being played (e.g., C, D, E, etc.)
\nb) Duration: How long the note is held (e.g., quarter note, half note)
\nc) Velocity: How loudly or softly the note is played
\nd) Instrument: Which instrument is playing the note<\/p>\n
\n( we can add tokens for velocities, bar structures, instruments, genres,…)<\/p>\n
\nThese series of notes-tokens are then concatenated and arranged in sequences, similar to how words form sentences in language models. Each token is assigned a unique numerical ID, enabling the GPT model to receive a similar input as for text generation.<\/p>\n<\/span><\/div>II. Diffusion models for music (Continuous generation)<\/span><\/h3><\/div>
<\/span><\/div>
\nWe have explored 2 different ways to generate music using AI. All of these technologies led to impressive and entertaining demos. However, the path from laboratory experiments to market-ready products is fraught with challenges. Technical hurdles abound, from ensuring consistent quality across various musical styles to managing the computational demands of real-time generation. Market challenges are equally daunting, as developers grapple with finding the right audience and use cases for their AI composers.
\nManagement and legal considerations add further complexity. Building and leading teams at the intersection of music and AI requires a unique blend of skills. Meanwhile, the ethical and legal landscape remains largely uncharted, raising questions about copyright, authorship, and the very nature of creativity in the age of AI.
\nAs we delve deeper into the current applications and products emerging from this technological revolution, it becomes clear that the impact of AI on the music industry will be far-reaching, touching every stakeholder from listeners to major record labels.<\/p>\n<\/div>2. Current Applications and Products<\/h2><\/div>
I. The most used today: Suno.AI<\/span><\/h3><\/div>
\nA filmmaker working on a short film might use Suno.AI to generate a custom soundtrack when unable to hire a composer. Advertising agencies are exploring the platform to quickly create jingles and background music for commercials, potentially altering their workflow for music in advertising.<\/p>\n<\/div>II. Other AI Music Platforms<\/span><\/h3><\/div>
3. Impact on Different Stakeholders<\/h2><\/div>
I. Listeners<\/span><\/h3><\/div>
II. Artists<\/span><\/h3><\/div>
III. Engineers<\/span><\/h3><\/div>
IV. Music & Platform Companies<\/span><\/h3><\/div>
4. Future Visions and Opportunities<\/h2><\/div>
I. Legal & Market Considerations<\/span><\/h3><\/div>
II. Potential Transformations<\/span><\/h3><\/div>
A Multifaceted Conclusion<\/h2><\/div>
I. AI’s Role in Creativity<\/span><\/h3><\/div>
II. Impact on Industry Jobs<\/span><\/h3><\/div>
III. Authenticity and Emotional Connection<\/span><\/h3><\/div>
References<\/h2><\/div>
\nMusic Generation with the Transformer. arXiv preprint arXiv:2008.06048 https:\/\/arxiv.org\/abs\/2008.06048<\/a><\/p>\n