AudioLDM 2
Latent diffusion model generating speech, sound effects and music from text through a shared audio representation.
| Language | Python |
|---|---|
| Category | ML & generative |
| License | CC-BY-NC-SA-4.0 (weights) |
| Platforms | Linux Windows macOS |
| Install | pip install git+https://github.com/haoheliu/AudioLDM2.git |
| First released | 2023 |
| Maintained | Limited |
| Links | Project · Source · Model |
Strengths
- One model covering speech, sound effects and music
- Text-to-speech with a supplied transcription, not just generic audio
- Runs via diffusers as well as its own CLI
- MPS support, so it works on Apple Silicon
- Widely used, so plenty of community examples exist
Limitations
- CC-BY-NC-SA: non-commercial and share-alike — derivatives must carry the same licence
- Jack of all trades; specialised models beat it on each individual task
- Upstream development has been quiet since 2024
- Music quality trails dedicated music models
Best for
Research and experimentation across speech, sound and music with a single model — provided the non-commercial share-alike licence works for you.
Minimal example
# CLI
audioldm2 -t "Musical constellations twinkling in the night sky, forming a cosmic melody."
# Text-to-speech with an explicit transcription
audioldm2 -t "A female reporter is speaking full of emotion" \
--transcription "Wish you have a good day"
# Pick a checkpoint and a device
audioldm2 --model_name "audioldm2-full" --device mps -t "gentle rain on leaves"Share-alike is the catch. CC-BY-NC-SA is more restrictive than the CC-BY-NC used by MusicGen: as well as barring commercial use, it requires derivative works to carry the same licence. That matters if you fine-tune. See the licensing matrix.
Related tools
See all AudioLDM 2 alternatives & how they compare →
See the whole ML & generative category →