TinyAudio: Compact and Efficient Text-to-Audio Generation for Low-Resource Deployment

TinyAudio is a compact flow-matching text-to-audio model that generates 44.1 kHz audio with only 87M inference-time parameters — over 90% fewer than representative billion-parameter pipelines — and 0.48 GB of peak GPU memory. Its core TA-DiT jointly models text and audio tokens in a single stream with layer-shared conditional modulation, while TA-CLAP provides audio-aligned text conditioning. TinyAudio-MF, trained with the Improved Mean Flows objective, reduces latent generation from 25 Euler steps to a single step, enabling real-time generation under a four-core CPU quota.

TA-DiT is TinyAudio's compact generative backbone. It removes two sources of repeated parameters in text-conditioned generative Transformers: separate modality-fusion modules and block-specific conditional modulation networks. Projected text tokens and noised audio latents are concatenated for joint self-attention, with independently indexed rotary position embeddings preserving the positional structure of each modality; only audio tokens predict latent velocity. Instead of a per-block AdaLN network, the shift–scale and gating MLPs are shared across all Transformer blocks, with lightweight block-specific embeddings preserving depth-dependent behavior. A single 18-block stream with hidden size 384 and 8 attention heads yields a 35M generator.

TinyAudio attains an AudioCaps FAD of 2.65 with 0.48 GB of peak GPU memory, more than 8× less than every compared system, and remains competitive in CU and PQ on TTA-Bench. TinyAudio-MF cuts sampling to a single function evaluation at a measurable quality trade-off.