A plain-language tour of a neural audio codec that encodes the amplitude and the phase of sound in parallel, squeezes them into a few numbers per moment, and rebuilds the audio on the other side.
A codec is a pair of programs: an encoder that turns sound into as few bits as possible, and a decoder that turns those bits back into sound that should be indistinguishable from the original.
One second of raw 48 kHz, 16-bit audio: 48,000 samples × 16 bits.
What APCodec needs for the same second, with almost no audible loss.
Compression. A 3-minute voice note goes from ~17 MB to ~135 KB.
Send a short recipe (pitch, loudness, timbre) and let the receiver re-synthesise it. Very few bits, but the result sounds robotic and breaks down with noise.
Reproduce the wave shape itself. High quality, but it needs many bits, or (for neural ones) hundreds of downsampling steps that make the model large and slow.
Treat the amplitude spectrum and the phase spectrum as the "parameters" of the sound, like a parametric codec does, but train the network with losses that demand a faithful waveform, like a waveform codec does. Because spectra already run at a low frame rate, only 8× downsampling is needed instead of 320×.
Cut audio into tiny overlapping slices (the short-time Fourier transform, STFT). For every slice and every frequency you get two numbers.
The energy at each frequency. It shapes what we hear as vowels, timbre and loudness. Stored as a log value so quiet details are not lost.
The timing offset of each frequency's wave. Get it wrong and audio sounds "phasey", metallic or smeared. Together, A and P rebuild the exact waveform through the inverse STFT.
Amplitude and phase travel through identical, parallel networks. They merge into one compact code in the middle and split again to be rebuilt.
An image-recognition backbone turned 1-D: depth-wise conv → LayerNorm → expand to 512 → GELU → GRN → back to 256, plus a residual shortcut. Eight per lane.
Waveform codecs shrink 48,000 samples/s by 320×. APCodec starts from 1,200 spectrum frames/s, so 8× is enough to reach 150 codes/s. Fewer layers means a faster, smaller model.
The phase lane predicts a "real" part R and an "imaginary" part I, then takes atan2(I, R). The answer is always a valid angle, so the network never fights phase wrapping.
Every 6.67 ms the encoder outputs a 32-number vector. Sending 32 floats would be costly, so instead we send which entry of a 1024-word codebook it is closest to: just 10 bits.
Pick the closest of 1024 learned vectors. Rough, but already in the right area.
Look at what's left over (the residual) and pick the best correction from a second codebook.
Repeat. Each stage fixes the previous one's error. The receiver adds the Q chosen vectors back up.
With Q = 4 codebooks the paper gets 6 kbps, and with Q = 8 it gets 12 kbps. More codebooks means more corrections, so better quality for more bits.
The encoder, quantizer and decoder are trained together as one network. The loss is a sum of several terms, each catching a different kind of mistake.
Amplitude MSE, phase anti-wrapping losses (instantaneous phase, group delay, angular frequency), complex spectrum + STFT consistency, and mel spectrogram L1+L2.
Pulls the encoder's output and the codebook entries towards each other, so little is lost when snapping to the nearest codeword.
Two critics, multi-period (MPD) and multi-resolution (MRD), try to tell real audio from decoded audio. The codec learns to fool them, which removes buzzy artefacts.
The decoded audio should also look like the real audio inside the critics' layers, not just at their final verdict.
That anti-wrapping trick is the key to learning phase at all. Without it, a prediction of +179° against a target of −179° would look like a huge error when it is really only 2° off.
For phone calls the codec can't wait for the future. The streaming version swaps layers that look ahead for ones that only look back, then learns from the full model.
Look-ahead convolutions become feed-forward layers, upsampling becomes causal, and the downsampling kernel is kept ≤ 2D−1 so no future frame is needed.
The non-streaming APCodec acts as the teacher. The streaming student is trained to match the teacher's internal features layer by layer (MSE).
Fixed latency at 48 kHz: one code frame (320 samples). That's far below the ~150 ms where calls start to feel laggy.
VCTK test set, 48 kHz audio at 6 kbps (Tables I and II of the paper).
| Codec | Streamable | LSD ↓ | STOI ↑ | ViSQOL ↑ | Phase error AWPDIP ↓ | Size ↓ | MUSHRA ↑ |
|---|---|---|---|---|---|---|---|
| APCodec | No | 0.818 | 0.875 | 4.07 | 1.68 | 65.4M | 88.48 |
| APCodec-S | Yes | 0.835 | 0.865 | 3.93 | 1.74 | 46.8M | 88.07 |
| Encodec | Yes | 1.04 | 0.793 | 3.31 | 1.80 | 83.2M | 85.91 |
| AudioDec | Yes | 0.847 | 0.804 | 3.98 | 1.81 | 108M | 88.34 |
| DAC | No | 0.841 | 0.906 | 3.81 | 1.78 | 282M | 88.28 |
Every other codec sits near 1.80 rad of phase error. Modelling phase explicitly brings APCodec down to 1.68.
65 M parameters versus 282 M, and about 14× faster on CPU, where DAC can't even keep up with real time.
Ablations: swapping ConvNeXt v2 for a plain residual net drops ViSQOL by 0.5; removing the quantization loss drops it by 0.33.
We use the authors' open-source model code (encoder, residual VQ, decoder, discriminators and losses) with our own training and serving pipeline.
train-clean-100: 100 h, 251 speakers. Test clips come from unseen test-clean speakers.
The paper's 16 kHz setup: hop 40, window 320, FFT 1024, latent at 50 Hz.
8 codebooks × 1024 with quantizer dropout, so one model can serve every bitrate. The paper uses 2 and 4 kbps at 16 kHz.
Trained on a 16-core Xeon with spectral, phase, mel and quantization losses. The paper used 1 M GPU steps with GAN losses.
The model is still training. The demo below always uses the newest checkpoint, so expect quality to improve over time. An early checkpoint is intelligible but less clean than the paper's fully trained model.
Upload a clip (up to 20 s) or record from your mic. The server runs the real APCodec encoder, shows the discrete tokens it would transmit, then decodes them back to audio.
Raw size (16-bit PCM)
Compressed bitstream
Compression ratio
Encode + decode time on CPU
Each column is 20 ms of audio, each row is one codebook, and the colour is the chosen index (0–1023). This grid of integers is everything the receiver gets.