AudioMap: Cloze-and-Choice Reinforcement Learning for Time-Aware Dense Audio Captioning

Learning fine-grained acoustic evidence and event-time grounding through verifiable reinforcement-learning rewards.

Dataset, code, and model releases are coming soon.

Model outputs

Demos

One model produces timestamped, fine-grained captions from either audio alone or paired audio-visual input. The outputs are shown verbatim.

Audio-only input

Workshop process

Without access to frames, AudioMap temporally organizes the tool sounds, background music, shift to instructional speech, and music-only ending.

The source video is displayed for playback context. Its visual stream was withheld from the model.

Click to view the full generated caption

At 0:00, the audio begins with a continuous, upbeat instrumental track featuring a steady drum kit beat, a bassline, and rhythmic electric guitar strumming, establishing a moderate-fast, 4/4 tempo that loops throughout the entire recording. Layered over this music is a loud, harsh, high-pitched mechanical buzzing, characteristic of an electric motorized tool like a rotary tool or Dremel being engaged. This motorized sound is accompanied by a harsh, metallic scraping and grinding noise, suggesting a hard object being moved or shaped against a solid surface. At 0:04, the motorized buzzing abruptly stops, replaced by a sharp metallic clack or snap, likely the tool being switched off or set down. At 0:06, the motorized buzzing and harsh scraping noise resumes at a higher pitch and intensity. From 0:08 to 0:12, the motorized tool abruptly stops, giving way to a loud, harsh, high-pitched squealing and scraping sound. This abrasive, friction-like noise features a fluctuating pitch and intensity, strongly suggesting a heavy, rough material like wood or metal being dragged or rubbed vigorously across a hard surface, such as a workshop floor or metal tool. From 00:12 to 00:19, the sound transitions to a slightly lower-pitched, steady motorized whirring, suggesting a drill or a different sanding attachment running continuously. At 0:19, the grinding stops completely and a male voice enters, speaking in a clear, conversational tone with a relaxed, instructional cadence. He states, "So I'll let it sit in the bin for about fifteen minutes and get a good soak of mineral oil." At 0:24, following a brief pause, he continues, "And then I'll, uh, apply some beeswax, mineral oil mixture to the sur- the surface of it," exhibiting a slight vocal disfluency by repeating the word "the." During his speech, faint handling noises, like objects being moved or set down, are audible in the background. At 0:32, the man finishes speaking, leaving only the continuous, upbeat instrumental music to play out. At 0:41, the music begins a gradual fade-out, reaching complete silence by 0:43.

Audio + video input

A Christmas windmill

With both modalities, AudioMap follows dialogue turns, voice characteristics, music and sound effects, the central conflict, and the crowd response.

Both the soundtrack and visual frames were available to the model.

Click to view the full generated caption

At 0:00, the audio opens with a wide-stereo, synthesized whoosh and a heavy, reverberant thud, immediately establishing a spacious, highly reverberant acoustic environment that strongly suggests a large, cavernous indoor space like a hall or church. Beneath this initial impact, a continuous, low-frequency ominous drone begins, accompanied by a faint, rhythmic ticking or pulsing sound, as a bright, resonant chime or bell-like sound effect rings out and gradually fades. From 0:01 to 0:06, a female speaker with a bright, welcoming tone and a General American accent says, "Come on in. Welcome to your Christmas tour. You, sir, with the camera around your neck." At 0:06, a male speaker with a deep, resonant, and slightly gravelly voice responds, "Brady Schultz," prompting the female speaker to reply with a teasing, slightly amused tone, "Welcome, Brady. Or should I say, welcome back." At 0:10, a second female speaker with a slightly higher, softer, and more conversational pitch asks, "You know that guy?" to which the first female replies, "We dated in high school. He dumped me right before prom," followed by a dramatic musical swell of brass and strings. At 0:15, a sharp, percussive orchestral hit punctuates the dialogue as a mature, authoritative female voice states, "I sold the deed to the Windmill Way property." The male voice then speaks with confident, business-like projection, "Windmill Way will be the biggest tourist spot on the map." At 0:21, the first female speaker counters with a direct, confrontational question, asking, "By demolishing the windmill on Windmill Way? You're gonna be on the naughty list for a very long time." At 0:26, the first female speaker urgently pleads, "Give me a chance to save it," prompting the male to reply, "I want to help. And keep my job, if possible." At 0:31, another loud, reverberant impact or boom sound effect hits, followed by a tense musical swell. From 0:32 to 0:38, the first female speaker returns to a softer, more sincere tone, stating, "The windmill is the heart and soul of this town," while the sound of a crowd cheering and applauding becomes clearly audible in the background, and the first female speaker adds, "It truly is a historical landmark. We have a thousand signatures so far." At 0:38, the first male speaker delivers a dramatic, subdued line, saying, "I truly underestimated the love for this mill." At 0:41, the first female speaker begins to say, "We're the big..." before the audio abruptly cuts off. Throughout the entire clip, the dialogue is mixed prominently in the center of the stereo field, exhibiting a dry, close-mic'd quality with heavy dynamic range compression and a noticeable low-frequency proximity effect, while the background music, sound effects, and cavernous room tone are panned widely across the left and right channels, creating a cinematic, highly produced atmosphere.

Paper at a glance

Abstract

Time-aware dense audio captioning (TDAC) aims to generate multiple fine-grained attributes (dense) of the audio with precise time boundaries (time-aware). Existing methods struggle to achieve these two goals and mainly rely on supervised fine-tuning, yielding sub-optimal performance. While reinforcement learning (RL) shows promise, applying it to TDAC faces two main challenges: (1) existing rewards are too coarse to supervise multi-event, multi-attribute, and multi-relation descriptions in a fine-grained manner; and (2) temporal supervision is difficult for free-form captions, where flexible event-time expressions make reliable event-time correspondence challenging. To address these challenges, we propose AudioMap, a novel RL-based TDAC framework, which shifts to a unified cloze-and-choice reward paradigm. Specifically, we introduce the Evidence Sufficiency Reward (ESR) with an asymmetric hierarchical scoring mechanism to promote fine-grained accuracy and descriptive richness across diverse acoustic dimensions. Furthermore, we design the Event-Conditioned Temporal Reward (ECTR) to structurally bind timestamps to event semantics via temporal IoU, accompanied by a dual-curriculum learning strategy to facilitate the training process. Finally, to support this task, we construct the first time-aware fine-grained audio captioning dataset, AudioMapCap-44K, which contains 44K carefully annotated captions. Extensive experiments across diverse benchmarks show that AudioMap achieves state-of-the-art (SOTA) performance among open-source models and delivers competitive or superior results relative to proprietary models.

Comparison of caption reward paradigms
Reward paradigm overview. Scalar and checklist rewards collapse supervision, whereas AudioMap maps individual evidence and event-time relations to verifiable signals.

Learning framework

Method

AudioMap converts one free-form dense caption into localized checks of acoustic evidence and event-specific temporal alignment.

01SFT0-30 s clips
02SFTlonger clips
03GRPOESR + length
04GRPOfull rewards
AudioMap training framework
SFT proceeds from shorter to longer inputs. GRPO first optimizes ESR with length regularization and then adds ECTR.

Evidence Sufficiency Reward

Multi-choice cloze questions cover speech, music, sound events, acoustic scenes, source attributes, audio quality, and event relations. Correct evidence receives +1, absent evidence -0.5, and incorrect evidence -1.

Event-Conditioned Temporal Reward

The examiner extracts the interval associated with a specific event from the generated caption. Temporal IoU scores the event-interval pair rather than isolated numeric tokens.

Dual Curriculum

SFT moves from clips up to 30 seconds to longer inputs. GRPO first learns evidence coverage and length control, then adds event-time grounding.

Training corpus

AudioMapCap-44K

43,870caption pairs
769.7 haudio duration
80 / 20audio / audiovisual SFT
12,500GRPO samples
1

Initial caption

Generate a time-aware description from the source clip.

2

Targeted follow-up

Probe missing acoustic attributes and event relations.

3

Integrate and filter

Merge evidence, score completeness, and inspect quality.

Empirical evidence

Results

Direct caption evaluation, caption-conditioned question answering, temporal grounding, and human ratings test complementary parts of the contribution.

Audio-only comparison with state-of-the-art methods
Audio-only comparison on Omni-Cloze, MMSU, MMAR, MMAU, and TACOS.
Direct caption evaluation 64.6 Omni-Cloze

+8.3 over the strongest listed open-source captioner.

Caption-conditioned QA 72.4 MMAU

+1.4 over the strongest listed open-source baseline.

Temporal grounding 57.4 TACOS

+10.2 over TimeChat-Captioner.

Audio-visual Omni-Cloze comparison
With audio-visual input, AudioMap-7B reaches 64.9 on Omni-Cloze.

Qualitative evidence

What the Captions Recover

Figure 4 contrasts the original Qwen2.5-Omni-7B captions with AudioMap-7B on speaker disentanglement and multi-type acoustic coverage.

Qualitative comparison between Qwen2.5-Omni-7B and AudioMap-7B
Figure 4. AudioMap separates overlapping speaker turns and covers speech, music, sound events, attributes, and temporal structure in greater detail.
01

Speaker disentanglement

Distinguishes multiple interviewers and grounds accent, delivery, and timing to the corresponding turn.

02

Cross-type coverage

Integrates speech, music, acoustic events, source attributes, and their temporal progression in one caption.

Component evidence

Ablations

Post-training and reward ablation
Effects of SFT, GRPO, curriculum, asymmetric scoring, and reward design.
Ablation of ESR, ECTR, and length reward
ESR improves semantic coverage; ECTR provides the largest temporal gain.