🎙️🔧 Audio2Tool Leaderboard
Voice-command tool calling across 8 difficulty tiers, scored on a held-out private set.
🥇 | GT transcript + Gemma3-27B | 🎙️ end-to-end | 0.793 | 0.397 | 0.421 | 0.875 | 0.805 | 0.813 | 0.372 | 0.915 | 0.818 | 0.651 | 0.849 | 6/8 | 2026-09-04 |
All numbers are percentages. Per-tier columns show the metric selected above; overall columns are macro-averages over covered tiers. “–” means the model has no valid result for that tier.
Per-tier scores for the top 10 models (by overall score).
What is this?
Audio2Tool benchmarks audio-language models on voice-command tool calling: the model hears a spoken command and must emit the right tool call with the right parameters. The eval set is a held-out subset of the private split of RVtech/Audio2Tool (up to 200 queries per tier, 1,579 samples). Labels are hidden — scoring happens inside this Space.
The 8 tiers
| Tier | Challenge |
|---|---|
| T1 Direct | Straightforward commands ("Lock the car") |
| T2 Param | Commands with parameter values ("Set driver temp to 21") |
| T3 Multi | Several intents in one utterance |
| T4 Implicit | Intent must be inferred ("It's freezing back here") |
| T5 Needle | Command buried in long distracting audio |
| T6 Correct | Speaker self-corrects mid-utterance |
| T7 Turns | Multi-turn conversations (turn audio provided in order) |
| T8 Blend | Overlapping speakers / blended intents |
Metrics
| Metric | Definition |
|---|---|
| Tool Acc | Predicted primary tool name matches ground truth (case-insensitive) |
| Exact Match | Tool name and all parameters match exactly |
| Param F1 | F1 over parameter key/value pairs (0 if the tool is wrong) |
Overall scores are macro-averaged across covered tiers. Entries marked oracle ASR consume ground-truth transcripts instead of audio — text-side upper bounds, not audio models.
How to submit
- Download the eval set:
hf download RVtech/Audio2Tool-private-eval --repo-type dataset --local-dir eval - For each row of
metadata.jsonl, run your model on the audio and produce a tool-call string. Tools are defined in the dataset'stools_registry.csv. - Write one JSON row per sample:Multi-intent samples: several calls in one string, primary first —
{"sample_id": "tier1_direct/00042", "prediction": "setZoneTemperature(zone=Driver, temperature=21.0)"}setLockState(state=Locked) setFanSpeed(level=7) - Upload the file on the Submit tab. Scores appear immediately.
The scorer uses the exact normalization of the benchmark code, so leaderboard numbers are comparable to local runs.