How to Build a Speech Dataset for ASR: Scripts, Recording Specs and QC That Hold Up
IndicVoices asked every speaker for 20 minutes and checked every clip by ear. A method for ASR dataset collection: scripts, consent, specs, QC and WER.
The short version
- Decide the speech type first. IndicVoices, the largest public Indian-language effort, is 74% extempore and only 9% read speech, because read speech alone does not match how people talk to machines.
- Split by speaker, not by clip. LibriSpeech, Common Voice and IndicVoices all keep each speaker in only one of train, dev or test, so the score measures new voices.
- Check every clip by ear early. IndicVoices coordinators checked audibility, verbatim match, relevance and length on the spot, then a central team labelled 23 error types.
- Measure with word error rate: substitutions plus deletions plus insertions, divided by reference words, as defined in NIST's sclite scoring tool.
Most ASR datasets fail before a single clip is recorded, at the point where someone decides what the model will hear. The largest public Indian-language effort, IndicVoices, is only 9% read speech. Its authors chose that mix deliberately, and if you want to build a speech dataset for ASR, their reasons are the place to start.
This is a working method, not a theory post. It covers script design, speaker diversity, consent, recording specs, quality checks and word error rate. Every rule comes from a published corpus that trained real models: LibriSpeech, Common Voice and IndicVoices.
To build a speech dataset for ASR, decide the mix of read and spontaneous speech, write short scripts, and recruit speakers who match your users. Collect consent and demographics first. Record one clip per prompt at 16 kHz or higher, check every clip by ear, split by speaker, and measure word error rate on held-out speakers.
Decide what the model will hear before you write a script
A speech recogniser learns the conditions it was trained on. If every clip is a quiet adult reading a news sentence, the model will struggle with a farmer asking for a bank balance on a noisy road. So the first decision is the speech type, not the tool.
Read, extempore and conversational speech do different jobs
Read speech is a speaker reading your sentence. It is cheap, because the transcript exists before the audio. Extempore speech is a speaker answering a prompt in their own words. Conversational speech is two people talking. Both need transcription after the fact.
- Extempore74%
- Conversational17%
- Read9%
We think most first datasets should still start with read speech. It is the fastest way to find out whether your speakers, devices and instructions work. Then add spontaneous speech once the pipeline is proven. A script-reading pilot that fails costs a week. A transcription project that fails costs a quarter.
Match the channel to real use
IndicVoices recorded most audio on participants' own phones, in farms, houses and open spaces, rather than in a studio. It also collected part of the data over an 8 kHz telephone channel, to represent users who reach services through phone lines. If your users call an IVR, record some phone audio. If they use an app, record on cheap phones, not studio microphones.
Script design for read speech
A script is a list of sentences, each recorded as one clip. Good scripts are short, varied and legally clean. Bad scripts repeat the same 200 easy words.
Keep sentences short and licences clean
IndicVoices notes that read sentences typically last 6 to 8 seconds. Short sentences mean fewer stumbles and fewer re-records. They also keep each clip well under the segment lengths models train on.
Licensing matters if you will ever share the data. Common Voice releases its corpus under CC0, so its sentences must be public domain too. If you take sentences from a website, check the licence first. Writing your own sentences avoids the problem entirely.
Cover the words your users will say
Wikipedia sentences give broad vocabulary. They do not give phone numbers, brand names or bank commands. IndicVoices drew read sentences from 13 domains and added everyday tasks such as hailing a ride or making a digital payment, to capture numbers, brand names and alphanumeric codes.
Write a coverage list before you write sentences: numbers, dates, money, place names, product names, and the commands your app accepts. Then make sure each appears in several sentences, said by several speakers. If your users mix languages mid-sentence, write some mixed sentences too. We look at that pattern in code-switching in voice answers.
Speaker diversity: who you record matters more than how much
Ten hours from 5 speakers trains a model that knows 5 voices. Ten hours from 200 speakers trains a model that knows voices. Plan the speaker list before the hour count.
More speakers or more minutes per speaker
IndicVoices faced this trade-off directly. The team targeted 100 to 150 speakers per district and asked each for at least 20 minutes. They explain why. Shorter sessions would have been mostly icebreaker answers, and each speaker still had to be paid a half-day wage, so 4 to 5 times more speakers would have multiplied the budget.
Your numbers will differ, but the logic carries over. Set quotas for gender, age band, region and device before recording, and track them weekly. IndicVoices also found that some registration data was wrong, sometimes deliberately to meet quotas. Verify demographics, do not just collect them.
Split train and test by speaker, never by clip
If the same voice appears in training and test data, your test score measures memory, not recognition. All three reference corpora avoid it. Common Voice's CorporaCreator buckets clips so any given speaker appears in only one split. LibriSpeech built its clean dev and test sets from 20 male and 20 female speakers each, kept out of training.
Consent and compensation come before the first clip
A recording is personal data: a voice can identify the person behind it. Get consent in writing, in the speaker's language, before you record anything.
IndicVoices sets a useful standard. The protocol went through the institute's ethics committee. Coordinators explained the task, its purpose, the time it would take and the pay, then asked participants to sign a consent form. Participants were told they could skip any task. Pay matched prevailing local daily wages.
Your consent text should answer five questions. What is the data for? Who will use it? Will it be published or sold? How long will you keep it? What will the speaker be paid? If you cannot answer one of them, you are not ready to record.
Recording specs: sample rate, format and the browser
Record at a sample rate of 16 kHz or higher, in mono. You can always downsample. You can never recover frequencies you did not capture. The reference corpora agree on the floor and differ on everything else.
| Corpus or tool | Sample rate | Format | Recorded on |
|---|---|---|---|
| LibriSpeech | 16 kHz | Audiobook audio, segmented to 35 s or less | Volunteer audiobook readers |
| Common Voice | 48 kHz, mono | MP3 | Web browser or app |
| IndicVoices (app) | 44.1 kHz | WAV | Participants' own phones, Karya app |
| IndicVoices (calls) | 8 kHz | Telephony | Cloud telephony bridge |
| Browser voice form (Would You Rather) | Browser default | WebM/Opus, or MP4 where WebM is unsupported | Phone or laptop browser |
The last row concedes a real weakness. Common Voice argued in its paper that MP3 is the most reliable format on the web and that its quality is appropriate for speech. We agree for most fine-tuning work. But lossless WAV from an app, as IndicVoices used, is the better choice if you need raw signal for acoustic research.
What a browser does to the signal
Browsers can apply their own processing before the recording reaches you. Would You Rather requests echo cancellation, noise suppression and automatic gain control from the browser. That makes clips cleaner to listen to. It also means your training audio is slightly less like a raw microphone feed. If your model must handle raw noisy audio, test on a sample before committing to any browser recorder, ours included.
A recording workflow with a voice form
If you already have speakers, a web form with a voice field is the lightest way to collect scripted clips. Here is the workflow we support in Would You Rather for script-reading projects. It has been used for read-speech corpora in Indian languages.
- 1Register the speaker firstName, age band, gender, region and a consent question as ordinary form fields before any recording.
- 2One voice question per script sentenceEach sentence is its own question, so each recording arrives as its own clip with a known transcript.
- 3Set a minimum and maximum lengthA minimum rejects accidental taps; a maximum stops runaway recordings. Both are per-question settings.
- 4Turn on transcript modeThe script sentence is prefilled under the recorder. The speaker edits it only if they said something different.
- 5Let speakers listen back and re-recordThe speaker hears the clip before moving on and can replace it. This is the cheapest QC you will ever get.
- 6Export clips with transcriptsAudio links and the edited transcript land in a Google Sheet or CSV, one row per speaker.
In transcript mode, typing the text does not count as an answer. The question still needs a recording. Typing in English letters offers suggestions in the target Indian script, which helps speakers whose phone keyboard lacks one. Our guide to forms in Hindi, Tamil or any language covers the script side.
What the form does not do is just as important. It does not transcribe audio, run a second-listener vote, trim silence or compute WER. Those steps sit in your pipeline, fed by the Google Sheets export or the public API.
Quality checks that catch most bad clips
Check every clip in the pilot, and a fixed sample at scale. IndicVoices used two layers. A coordinator checked each response on the spot, then a central QC team listened to every file and labelled it Excellent, Acceptable or NotAcceptable.
| Check | What it catches | How to run it |
|---|---|---|
| Audibility | Low volume, constant background noise | Listen; flag clips a human could not transcribe |
| Script mismatch | Skipped, swapped or added words | Compare audio to script by ear, or run an ASR model and flag high WER |
| Clipping | Distortion from shouting or a hot microphone | Count samples pinned at full scale |
| Silence and length | Empty clips, long dead air, cut-off endings | Voice activity detection plus min and max duration |
| Wrong language | Speaker answered in another language | Listen, or run language identification |
| Identity and metadata | Same voice under several IDs, wrong age or gender | Spot-check voices across IDs; verify demographics |
The coordinator layer is worth copying. IndicVoices coordinators checked that each response was audible, matched read text verbatim, was relevant to the prompt and was neither too short nor stretched. Repeats happened while the speaker was still there, which is far cheaper than a callback.
Trim silence and segment with voice activity detection
Long recordings must be cut into short segments. IndicVoices first asked annotators to segment by hand and got wildly varying lengths. They switched to Silero voice activity detection, lowering the minimum silence threshold step by step from 1,000 ms to 40 ms and keeping segments under 20 seconds. Merging very short pieces gave an average of 9 seconds.
Catch script mismatch with a model, then confirm by ear
For read speech, the cheapest mismatch check is to run any existing recogniser on each clip and compare its output to the script. Clips with a very high error rate go to a human. The model will be wrong about accents and rare words, so never auto-reject on its score alone.
Word error rate basics
Word error rate, or WER, is the standard ASR metric. NIST's sclite scoring tool aligns the system's output to the reference transcript and counts three error types: substitutions, deletions and insertions. WER is their sum divided by the number of reference words.
A worked example
Take the reference "send five hundred rupees to amma", which has 6 words. The system outputs "send five hundred rupee to mama now". That is 2 substitutions (rupee, mama), 0 deletions and 1 insertion (now). WER is 3 divided by 6, or 50%.
Two details trip people up. First, insertions add errors without adding reference words, so WER can exceed 100%. Second, text normalisation changes the score. Decide early how you write numbers, punctuation and spelling variants, and apply the same rules to references and outputs. IndicVoices wrote a two-level transcription guideline for exactly this gap between written and spoken Indian languages.
Where this method is weakest
This post leans on three corpora built by large teams. Their choices fit their goals: open research benchmarks and broad language coverage. A startup fine-tuning one product's commands may need far less diversity and far more domain vocabulary.
We do not give a target hour count, because no source here supports one for fine-tuning. The useful answer depends on the base model and the language. Any blog post that gives you a single number is guessing.
Our own tool covers collection, not the pipeline. It has no built-in validation vote, transcription or WER. And browser recording trades raw signal for reach. That trade suits scripted reading on phones. It does not suit acoustic research that needs untouched audio.
How to run a pilot this week
Write 30 sentences that include your hardest vocabulary: numbers, names and commands. Recruit 10 speakers who match your users, with a mix of gender and age. Write a one-paragraph consent text and pay them fairly for their time.
Record every sentence as its own clip, then listen to all 300 clips yourself with the QC table above beside you. Count rejects by reason. Fix the script and instructions, then run it again with 10 new speakers. When rejects drop and stay low, scale. If you are still choosing how to collect, compare the options in our speech data collection tools comparison.
References
- Javed, T. et al., IndicVoices: Towards building an Inclusive Multilingual Speech Dataset for Indian Languages, arXiv preprint, March 2024. Used for speech type mix, script domains, speaker targets, consent, recording channels, QC categories and segmentation.
- Ardila, R. et al., Common Voice: A Massively-Multilingual Speech Corpus, LREC 2020. Used for the MP3 48 kHz format, the validation vote and speaker-disjoint splits.
- Mozilla Common Voice, cv-dataset repository, Scripted Speech v27.0 statistics, 11 September 2026. Used for total and validated hours.
- Mozilla Common Voice, CorporaCreator. Used for how speaker-disjoint splits are produced.
- Panayotov, V., Chen, G., Povey, D. and Khudanpur, S., LibriSpeech: an ASR corpus based on public domain audio books, ICASSP 2015. Used for the 16 kHz rate, 35-second segments and speaker-balanced test sets.
- NIST, SCTK scoring toolkit and sclite documentation. Used for the word error rate components.
The weakest point in this source base: IndicVoices is a preprint describing one project's choices, and none of these sources tests how much browser processing or MP3 compression changes ASR accuracy.
Questions people ask
How do you create a speech dataset?
Write prompts or script sentences, recruit speakers who match your future users, and collect written consent. Record each sentence as its own clip with the script as its transcript. Have a second person check every clip for audibility and verbatim match. Then segment long audio, normalise the text, and split the data so no speaker appears in both training and test sets. Measure progress with word error rate on the held-out speakers.
What sample rate should I use for an ASR dataset?
Record at 16 kHz or higher and keep a mono channel. LibriSpeech, the standard English benchmark, is sampled at 16 kHz. Common Voice releases 48 kHz mono MP3, and IndicVoices recorded WAV at 44.1 kHz on phones. Higher rates can always be downsampled later, but you cannot recover detail you never recorded. If your users call in on phone lines, also collect some 8 kHz telephony audio.
How many hours of speech data do I need for ASR?
It depends on whether you fine-tune a pretrained model or train from scratch. LibriSpeech offers about 1,000 hours of English. IndicVoices aimed for about 20,000 transcribed hours across 22 Indian languages. A fine-tuning pilot can start much smaller. Collect a pilot, hold out speakers for testing, and measure word error rate. Add data where errors cluster rather than adding hours evenly.
How is word error rate calculated?
Align the recogniser's output to the reference transcript word by word. Count substitutions, deletions and insertions. Add them and divide by the number of words in the reference. NIST's sclite tool reports each of these as a percentage of reference words. Because insertions add errors without adding reference words, word error rate can exceed 100% when a system hallucinates many extra words.
What should a speech recording consent form include?
State the purpose of the collection, who will use the recordings, whether they will be published or sold, how long they will be kept, and what the speaker will be paid. Say that the speaker can skip any prompt. IndicVoices also collected age, gender, language proficiency, occupation and location at registration, explained the task and pay in the speaker's own language, and obtained explicit consent before recording.
Should I use read speech or spontaneous speech for ASR?
Use both, weighted toward how your users will actually talk. Read speech is cheap because the transcript exists before the audio. Spontaneous speech is closer to real use but must be transcribed. IndicVoices chose 9% read, 74% extempore and 17% conversational audio. A voice assistant for commands needs more short, scripted phrases; a call-centre model needs more conversation.
Drafted with AI from published research, then edited and approved by Shivam Singh.
Would you rather hear the answer?
Make a voice form in a minute. It’s free.
Create a form