Context
The same organisation as the earlier case studies in this programme. Its archive holds
tens of thousands of recorded live sessions, in the dialects of five regions: Egypt and
Sudan, the Gulf, Iraq and Yemen, the Levant, and the Maghreb. About 11,900 sessions had
transcripts made one at a time with a general-purpose AI tool, without timings. Tens of
thousands had none, so the only way to find a moment in a session was to watch it.
The problem
There was no answer key. Every existing transcript had itself been produced by a
machine; nobody had ever corrected one by hand. The usual way to choose a speech model,
scoring it against human transcripts, was not available.
Dialects are not equally hard. Maghrebi Arabic is the furthest from the standard
language and the weakest spot for most speech models, and it was a large part of the
archive.
The scale had to fit one machine. 62,205 recordings and about 21,300 hours of audio,
some single sessions longer than seven hours, on one office computer that also restarts,
loses power and has its archive disk unplugged.
Many of the voices are children’s. Sending the audio to a cloud transcription service
would have meant another processor holding children’s voices. The work had to stay in the
building.
What we built
Local by default. Transcription runs on one consumer graphics card in the office, with
open-weight models loaded from local storage. No audio, and no text, goes to an outside
service to be transcribed.
The model chosen by measurement, region by region. Two strong candidates were first run
on 32 sessions, 4.6 hours spread across all five regions. Cohere’s Arabic model was closer
to the existing text in every region and about six times faster than Whisper large-v3.
Because the existing transcripts were machine-made too, closeness to them proves little.
So 146 sessions, 60 of them Maghrebi, were transcribed by three independent systems and
compared word by word: where two agree against the third, the third is probably wrong.
That majority vote gave each system an estimated error rate without any human reference.
Tuned on the hardest dialect. On the 59 Maghrebi sessions the chosen model at first
trailed the earlier tool. Longer speech segments and a wider search cut the words it lost
in disagreements by 21%, from 1,689 to 1,329, against 1,565 for the earlier tool. Across
all five regions the same change cut them from 3,995 to 3,495. Grouping segments into batches then doubled the
throughput in testing with no meaningful loss of accuracy.
Dialect routing tested, and deliberately not switched on. A dialect-identification
model was tested as a router, to send each recording to a specialised model. It agreed with
the country named in the session in only 78 of 140 cases, partly because the interviewer
and the trainee often speak different dialects. Routing on a guess that is right just over
half the time sends many recordings to the wrong model, so one model tuned on the hardest
dialect runs for every region. Specialised Maghrebi models are shortlisted; one replaces it
only if it beats it on the same fixed 59-session test set.
Every segment checked. Empty text over audible speech, looping repetition and
implausible character rates are caught automatically. A flagged segment is redone once with
a stronger setting and the better reading is kept; anything still flagged is counted per
recording for a person to check. In the 202.7-hour pilot, 291 recordings ran with no
failures, sessions up to 7.6 hours long ran at the same speed, 77 segments were fixed by
the retry and 2 were left flagged.
Old and new compared, which found misfiled transcripts. Each new transcript is
compared with the one already stored. About 9% of the stored transcripts in the pilot
(25 of 281) turned out to belong to a different video. The split was clean, with no
borderline cases, and 21 of the 25 were traced to the video they really describe: a whole
batch had been shifted by a growing offset. Nothing was overwritten to fix it. Those videos
are transcribed again and the old text stays where it is.
Built to be interrupted. The job starts with the machine, without anyone logging in.
A watchdog restarts it after a crash or after 30 minutes without progress. It resumes from
the last saved segment, even in the middle of a seven-hour recording, and waits for the
archive disk instead of failing when it is unplugged.
A person has the last word. Each transcript is published to the catalogue with timed
subtitles, stored as a separate version beside any older transcript. The team marks the
version it trusts, and a correction becomes a new version rather than an edit.
The result
| Measure |
Result |
| Recordings transcribed |
62,205 |
| Audio |
about 21,300 hours, five dialect regions |
| Speed in production |
about 32 times real time, around 770 hours of audio a day |
| Hardware |
one consumer graphics card in the office |
| Estimated word error, 146-session comparison |
12.5%, against 14.0% for the earlier tool and 27.5% for the second candidate |
| Maghrebi words lost in disagreements |
1,689 before tuning, 1,329 after |
| Segments still flagged after the automatic retry |
fewer than 1 in 10,000 |
| Stored transcripts found to belong to another video |
about 9% of the pilot sample |
| Cloud transcription cost |
none |
What we would tell another organisation
- If no human-checked transcript exists, three independent systems voting tell you more
than any single accuracy score.
- Tune on your hardest case. The easy cases follow; the reverse is rarely true.
- Do not route on a classifier you have not measured. A router that is right half the time
is worse than one strong model.
- Compare new output with old output. It finds data problems nobody was looking for.
- Assume a long job will be interrupted, and make the interruption cost one segment, not
one day.