WhisperX Speech-to-Text Case Study | 2muchcoffee

Case study

WhisperX speech-to-text,
diarized and word-aligned

Hand-drawn scribble illustration of speech-to-text for EdTech: a microphone and audio waveform flowing into a timestamped transcript with two speaker bubbles, a magnifying glass over the notes, and a generated quiz card with a checkmark under a graduation cap, plus a coffee mug.

We built the speech-to-text engine for an AI EdTech platform: WhisperX transcription with word-level timestamps and speaker diarization at scale, pgvector retrieval that turns a lecture into structured, searchable notes, and a grounded generation layer that turns those notes into practice tests and study guides.

At a glance

Category
AI EdTech, lecture capture, notes, and assessments
Our role
Speech-to-text, retrieval, content generation, and CTO leadership
The hard part
Word-level timing and speaker labels at scale, not just a transcript
Architecture
WhisperX and pgvector retrieval, plus a grounded generation layer over multimodal inputs

The engineering

A transcript is the easy part. Timing, speakers, and retrieval are the build

Anyone can call a speech-to-text API and get text back. What made the product useful was everything the raw transcript does not give you: exactly when each word was said, who said it, and a way to find the right moment later.

Word-level timestamps, not block captions

WhisperX aligns the transcript to the audio at the word level, so a note links back to the exact second it came from and playback can follow the text.

Speaker diarization at scale

The pipeline separates who spoke, so a lecture with a professor and a room of questions reads as a labeled conversation instead of one undifferentiated stream.

Retrieval over the transcript

Transcripts are long. A pgvector layer turns them into structured, searchable notes, so a student finds the passage that answers a question instead of scrubbing an hour of audio.

Scope

What we built

WhisperX transcription
A production speech-to-text pipeline with word-level alignment, tuned for long-form lecture audio.
Speaker diarization
Speaker separation so every line is attributed, not one continuous block of anonymous text.
pgvector retrieval
Vector search over the transcripts, so lectures become structured notes a student can actually query.
Structured, searchable notes
The output is not a raw transcript but organized notes tied back to the moment each point was said.

What came next

From notes to graded practice, generated and checked

The transcript and the notes were the input, not the product. The next iteration turned them into study material a student could revise from, generated by AI and checked by a person.

Grounded generation, not guesswork

The AI wrote practice tests, quizzes, flashcards, summaries, and study guides constrained to the actual class material, so it recalls what was taught instead of inventing it, with a person reviewing before anything reached a student.

Multimodal source, one pipeline

It did not stop at the transcript. Photos of the board, scanned notes, slides, and PDFs fed the same generation layer, so the study set reflects the whole session, not only what was spoken.

Video, streamed and stored

Adding video meant building the streaming and storage architecture to hold and serve recordings at scale, so the source stays available behind the notes and the generated study set.

See what speech-to-text looks like for your audio

Built with

The stack

WhisperXSpeech-to-textSpeaker diarizationWord-level timestampspgvectorPostgreSQLPythonRAG retrievalLLM content generationVideo streamingObject storage

What it delivers

Word-level
jump to the exact second a point was made
Diarized
every speaker attributed, automatically
Searchable
ask a question, get the passage, not an hour of audio

FAQ

Speech-to-text and WhisperX, answered

What is WhisperX and how is it different from Whisper?

WhisperX builds on OpenAI Whisper and adds word-level timestamps and speaker diarization. Plain Whisper returns a transcript in rough chunks; WhisperX tells you exactly when each word was said and who said it, which is what makes a transcript useful inside a product.

Can speech-to-text label who is speaking?

Yes. Speaker diarization separates a recording into speakers and attributes each line, so a lecture or a meeting reads as a labeled conversation rather than one continuous block of text.

How do you make a long transcript searchable?

We store the transcript in a vector database (pgvector on PostgreSQL) and retrieve by meaning, so a user finds the passage that answers their question instead of reading or scrubbing the whole recording.

Do you build speech-to-text into an existing product?

Yes. We build the transcription, diarization, and retrieval layer and integrate it into the app you already have, or lead the engineering end to end. This EdTech platform ran with one of our engineers in the CTO seat.

Can you generate quizzes and study guides from lecture recordings?

Yes, that was the next iteration here. An AI layer turns the recording and its supporting material (photos of the board, scanned notes, slides, and PDFs) into practice tests, quizzes, flashcards, summaries, and study guides. It stays grounded in the source so it does not invent facts, and a person reviews the output before a student sees it.

Can it run self-hosted, without sending audio to a third party?

Yes, and for EdTech that is often the point. WhisperX is open and runs on your own infrastructure, so student and lecture audio never has to leave your environment or train someone else's model. We deploy it self-hosted when data residency or privacy requires it.

How accurate is it on accents and long, noisy lecture audio?

Accuracy comes from the pipeline, not a single number: choosing the right WhisperX model size, aligning at the word level, and processing long audio in overlapping chunks so nothing drops at the seams. We tune it against your real recordings, not a clean benchmark.

What does it cost to run, and does it need a GPU?

WhisperX is fastest on a GPU, and self-hosting trades a per-minute API bill for your own compute. We size the hardware, cloud GPU or on-prem, to your volume and latency, so a large back catalog and a live pipeline are costed differently.

Let's build yours

We built the speech-to-text engine behind an AI EdTech platform, with one of our engineers in the CTO seat and four senior people across the pipeline, retrieval, and UX. Tell us what you're transcribing and we'll set up a 30-minute review of how we'd build it.

Related

Production AI we shipped AI development company EdTech software development Ultimate guide to speech-to-text
<?xml version="1.0" encoding="UTF-8"?>