WhisperX speech-to-text,
diarized and word-aligned
We built the speech-to-text engine for an AI EdTech platform: WhisperX transcription with word-level timestamps and speaker diarization at scale, pgvector retrieval that turns a lecture into structured, searchable notes, and a grounded generation layer that turns those notes into practice tests and study guides.
A transcript is the easy part. Timing, speakers, and retrieval are the build
Anyone can call a speech-to-text API and get text back. What made the product useful was everything the raw transcript does not give you: exactly when each word was said, who said it, and a way to find the right moment later.
Word-level timestamps, not block captions
WhisperX aligns the transcript to the audio at the word level, so a note links back to the exact second it came from and playback can follow the text.
Speaker diarization at scale
The pipeline separates who spoke, so a lecture with a professor and a room of questions reads as a labeled conversation instead of one undifferentiated stream.
Retrieval over the transcript
Transcripts are long. A pgvector layer turns them into structured, searchable notes, so a student finds the passage that answers a question instead of scrubbing an hour of audio.
What we built
WhisperX transcription
A production speech-to-text pipeline with word-level alignment, tuned for long-form lecture audio.
Speaker diarization
Speaker separation so every line is attributed, not one continuous block of anonymous text.
pgvector retrieval
Vector search over the transcripts, so lectures become structured notes a student can actually query.
Structured, searchable notes
The output is not a raw transcript but organized notes tied back to the moment each point was said.
From notes to graded practice, generated and checked
The transcript and the notes were the input, not the product. The next iteration turned them into study material a student could revise from, generated by AI and checked by a person.
Grounded generation, not guesswork
The AI wrote practice tests, quizzes, flashcards, summaries, and study guides constrained to the actual class material, so it recalls what was taught instead of inventing it, with a person reviewing before anything reached a student.
Multimodal source, one pipeline
It did not stop at the transcript. Photos of the board, scanned notes, slides, and PDFs fed the same generation layer, so the study set reflects the whole session, not only what was spoken.
Video, streamed and stored
Adding video meant building the streaming and storage architecture to hold and serve recordings at scale, so the source stays available behind the notes and the generated study set.
