OpenAI Whisper Speaker Diarization - Transcription with Speaker Names

High level overview of what's happening with OpenAI Whisper Speaker Diarization:

Using Open AI's Whisper model to seperate audio into segments and generate transcripts.
Then generating speaker embeddings for each segments.
Then using agglomerative clustering on the embeddings to identify the speaker for each segment.

Speaker Identification or Speaker Labelling is very important for Podcast Transcription or Conversations Audio Transcription. This code helps you do that.

[DwarkeshPatel]

Thanks so much for making this video and highlighting my code! Really cool to see it's useful to other peopl!

[estrangeiroemtodaparte]

As always, delivering the goods! Thanks 1littlecoder!

[stebe]

I was working on a model to do this exact thing, as we speak. Thanks for the resource, this will save me lots of time

[IWLTFT]

Hi Everyone, Thanks 1littlecoder and Dwarkesh, this is fantastic, I managed to get it working and it is helping me immensely and I am learning a lot. I am struggling with Google as I always end up with 0 compute units and that causes all sorts of issues and I am unable to complete the transcriptions (i have quite large files I am processing, several 1 hr coaching sessions). Does AWS have a better option going? And the next question would be how easy would it be porting this to an AWS linux environment? if that is an option

[mjaym]

Amazing videos!! Keep going!. I have a request though. Could you please publish a video for customizing GPT-J-6B on colab using 8bit version.

[rrrila]

It is interesting, although, i think it would be way better to autodetect how many speakers are there, and then start the transcription.

[SustainaBIT]

Thank you so much, by any chance, do you think there could be a method to make it do all of that in real time, during a call let's say.

Any ideas of where could I start would be very helpful ❤❤

[MixwellSidechains]

I'm running it locally in Jupyter notebook but I can't seem to find an offline model PreTrainedSpeakerEmbedding

[kmanjunath]

I almost did it manually
1.Created rttm file using pyannote.
2.Slice full length audio using rttm file in and out for each.
3.Run thru wisper for transcription.

It was like 5 times slower.
I was thinking hard how to do other way around. First generate full transcript and then separate segments.
Some how i saw your video and am impressed and the AgglomerativeClustering at he end blowed my mind.

Thanks for sharing knowledge.

[IgorGeraskin]

Thank you for sharing your knowledge!
Everything works fine, but an error started appearing:
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behavior is the source of the following dependency conflicts.
llmx 0.0.15a0 requires cohere, which is not installed.
llmx 0.0.15a0 requires openai, which is not installed.
How to fix?

[klarinooo]

I m trying to upload a wav file of 5MB and I m getting "RangeError: Maximum call stack size exceeded" .. is this only for tiny file sizes?

[udcqlql]

i started to love it when you used bruce wayne's clip

[zwxhboz]

hello sir...i have small doubt...if we have no of speaker more than "2" ..how could the parameter "num_speakers" be vary..

[geoffphillips]

None of the ones I've played with cope particularly well with more complicated situations. For instance where one person interrupts another, or if there are three people or more. They can all cope with two very clearly different speakers, but then I figure I could do that with old school techniques like simply averaging the frequency. It's weird because the text to speech itself is enormously clever, it's just surprising that the AI can't distinguish voices well.

[gaurav]

The colab notebook is not accessible. Can you share the new link.

[JohnHumphrey]

I'm not much of a dev myself, but it seems like it might be simple to add a total time spoken for each speaker. I would love to be able to analyze podcasts in order to understand how much time the host is speaking relative to the guest. In fact it would be very cool if someone built an app that would remove one of the speakers from a conversation and create a separate audio file consisting of only what the remaining speaker/s said.

[frosti]

It doesn't work well (detects language as Malay, also, does not offer custom names for speakers) - anyone has a better working solution?

[kmanjunath]

What if new speaker enters in between then number of speakers will become? old + new or old?

[jordandunn]

on the last cell I get an error for UnicodeEncodeError: 'ascii' codec can't encode characters in position 5-6: ordinal not in range(128) any ideas whats going wrong and how do I fix it?

[DestroyaDestroya]

Anyone tried this recently?
Code no longer works. Looks to me some dependencies have been upgraded.