Getting Started with Groq API | Making Near Real Time Chatting with LLMs Possible

Let's have a look at the Groq API that offers access to their Language Processing Units (LPUs) for free (for a limited time). The API can provide close to 500 tokens per second with Mixtral-8x7B. Enjoy :)

LINKS:

TIMESTAMPS:
[00:00] Introduction
[00:34] How to Access the Groq API?
[01:06] API Playground
[03:15] Getting Started with Groq API
[05:33] Near real-time speed
[07:24] Advanced API Features: Streaming and Stop Sequences
[11:02] Building a Chatbot with Streamlit and Grok API

All Interesting Videos:

[martg]

Thanks for the video! I will start testing this API with a POC I am working now to learn.

[thierry-le-frippon]

They should sell their LPUs instead and compete with Nvidia. They would surely get lots of backup and investments. They will probably be copied instead othetwise and fade away quickly.

[jonoburcham]

Great video! Can ou make a voice chatbot using groq in one of your next videos please? I would also love to see if you do this on streamlit or if it's too slow and you use something else. Thanks so much for your videos

[yztfuvy]

Please Create a step-by-step video guide on using the Groq API with Streamlit.

[osamaa.h.altameemi]

this is next level. OpenAI got some serious competition.

[KOTAGIRISIVAKUMAR]

why you cant use the conversational retrieval chain instead of the conversation chain Because it can handle the memory by default no need maintain externally?
@prompt Engineering

[DestanBegu]

Thanks for your content! I´m using Streamlit as well and want to give Content as the System role. For Example "answer me in short sentences in italian" so it will do this for each prompt i do. Where can i do this in the code? I used the Streamlit Chatbot Repo.
Thanks in advance

[ConnectorIQ]

almost a baby version of a quantum computer if you can actually perfect a model based on speed of responses to your questions and using the groq gpu...

[hmsfaceface]

how can the groq fpga use mixtral 8x7b with just 250gigs of vram?

[jmay]

If temp can adjust to minus what is impact on generation ( consider it as hypothetical if case don't exist )

[prestonmccauley]

I tried a few things with this and it is incredibly fast.

[ramimithalouni]

what is the time to receive the first chunk in streaming?

[vishnuprabhaviswanathan]

How to control the output of LLM for a single input?

[mickelodiansurname]

Heres the question, can Groq cards also work on inference for art and audio and voice models? or is it just LLM inference specific? It is like, well superfast... the only worry is literally the latency from you to the endpoint... so if its say, a streaming interruptible feed you are giving the model then the use cases for TTS and Speech applications just went through the damn roof!

[Francotujk]

What are the rate limits of the free api? Is it necessary to provide credit card?

[jesusleguizamon]

Hi, this api have function calling? regards

[ZombieJig]

Fuck all these cloud only AI services, release the cards!

[TheJscriptor]

YALLM ... it is almost becoming daily news ... Yet Another LLM.

[savire.ergheiz]

Fast but useless. These oss models still way far behind cgpt4.