NPUs: the most overhyped new chip?

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

►►► This video ◄◄◄

Microsoft with Copilot+ PCs, Google on the Pixel, Apple with iPhone and Macs, Qualcomm with Snapdragon X, Intel and AMD, all the tech companies want you to care about the NPU, Neural Processing unit or Neural engine in their new machines to run AI workloads, neural networks, etc. But are these chips any good?

The Story Behind

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

►►► TechAltar links ◄◄◄

Social media:

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

►►► Attributions & Sources ◄◄◄

[ivonakis]

People: we want better battery life in laptops.
Microsoft: we will use NPU to do tons of work in the background.
People: disable it to get better battery life.

[graxxor]

Engineer: AI Needs a ton of ram.
Tim Cook: Great, let’s ship our MacBook *PRO* with 8Gb.

[jorge]

I already saw laptops with NPUs but 8gb of soldered ram 😂

[skygodofallurya]

I've transitioned from 'wow, that's kind of cool. Wonder what they'll come up with next?" to "Fuck off with all the AI, please, " in about six months.

[JamesRoyceDawson]

Looking forward to the AI bubble bursting. There might still be some AI that's worthwhile but it's way overhyped and underdeveloped right now

[aximatic]

I'm sure Apple's 8GB laptops will be totally 'equivalent' to 16GB for AI use...

[ScottAshmead]

guess that is why Apple was saying 8G is enough in the recent past so they would have a solid up-sale later

[TheNJK]

We need a video explaining the clean shave, brutal betrayal 😂

[HarisAzriel]

This is the best explanation of NPU so far! Other people that have talked about it are either only talking from political/emotional standpoint or from the money aspect. No one has ever discussed it in a practical application angle like you did. Great work, man!

[jagersama]

Curious note: in the early days of NPU in phones, they were less powerful and power efficient than GPUs. Huawei, Samsung, and Qualcomm relied a lot on their GPU to do AI work. Google with Visual Core started to deviate and then with Tensor, few months after Tensor Snapdragon 8 Gen 1 was the first Qualcomm chip with a really powerful NPU.

[ZILtoid]

One way I could be more interested is if the NPU had a little more data types, and would be good for other DSP-style calculations, like impulse responses for audio.

[vilelive]

I thought the Intel 4004 had wood paneling on it and I thought "That's so retro." Then I realized it's copper.

[kwcu]

Just when they faced limits with CPU and GPU suddenly out of nowhere appears new NPU thing that you definitely need in every device.

[aarongeerer]

They're designed to monitor everything you say, read, and type. There's no way this ends up going badly.. It will likely only be niche manufacturers who will offer laptops without a neural engine for privacy reasons. After Microsoft's debacle with Copilot, there's no reason to think they won't be pumping this in with the usage data sent back to them.

[timothywells]

I would be interested if the NPU can run larger open LLMs utilising RAM which is cheap compared to trying to increase VRAM. Unless you're willing to give Nvida the price of a car and your kidney and maybe your first born child then 24gb and maybe 28 if they bump the 5090 a little. Ultimately I'd love to see expandable VRAM like we had back in the 90's.

[teamredstudio]

I'm impressed that you explained a simple neural network correctly. Not a lot of people actually understand it. There's of course a lot more to it like back propagation and quantisation.

[rbrisita]

To give some clarity and correct some misunderstandings:

NPUs are way cheaper than GPUs.

NPUs use a lot less power and are more efficient than GPUs.

The total of parameters are cross all layers.

To do a neural network (NN) calculation you’ll only need the previous output and the current layer. This is why NPUs don’t need excessive RAM.

Specifics:
Face recognition on an image only needs a model that is less than 500MB in size and a NPU of about 1TOP.

Small Language Models (SLMs) need about 2-6TOPs to have a reasonable response time.

Real-time object detection needs about 10TOPs.

This is all running under 2W.

A 13TOP NPU is about $70 USD.

[boltez]

Dude i must say this is the best video that i have come across that explains the need and differences of various components (such as CPU and GPU) in simple terms.

[KH-lgxc]

- Computer, what big NPU you have!
- I need it to better spy on you.

[Big-Chungus]

Frankly I don’t want it if it means I can avoid AI features. Unfortunately it looks like the options are “run the AI locally” or “give us your data and run it in the cloud”. I don’t want either.

Absolutely hate how AI is being forced into everything now.