The NEW Chip Inside Your Phone! (NPUs)

Thanks to Dr. Ian Cutress for his help with this video! Check out his blog and YouTube channel:

Neural processing units (NPUs) such as Apple's Neural Engine or the machine learning engine on Google Tensor chips can be found on the iPhone and the Pixel. How do they help run AI right on your phone?

Leave a reply with your requests for future episodes.

FOLLOW US ELSEWHERE
---------------------------------------------------

[ErazerPT]

The crux is not processing power. Its the memory to hold the model. You can wait for things to get done, but if you can't even hold them in memory to begin with, its a no starter. So, the great models are restricted to "wherever you can fit them in", leaving "small but still useful models" to everything else. NPU's, like any other ASIC, will simply do it faster and more efficiently. And they won't need that much space, because, as we've established, they'll only run very small models anyway. One thing i can see thrown at them is "voice quality".

[Ceo_of_RACING]

Why does this feel like I'm watching techquickie in 2016

[roomierent]

I'm starting to feel the definition of "AI" or "AI-enabled features" is expanding in scope to encompass what was just traditional software before. Facial recognition software, for example, has existed long before ChatGPT.

[paZ]

3:06 I tried that with my 15 pro and it takes about 6-7min for a 1000x1000 image. Which is painfully slow compared to midjourny etc but is still amazing to see. To have this feature with you at all times without relying on services is amazing

[FredericHeckmann]

There is also the tradeoff between modem/cellular power consumption and NPU power consumption. There are many scenarios where sending the data to the cloud would actually consume more power than doing it locally.

[RageQuitSon]

Sorry we can't fit an audiojack in your phone, but here's the AI chip. and no we won't include a charging brick and lie that it is to save the planet instead of save 10 cents per phone

[a.i.privilege]

Can I trust any companies with my info/data? The answer is no.

[brokenchad]

the green screen spilled on Linus's beard

[seltonu]

0:46 "They are embarrassingly parallel"

"In parallel computing, an embarrassingly parallel workload or problem (also called embarrassingly parallelizable, perfectly parallel, delightfully parallel or pleasingly parallel) is one where little or no effort is needed to separate the problem into a number of parallel tasks.[1] This is often the case where there is little or no dependency or need for communication between those parallel tasks, or for results between them."

Smooth reference, nice to see the Techquickie writers do their homework!😊

[mrknighttheitguy]

I'm sorry Dave, I can't do that...

[kenzieduckmoo]

we've had AI chips on desktop for years, with Nvidia's Tensor cores, but building neural engines into Intel and AMD cps might actually make it useful

[stalbaum]

Also, surprised a bit but you did not mention that Apis like Tensor Flow lite are optimized for - yep - 256 bit operations. Which works ok in the image space, for example accelerating face recognition (which it does with downscaled grayscales...)

[jmoney]

Apple has been making chips with neural engines since 2017 with the A11 in the iPhone 8, iPhone 8 Plus, and iPhone X. Clearly they made the right call.

[Bruno-cbgk]

It's like how they added RT cores on the 20 series, but too few to actually run any meaningful raytracing at high FPS. But it started the software integration of ray tracing features, which makes it worth dedicating more die area to RT cores in later generations.

[paxdriver]

If I've learned anything in my 38 years it's that AI chips will get saturated by software extracting value out of the hardware of the people who paid for it. Then they'll tell us our devices are slow because they're old, not because they can't do what we need them to but because our devices can't meet the demands of companies violating our privacy and resources.

[rg]

Wait, haven’t NPU’s been in phones for years at this point?

[chrono]

It makes sense to run it locally for two reasons privacy and as the number of smartphones the demand on cloud resources becomes higher if you could offshoot most of those processes to your local device it would decrease latency and allow the cloud to deal with more processes not your phone can't run rather than just doing huge numbers of small tasks and slowing everybody down

[IncredibleMeep]

So in other words turn everyone's phone into one giant super cluster computer to collect massive amounts of data to feed into ai models.

[biexbr]

0:47 he did! he did! he said! he said the thing.

Some AI models are being deployed on the edge of network. I think we'll see a lot of the mixed AI functions using NPUs and edge computing, reducing costs on cloud services and keeping response time in an acceptable range for large models.