Siri's secret weapon: app intents

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

►►► This video ◄◄◄

Siri today is pretty dumb. Apple Intelligence and generative AI is supposed to change that, but even more important is Apple's plan to supercharge it using App Intents. This is a framework for exposing 3rd party apps to Siri. Will it work?

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

►►► TechAltar links ◄◄◄

Patreon:

Social media:

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

►►► Attributions & Sources ◄◄◄

Stock images from Getty & AP Archive

[fayenotfaye]

Rabbit's "Large Action Model" was literally just a handful of browser automation scripts, if the website changed at all, the script would no longer work, it wasn't A.I.

[SvenReinck]

That’s exactly what Tim Cook meant when he said to investors that Apple is in a unique position when it comes to AI

[NicolaiWeitkemper]

Fun fact: Android has had "Intents" (with this exact name) basically forever. An app opening a file picker or the camera is an Intent. One *could* do fairly advanced stuff with this, but as of now, it isn't really user-facing.

[desmond-hawkins]

This is a great take, and the kind of perspective that's too often missing in reports about Apple. The key feature Apple's competitors are lacking is the introduction of a *semantic* aspect to app data and app actions: instead of trying to detect – and mostly guess – whatever is on screen, having these components labeled internally with semantic information enables the most powerful integration. Why "semantic"? Because labeling data as _subjects_ an _objects_ and actions as _verbs_ lets an LLM directly connect them together in an query (a sentence) like in the pizza example (2:03), or "text mom I'm on my way and tell her how long it will take me". _Text_ is an action of Messages, _Mom_ is an object of Contacts, _how long_ maps to an action of Maps.The LLM can extract these from the query, find the appropriate subject/verb/object/etc. entities exposed by apps as intents, connect them all together and perform the action. All of this is pretty much only possible with "the ecosystem" that Apple built, the LLM being the -pizza- glue that joins them through _meaning_ as a high-level concept.

[Silas_]

Intents on Android have actually always existed from the very beginning!
They are not user-friendly (originally not aimed at users at all) and you have to know the exact name of the intent, but basically they have always existed. Apps like Tasker are based almost entirely on this functionality, but are also aimed more at power users. With app shortcuts and app links, intents have become somewhat more user-friendly, but the possibilities for using them are still far from exhausted.
Once again, I am amazed that everyone is now acting as if Apple has just reinvented the API, even though I think it would be very helpful if apps could work together better on iOS.

[teamredstudio]

Before the Shortcuts app we actually had Automator and AppleScript on the Mac to do similar things, it's bascially been in the making for 20 years and I hope we can keep improving this. The biggest problem is turning LLM prompts into actions performed by the OS because LLMs don't give predictable data they spit out human language but for a computer that is still hard to understand. They have to train a model to turn language into a set of strict commands. We will get there eventually, just need to wait a bit.

[natekwezi]

The amount of mac items in bro's backround is well over $10k

[aucontraire]

Rabbit only has a very surface level access to your apps, AI must have perfect integration.

[Imperial_Squid]

I can't help but feel like most big companies are overplaying their hand when it comes to how important AI will be in most people's lives, that said, at least Apple have a semi reasonable plan from what's presented here. Even so, AI Winter can't come soon enough imo

[gehteuchnichtsan]

When theres a hype in tech, TechAltar is the only Channel that doesn't tell you the same basic infos you get in every video. And the Videos are incredible in depth looks into the tech world you dont get in most tech channels. 🙏

[drac]

Isn't this what google assistant tried to do with apps integration years ago and I don't know why developers didn't care? You could say: Ok Google, talk to <app name>. And from that all commands would be for that specific app. For me that was the solution we needed, but somehow just didn't work, famous apps never got on board.

[Company-]

Create content. I love it that you are to the point and do not have a lot of banter around every master of information that you give us.

[vernearase]

Apple apps have long been built with a processing back end, a UI front end, and wiring to connect the two. That's how Applescript worked - by circumventing the wiring between the two and plugging into the back end to circumvent the UI through automation. Shortcuts and Automator are simply the latest GUI versions of Applescript which allowed scriptors to automate their Mac Applications.

I don't know if any other platforms do this type of thing, and automating apps will be _very_ difficult to do without this type of architecture.

It made the Apple frameworks harder to code, but it sure seems to be paying off now!

[Chris-ycmm]

1. Sounds like a privacy nightmare
2. Sounds like an api

[Äpple-pie-k]

This was good but I wonder how many people who watched it got the message about App Intents or it flew over their head just like at WWDC

[ayanmajumder]

If all features of an app is made accessible via app intents, then user won't open the app, so there would be no ad revenue. Why would developers want that ? Except for those apps which run on subscription model.

[bartzrt]

Yeah, my benchmark is the following query: "My friend from Norway is in town next week. Please make a reservation for us at that Chinese restaurant she likes." The assistant would need complete access to pretty much everything on my phone in order to do this, so even if it could reliably do this -- given said access to my messaging apps, contact list, photos, location history, calendar and email or phone -- I'm not sure what would need to happen for me to trust it that much. Like, if I were to hire a _human_ personal assistant I'd have reservations about just handing them my unlocked phone.

[BriefNerdOriginal]

Essentially a way to replace even more the user in operating him/herself the device, so that the walled garden will be even more inescapable.

[theodejong]

Great video. Very interesting. Please add chapters with descriptive titles

[TheIntelligentVehicle]

A level up (if not more) on the kinds of insight in the typical analyses I see. As usual. Really appreciate it.