Counting with OpenAI o1

[entrepreneerit]

Is THIS the reason they named the model Strawberry? Because it can solve the strawberry r problem!

[hyungwonchung]

I had fun recording this demo with quite a trivial question: “how many r’s in strawberry?”

Why did I ask this to showcase one of the most intelligent AI models? LLMs process text at a subword level. A question that requires understanding the notion of both character and word confuses them. GPT-4o can't get this.

Our new model o1-preview "thinks harder" to avoid mistakes and gets it correct

[DragonFly]

Marvelous progress, we are getting closer to the AI singularity.

[romaingerard]

As a math major, I am now preparing psychologically to become unemployed

[crisrampante]

Holy moly this is it we are finally getting a new model

[ugurergun]

you put us on hold again for the new sound mode.

[Sree-xf]

it fails if you misspell it with an extra 'r'.

Q: how many r's in strawberrry
O1 : There are three letter "r"s in "strawberrry".

[issiewizzie]

Still waiting for the Voice feature after a year of announcing it

[arashterooni]

I see the same reasoning mistake happening in o1, how can I reach out to your team for feedback?

[justkoru]

Amazing model as it seems... We'll have to wait for more benchmarks, though.

[MrAbkejoe]

I have a strong feeling this is simple embedded XML tags, nested tags, identifiers, attributes, and a structured tag-based processing instruction, as I've a custom GPT that gets that answer right every time because of the "thinking" XML tags integrations.


I know that sounds extremely simple for something like a new model from openai but I can't help but think about all the researchers that had left.... Just prior to this o1 announcement🤔

[Robocafe]

I queried GPT-4 and GPT-4. o about ten times, asking each how many 'r's are in the word "strawberry." Nine times, both versions correctly responded that there are three 'r's. However, one time, GPT-4. o incorrectly stated that there are only two 'r's.

[outeast]

No real progress in this specific task. You can ask o1 how many "t"s in strawberry and it will tell you that there is 0 "t"s. It gets "r"s correctly only because it's learning dataset is contaminated with this meme question and answer, but there is no true reasoning happening in this class of tasks.

[lorenzoblz]

I do not get it. So are they using a different tokenization method to do this? Classic tokenizations does not preserve this kind of textual information very well if at all. Are they adding some "metadata" to the vectors to preserve details about the textual representation (that may even be split into subwords?)? 🤔

[Kolstee]

It’s crazy how the errors of generative AI are being quickly resolved.

[DamianReloaded]

It would be awesome if chatgpt could be trained to play tabletop games, the RPG - dungeon crawling ones with cards and involved evolving gamestate. I figure this would also be a good way to study how to make the language model navigate a specific domain requiring human interaction. I imagine a hidden prompt tailored to each game would be required, either by hardcoding it or maybe the LM can be trained to generate and update the hidden prompt by itself? my2c

[Logan-hhme]

I like how they praise every model when it comes out and then they trash it to shit when a new one is made.

[CameronLestagez]

so they have a longer loading times for it to do better

[stonechen]

Why is the prompt how many r’s IN are in strawberry? Why is there an extra IN? What is this English?

[AerisRG]

GPT4o can answer this question too, just add "think step by step" in prompt