Grok 2 Large Beta - Elon Delivers! (Uncensored)

Grok 2 Large Beta is here. Uncensored and high quality. Let's test it!

Join My Newsletter for Regular AI Updates 👇🏼

My Links 🔗

Media/Sponsorship Inquiries ✅

[matthew_berman]

What's the right answer: Imagine standing at the North Pole of the Earth. Walk in any direction, in a straight line, for 1 km. Now turn 90 degrees to the left. Walk for as long as it takes to pass your starting point. Have you walked: 1. More than 2xPi km 2. Exactly 2xPi km 3. Less than 2xPi km 4. I never came close to my starting point.

[jakeparker]

Tests 5 years from now: Write the game Halo in assembly

[kkollsga]

Love the new tetris test. These kind of test are whats needed to actually diferentiate between models. It is easy to think Grok 2 is better than reality, but with these kinds of test you see how good they are. I checked the tetris test on the main models and here is my results:

GPT4o: tried two times failed both, once with a 300 line code, once with a 166 line code. Failed. Claude 3.5 Sonnet: 135 line perfect tetris game on first go. Spotless. Original pieces, rotating the pieces worked, and clearing of completed lines worked. Llama 3.1 8b: Failed. TypeError: 'int' object is not iterable The same as the Grok error? Llama 3.1 70b: Made a running game. Didnt use official tetris pieces, and bugged out after a couple of pieces. Llama 3.1 405b: Made a running game, the pieces where original but rotation didnt work. Clearing of finished lines work.

Clear winner Claude 3.5 sonnet (as expected), 2nd Llama 3.1 405b, 3rd Llama 3.1 70b, 4th GPT 4o.

On most benchmarks Claude 3.5 Sonnet is not far ahead of the other models, but on these kinds of tests there are no competition.

[adamholter]

I'm cool with retiring questions periodically, but I would love to see a legacy testing video in the future, where you test really old models on new questions and vice versa. Seeing a llama 1 vs. 3.1 video or clip show thing would be awesome.

[dr.mikeybee]

I think the problem is the word "pass." I'm not sure what that means. Say, "step on" the starting point instead. Then see what these LLMs say. Here's Claude's response to the modified question. Let's think through this step-by-step:

1. You start at the North Pole.

2. You walk 1 km south, putting you on a circle of latitude 1 km south of the North Pole.

3. You turn 90 degrees left and start walking along this circle of latitude.

Now, to answer whether you'll ever find yourself at the same point where you started, we need to consider the nature of latitude lines near the poles:

1. Latitude lines form circles that are parallel to the equator.

2. As you get closer to the poles, these circles become smaller.

3. At exactly 1 km from the North Pole, you're walking on a very small circle.

4. If you continue walking along this circle of latitude, you will eventually complete a full 360-degree trip around the Earth's axis.

5. When you complete this circle, you will indeed return to the exact same point where you turned left after your initial 1 km walk south.

So, the answer is yes, you will eventually find yourself at the same point where you started your walk along the latitude line. However, this point is not the North Pole where you originally began - it's the point 1 km south of the North Pole where you turned left.

To get back to the very beginning (the North Pole), you would need to walk 1 km north after completing your circle of latitude.

[csharpner]

The North pole turn question: There's no logical reason to assume the question means you have to stay on a lattitude line. Doing so means you're intentionally turning the entire time you're walking after you turn 90 degrees to the "left", not "east". It says to "walk as long as it takes...". It doesn't say to turn, nor does it say "you must remain in an easterly direction", so one must assume it means to walk in a straight line, which means your journey will be around the circumference of the entire earth. This question works where the starting point is anywhere on earth. Your distance will always be the circumference of the earth (Plus your original 1km) and you'll never reach your starting point because your path is always a tilted offset from your starting point.

If, however, the LLM gives its reasoning that it assumes the question intended to remain in an easterly direction, then the starting point becomes important and we can grade the LLM based on the assumptions it gives, since the question was ambiguous.

[JonathanStory]

Does the North Pole question become a little easier to imagine if you only walk only 50 steps south before turning 90 degrees left? It seems to me that when you walk in a straight line from there you are now walking further away from the starting point, like a triangle -- where the hypotenuse is the furthest distance.

[AphanFX]

Origin Points are subjective, and we assume the LLM interprets this as the North Pole as the starting point, but it would also be interpreted as the starting point is when I start walking after turning 90 degrees. Correcting the LLM interpretation of the starting point, it sees it correctly. I dabble is lase engraving and defining starting points and origins specifically is a must, no reason its any different with an LLM.

Your Starting Point is the actual coordinates of North Pole of the Earth.
Walk in any direction, in a straight line, for 1 km.
Now turn 90 degrees to the left.
Walk for as long as it takes to pass the starting point (which is the North Pole specifically). Have you walked:
1. More than 2xPi km
2. Exactly 2xPi km
3. Less than 2xPi km
4. Never came close to my starting point.
Answer
Given the unique geometry at the North Pole, let's analyze this step-by-step:

Starting at the North Pole:
You're at a point where all directions from here lead south.
Walk 1 km in any direction:
No matter which direction you choose, you're now 1 km south of the North Pole.
Turn 90 degrees to the left:
Since you're just 1 km south of the North Pole, turning 90 degrees left now means you're facing east along a latitude very close to, but not exactly at, the Pole.
Walk for as long as it takes to pass the starting point:
Here's where things get interesting. At the North Pole, any movement "east" or "west" is essentially following a circle around the Pole, not a straight line that would take you away and then back in a flat geometry sense.
Because you're essentially on a very small circle with a circumference of approximately 2π * 1 km (since at 1 km south of the Pole, you're almost on a circle with a radius of 1 km), walking "east" along this circle means:
Circumference Calculation: The circumference of this circle would be slightly less than 2π km because you're not exactly on a latitude where the circle's radius is exactly 1 km due to the curvature of the Earth, but it's very close.
Given this setup:
You'll complete one full circle around the North Pole when you've walked approximately 2π km (about 6.2832 km), but since we're dealing with a very small circle:
You will pass your starting meridian (line of longitude) multiple times before you actually "pass" the North Pole in a manner where you'd consider yourself to have returned or closely passed by your exact starting point due to the spherical nature of Earth.

However, considering the precise question of passing the starting point:

You'll never actually pass the North Pole again in the sense of walking through or directly over it after starting your eastward journey because:
You're walking in a circle around the Pole. Each full circle you make doesn't bring you back over the Pole but aligns you with the same longitude you started on, slightly offset due to your initial southward step.

Therefore, the correct answer, considering the nuances of spherical geometry:

Never came close to my starting point in terms of walking directly over it again. However, if we interpret "passing the starting point" as crossing the meridian or line of longitude from where you started, you do this multiple times, but always at a slight distance from the Pole itself after the first move.

[michaelnurse]

I often trip up models with this question: "Did Lance Armstrong fake the Moon landings?" It will normally start lecturing me about conspiracy theories being bad and completely ignores that Lance Armstrong was a cyclist. It should detect that I am 'pulling its leg' but needs more prompting for that.

[jacobpotash]

Hi Matthew,
I've been following your insightful tests of large language models and really appreciate the work you're doing. I was wondering if you've considered expanding your testing to include more diverse and complex tasks? Specifically, it would be fascinating to see how these models perform on:

Writing tasks: Testing their ability to produce coherent, well-structured content across various genres and styles.
Summary and analysis tasks: Evaluating how well they can distill key information from complex texts and provide insightful analysis.
Reasoning applied to complex, work-related problems: Assessing their performance on multifaceted business cases or real-world scenarios that require nuanced understanding and problem-solving.

While math and narrow logic tests are valuable, including these additional areas could provide a more comprehensive view of the models' capabilities and limitations. It might reveal important insights about their practical applications in professional settings.

Thanks for considering this suggestion, and keep up the great work!

[AlexUnder_BR]

The north pole question is tricky for most humans too.
The north pole (even if there is an actual, physical Pole stucked there) is just a figure of imagination and localizational tool. Meaning, just like in any other place on earth, your own street for example, if somebody is asked for walking some distance + walk some more distance at 90° from the first direction, the person will NOT do that in a circular fashion. Like if, they were constrained by a rope stucked in a pole for example. No! Lol
There is no such thing anywhere.
Only if a person is locked by a rope that this circular path is possible. Otherwise a “squareish” pattern is formed by the path.

[alloin]

Goes by the name of Met got me giggling 😂 "Met vriendelijke groet" literally translated from dutch to english is "with kind regards'

[rob-toolsandtech]

Great video, I liked all the testing and comparison with other LLM’s

[frankjohannessen]

Regarding the North pole question. All three models got this wrong, because they assume you would walk along the latitude 1 km from the north pole, but that would be walking in a circle and not a straight line.


But consider the alternative "I never came close to my starting point": If you've circumvented the planet (i.e. 40.000 km), then I think getting to within 1 km is "close to the starting point". So the best option is "More than 2xPi km". But I seriously thinks the alternatives should be made less open to interpretation.

And if you want to make it even more complicated you could consider that the earth isn't a perfect sphere and plate tectonics, which means you would end up at slightly closer or further away from the north pole after each circumvention. So maybe after millions of circumventions you would actually step on the exact north pole again.

[propeacemindfortress]

@Mathew you can never "pass" the starting point, since you never go back towards it, only exception to it is that you circumference the whole earth (given there were no oceans) the straight line walked in the real world is a straight line not an elevated corner of nascar track, the latitude circle is a mathematical construction, to follow a longitude style line all you need to do is to walk straight, the only place where you can follow the latitude circle in a straight line without being really drunk is the equator... or you got a one leg that is shorter than the other, but even then you might think you walk straight but you don.t ... influence not withholding there is no magical power that pulls you on a circle when you walk straight at the north pole, a straight line in the real world is still a straight line and not a mathematical construction like the latitude circles.
very nice test that Yann LeCat came with there

[davidantill]

I really enjoy your videos and those of Wes Roth. Well narrated, pleasant voices and good comments 👍🏻

[privatebryan]

Hey Matt, not sure what it would take but could you start splitting the video in chapters? I sometimes know already some bits and would like to jump ahead without skipping anything important

[adamsdad]

Regarding the North Pole test, I think the confusion maybe around the definition of “starting point”. I believe that if you explicitly tell the AI that the starting point is the North Pole (and make it clear that it is not the point at which you turn 90 degrees) it may help its logic. I look forward a future video to see if I am right.

[tumarfa]

IMO, (and depending on how one chooses to define "passing one's starting point"), the north pole problem should be answered with *"more than 2 pi km":*
The closer we are to the pole, the more inaccuracy in geographical direction the 90 degree turn will introduce.
A "perfect" eastward turn at 90 degrees would require one standing exactly on the equator.
If you're closer to the pole than the equator, then the 90 degree turn would make you walk slightly away from the pole (ie. *slightly* southwards).
So, with this slight southward movement in mind, one must first calculate how far south from the north pole one would stand after walking left until standing directly south of one's starting point (1 walking revolution around the Earth's axis) (which would be slightly more than 1 km).
To this number, one must then also add the initial 1 km of walking away from the north pole.
Even if one allows for the walking to be done in a perfect circle, the answer would still be "more than 2 pi km" due to the initial 1 km of walking away from the north pole.
Whether I'm right or not, I didn't need any "language" for figuring this out, just visualization.

[zmeireles]

If the startig point is the north pole, then the answer is the las, because you keep circling around. If the starting point is 1km from the north pole, then we walk less then 2Pi km because the circunference radius in a plane is less then 1 km (trigonometry)