Reflection 70b Problems?! What We Know So Far...

Reflection 70b might be too good to be true. Here's everything we know and my own "reflection" on how I can do better next time as your source of AI information.

Join My Newsletter for Regular AI Updates 👇🏼

My Links 🔗

Media/Sponsorship Inquiries ✅

Links:

[matthew_berman]

I will try to approach things with more skepticism in the future. This is certainly a learning moment for me.

I'm open to your feedback, let me know how I could have handled things better.

[thirien]

You corrected yourself in 3 days, i think its fair to say that you didn't misled anyone for a significant time.

[daschewie]

Mathew, please don't change anything with your content. I enjoy your optimism and excitement when covering AI over dry news.

[sovthpawsenter]

Don't beat yourself up too hard. This is exactly the kind of industry to attract snake oil salesmen. Don't get jaded, you're on the right track with your content. Follow-ups like this are important, and so many look to you for the AI news digest.

We all got excited, we all got duped, and you followed-up very quickly. We all went on this journey, keep documenting the whole ride.

[LailaSharshar]

You're good. You weren't trying to sell it. You were curious, trying to show it to people and if it turns out to be bad, you kept us in the loop, knowing as much as you did. No one was harmed in the filming of that video.

[Esteband]

You were right interviewing him and reporting what you saw. That is why we follow you. There will be some bad/dumb actors and we all will fall for them.
Please don't delete the videos they are historic.

[Clbhrdwck]

You did perfect man this is exactly how someone should handle this situation

[andydataguy]

I think you should coverc everything and leave it up to your audience to make the decisions ultimately.

You've been immaculately transparent and up to date about this whole situation.

Mad respect brother please keep it up

[HAmzakhan]

You're good. I liked that you kept asking him how it works, how it is better than just currently what we use i.e custom prompting, and he kept on dodging questions and never gave a straight answer.

[AAjax]

Regardless of how this comes out, you did nothing wrong at all. The new model was news, and you did a great job covering it.
Keep on keeping on!

[rononeil]

It's refreshing to see a creator own up to initial enthusiasm and then dig deeper. Your honesty helps the whole community stay informed.

[brunodangelo]

Anyone can mess up, especially about stuff that they are excited about. Also many people eat fake news without questioning them.

Not many come forwards admitting a mistake. That deserves props.


Keep it up, Matthew.

[TheWhiteWolf]

If this whole scenario proves anything, it’s that we need to be more sceptical when it comes to these benchmarks and claims, especially when it comes from tweets…

[FrederickMbuya]

I am still very new to the whole LLM scene, (a couple of weeks), but I have watched many of your videos and I saw the initial one about reflection. 1) I remember a couple of times you basically said/asked that their magic sauce is just doing what you would do at the prompt stage into the model refining stage, (terminology?!). 2) I think it's awsome that you are immediately owning how you could have done more research etc, imagine if mainstream media took the same approach. 3) don't change still give is the latest even if it is with a disclaimer

[serg]

I think you did great, Matthew. Didn’t hype up the model before anything concrete could be tested, and most importantly self reflected on your mistakes and explained to us what went wrong.

[Tarek.AbdELKhalek]

Amazing Reflection Video, You just "provided your reasoning step by step" :), I love it and Gotta say I learned a lot from your videos, and now I am learning How To Reflect too & "Take deep breath and Think step by step" :)

[vickmackey]

That "Anthropic" response seems pretty definitive to me. How would that happen by accident if it's a Llama model from Meta? He's busted, and that's probably why he's gone completely silent on Twitter.

[toadlguy]

I listened to your original interview and I have to say that Matt seemed on the up and up. I do believe that what he described is a reasonable area for study and there is no doubt that by providing fine tuning to instill the process that is used in your prompt engineering is not only reasonable but is what the major models such as Claude are doing. In fact the Claude model uses an <antthinking> tag themselves. What did not make sense were the benchmark results, but I would not want to claim fraud until Matt has had time to sort out what happened. In general, however, I think all claims made by ANY of these companies need to be taken with a grain of salt. That includes claims by the major closed sourced models who are actively trying to raise absurd amounts of money. Everything with “Reflection” was at least claimed to be open sourced. I’m not sure what would be gained by purposefully faking something and then releasing it all?

[nathanieledwards]

Chants: "Berman, Berman, Berman, Berman!"

You're doing great! I'm glad you cover all new models, and your coverage throughout this case (the question of accused fake models or dishonest actors) strengthens the need for you and people like you! We, as a society, need more people covering "live media" like you do, and having, like you do, the backbone to question when something reported may have been false.

Keep it up! I (and I suspect many others) want to see you succeed!

Great video. Glad you addressed everything and over all, good content!

[ProbablyPOPP]

"Fool me once, bad on you..."
AI is moving so fast, you're respectfully reporting live!