Evolution vs ChatGPT-4 | A Stated Clearly Review

In today's "Stated Clearly Review" (a thing I'll be doing every once in a while from now on) I go over ChatGPT-4, the new A.I. model taking the world by storm. I ask it common questions about evolution to see if it "understands" the science well enough to teach it.

The questions start easy and get harder, the final question requires extreme speculation to answer, thus testing ChatGPT-4's ability to imagine. Questions were submitted by patrons: Jeremy Smith and
Michael McGuffin.

The development of ChatGPT-4 is an amazing achievement. I think it has potential to do great things for education, but ChatGPT-4 is also worrisome for several reasons, including its tendency to gaslight users, hallucinate fake information (and give to users with full confidence) and it's not clear how ChatGPT-4 will be viewed by users with specific cultural sensitivities.

For example, suppose I were a Young Earth Creationist just wanting to dip my toes into the science of evolution. If I would have asked ChatGPT-4 the questions from this video, its responses might have been a bit too cold and intense for my needs.

The future will be strange, hopefully in a good way.

#GPT-4 #ChatGPT #AI

[RyanSmith-onhq]

A tip when using GPT-4. Don't waste your time arguing with it, as this can strengthen the gaslighting 😂. If it doesn't provide the answer you want, simply edit your last question and submit it again. Due to its probabilistic nature, you may get different answers for the same question. Each time you prompt, it sends the entire message history in that chat window to the API. This is how each window maintains its own context. All of the messages are sent to the same model, but your message history provides it with context. If it gives you an incorrect answer and you continue the conversation without 're-rolling' for a new answer, the incorrect response becomes part of the context and will be used in the decision-making process.

[DampeSN]

About the argument you had with it. You mentioned something about it not digging deep enough into its database. One of the hardest parts to grok about how GPT (and LLMs generally) work is that they don't store knowledge the way we're used to with computers. What it was doing was picking up on clues in how you were interacting with it that suggest that you _expected_ an argument. LLMs are really complicated auto-complete. This makes them extremely useful for many things, but what they aren't doing is pulling knowledge from a store of facts. It is a lot more like those Exquisite Corpse drawings you can make with other people. The way the chat systems work can be thought of as if the whole conversation is being passed into an auto-complete with "He said:" and "She replied:" after each other, and then it predicts what text should come next.

[flensdude]

I think the reason why ChatGPT didn't give you the term "techno-unit" is because ChatGPT is more of a language model than an information model that is inquiring, qualifying, accumelating, improving, and sharing information. Let me explain my thinking.

In human language, we don't always comply to the requests of others. Instead, we also get into arguments, we lie, we keep secrets, and we misinform without correcting ourselves. All this despite the fact that we could have answered in accordance to what is asked of us.

ChatGPT is simply just modelling as many forms of written human interactions it can, incidentally even arguments, lies, secrets, and misinformation. It obviously has some constraints and biases that lead it into certain types of interactions rather than others. But the point is that it's not modelling processes on how to inquire, qualify, accumelate, improve and share information, which was the kind of interaction you expected to have with ChatGPT.

It doesn't care about the truthness of its conversations. It just cares about *having* a conversation based on how it perceives how humans interact.

[kedrednael]

In a way the GTP LLMs are similar to the image generators like Midjourney and DALL-E. Those are trained on many photo's and paintings, but you cannot ask it to exactly recreate a specific training image. The data is heavily compressed in a complex way. This is the same with GTP, you cannot get extremely specific niche facts which it only saw one or two times. It has a hard time recreating exact training data text. Remember, it is not connected to the internet anymore, everything it outputs comes from the neural network.

That is scarily similar to humans: we very quickly forget the exact wording someone used, but we still remember what they ment.

[Aliasjax]

The most thorough defense of memes, Dan Dennett's "From Bacteria to Bach..." Also, Steve Stewart-Williams attempts a synthesis of Heinrich's work and memes that's worth a read in, "The Ape that Understood the Universe."

[rizean]

Have to say that EPIC intro hooked me to stay and watch. Nothing surprising here. Work with GPT4 daily on coding projects. Generally, it's amazing, but occasionally, it will just make up methods that don't exist and call it an optimization. 😂

[bazoo]

5:05 - "Neutral" meaning that no particular adaptation has inherent value, "better" or "more evolved" than any other. They have value only in the context of a particular environment; sometime loosing a trait, or simplification, can be advantageous, and should not be viewed as "regression".
In other words, I think that "neutral" is not a bad term, that only needs slightly better definition.

[eefaaf]

The problem with this kind of system is that, if you already know what the answer should be, you can judge the response to be correct or complete enough.
Once you start with triggering responses you don't know about beforehand, there is no way to judge it, as right or wrong, it sounds quite authoritative.

[UntoTheLaot]

You should start a GPT focused YouTube channel. So many people could benefit from that!

[stuwest]

Great video Jon. Super cool idea. Made me ask it some questions about natural selection, which is got partially right, partially wrong. Awesome Hal.

[flyingfree]

Neutral just means that evolution has no bias, motivation, goal or agenda.

[mr.spinoza]

5:16 the word "neutral" was fine. It wasn't speaking about what's effective, it was saying it doesn't follow a normative standard of "good" and that it's neutral with respect to direction and teleology.

[cowboyfrankspersonalvideos]

A point that the media has missed when talking about AI is, any computer based software system is no better than the data it has to work with. I assume that ChatGPT-4 is using the Internet to gather its data. So what happens when it also starts assimilating incorrect, false or misleading data? That's when I start getting worried.

[Obscurai]

The apology was most surprising. An apology implies knowledge of previous intent and a subsequent change of intent. Fascinating.

[tomasnemec]

It is not at all strange to see GPT contradict itself. People tend to forget that the machine does not think. It has no concept of understanding what it outputs. Given a text input it merely links output words one after another based on a very sophisticated training.

[billwaterson]

I've only played with it for a few hours. But what I've observed is exactly what you're illustrating. I have to assume it's learning how to adjust to certain unwanted semantic barriers.
It's baiting you into solving for the human version of "bullshitting" it.

[YonsuKu]

Always good to see an expert challenge an advanced generative AI and challenge its response. Thank you for your work

[stan]

I know I'm a little late, but I stumbled upon this video by accident as an AI researcher. You asked GPT whether it had some data in its database and later, after 'arguing' with it, it suddenly knew something that it didn't want to share before. This is a common misconception about large language models. They don't have a database, they don't contain a knowledge base per se. GPT is, in its core, nothing more than a "text completion" model. Given an arbitrary amount of words, it will calculate the most probable next word. In a sense, it is very similar to word prediction technology in smartphone keyboards, just much more complex than before.

The main difference, however (without getting too technical) is that this is a model with 175billion parameters (GPT4 has slightly more, not the 1 trillion you sometimes hear). Imagine a graph with 175 billion axes. Eacht point on this grap, denotes a specific 'meaning'. Additionally, transformer models, like the ones GPT are (general pretrained transformer = GPT) implement a technology called "attention layers". These are special layers in a neural network, similar to what a specialized region in a brain does - that, simply said, functions as a memory. A neural network consists of millions of pathways that a given input can take, but an attention layers gives priority to some, based on the task.

For example, if you ask GPT to write a poem, we can see that the attention layer will give extra attention to the word that appears right before a newline. In other words, if you add "poem" in your request, it will pay attention to the last words in a line, since that is where rhyming takes place, an important part of poems.

This being said. There is no database it pulls from. GPT was probably trained on all information you requested, it "knows" this data (or rather: given the right guidance, it can complete your input in a way that all source data comes from the source you are referring to). This is why it "suddenly" knew everything, once you gave it a push in the right direction. This is also why it might start hallucinating when you push it into a direction that implies bs information.

I believe it is very important that people understand this technology. It is very powerful and very impressive. I work daily with GPT, you might even say that I talk to GPT more frequently than to humans, but understanding its inner workings also gives insight to how we can use this technology. GPT already showed impressive capabilities in your example, if you had used the API with a clear system message, I suppose it might have answered all questions correctly, without a need to argue.

Very soon these inconsistencies, where it 'knows' something, but is unable to retrieve the information, will be solved. You can already circumvent this by opening another GPT window, or using the API with a completely new chat history and a specialized system messagem, and ask it to review the previous responses. By having two 'instances' interact with each other, converse with eachother until they reach common ground, you will eliminate these inconsistensies.

I hope this will help you out. I see so many very useful implementations with GPT daily, and I truly believe this technology and everything that will follow can change our lives, if we can guide it, but I also see a lot of mystery around the technology.

[MalachiMarvin]

14:11, I find that one of the tricks in prompting these chat AIs is recognizing that what was discussed earlier in the chat will in part inform subsequent responses. This is good because it means I don't have to repeat things, I know it has the context to my next question. Other times though, it's just better to start a new chat. I'm guessing in this case, because it already went down the path of 'thinking' there isn't such a term that became part of what informed the responses that followed.

No point in thinking of it as 'gas-lighting' because it has no such intention.

[jorgwei]

I think it is a mistake to conceptualize it as "going through its data base". I think you should think of it as "exploring its network of associations" given the input before – which is your prompts as well as its answers. A speculative reason it didn't give you the answer: It has declined the existence of techno-point (correctly) and also gave you a lot of alternatives. From a certain perspective, the pattern both of you created was "you suggesting a term – it telling you: nah, it's something different". I think it's prone to pick up on such patterns even after going through it only once :) As Ryan wrote below, editing your prompts or forking out in new conversations is the way to go. Knowing when to do this, is one of the hard to define skills when using GPT as a tool.