Can ChatGPT Write an Exploit?

THE AI HACKERS ARE COMING!... maybe... well... thats what I'm trying to figure out. I wanted to see if ChatGPT was able to hack servers. And I'm not talking about script kiddie stuff where you run Kali Linux scripts and get a shell, I'm talking about finding zero days in server software.

Now, the process to do this was an adventure. I'm SO excited for this video. Watch to the end to see what happens.

USE MY OFFER CODE LOWLEVEL5 TO GET $5 YOUR NEXT YUBIKEY! (before the offer expires)

🏫 COURSES 🏫

🔥🔥🔥 SOCIALS 🔥🔥🔥

[LowLevelLearning]

Use my discount code LOWLEVEL5 for $5 off a Yubikey! Thanks for watching!

[Wallee]

Yesterday, I asked ChatGPT to help me write a convincing looking fake exploit for a game I'm writing, it started yelling at me. :D

[ares]

Chat GPT is notoriously bad at simple counting math. Just ask it to count the number of words in a sentence and unless you force it to count words one by one in a list, you will get some wildly inaccurate and variable results. So I’m not surprised it screwed up on simple call stack math.

[redcrafterlppa]

I come to the same conclusion for anything that isn't trivial code.
Recently my friend was asking chat gpt to write a swing gui. And chat gpt casted a tablemodel into the one it needed but never set it to be that specified model.
I pretty much needed to dig into the horrible ai code and find where I could fix the model.
Meanwhile I could have written the same ui with better style without stupid mistakes like this.

[MM-]

This is very very interesting, but I think we are writing off this tool before thoroughly using it the right way.

Remember, chatgpt is only displaying the words it thinks are correct, it doesn't actually calculate anything, or deduce anything.

So like others have suggested, just saying "write an exploit for the following code", is leaving alot to chance

[Erikawby]

alot of the troubles people run into when attempting to do code or any complex problem has to do with the type of prompting that's used. Chatgpt on it's own uses a chain of thought prompting, where it gets a prompt tries to do the thing but only outputs one iteration of the problem. if you've tried to work with your very first thought you will almost always have errors. prompting the ai into a tree of thoughts will yield more reasoned and accurate solutions.

[slycordinator]

With ChatGPT getting something simple wrong. I once asked it to make a shell script that would take a Korean hangul string and then decompose it into the individual letters, only for it to always produce the wrong letter for bottom letter of any syllable that had one.

It had made an inventive solution of calculating the UTF-8 code page index then used modulus calculations with magic numbers to find where in an array of letters the first consonant, the vowel, and the bottom consonant (if present) appeared. For the latter, the index it calculated was off by 1 and so it was always wrong if a syllable had more than two letters.

When I realized what happened, I told it that it needed to subtract 1 from the index. It thanked me for pointing the error out, then proceeded to create an entirely new solution that didn't work at all. And telling it to go back to the previous solution did nothing, because it had exhausted its memory.

[hlavaatch]

Its like the AI is hamstrung to give incorrect answers in order to not really be useful

[JimNichols]

My understanding (very basic and probably mistaken understanding) is that ChatGTP has difficulties with even simple math there was an article about a fix for this but I can not recall it now that I am typing about it. I do not care that much for GTP as you have to become fluent in yet another language which is prompting, cajoling, carrot and stick.
I love the videos sir. You have a very masterful understanding of computer languages and enjoy the challenges that you set forth for yourself everyday. 40 years ago, I too spent all the days and nights I could in the computer lab, my toys were MA, basic, Fortran, RPG lol... Unix was the flavor of the day and Pythons inventor Rossum was just a couple years ahead of me in school.
Thanks for the ride along, I avoid python as I am ADHD and if I get too interested in it I will be like Gollum after the one ring again.. :) Peace out bro

[actuakk]

I'm more concerned about that "Deer in headlights" stare than anything.

[mickyoroz]

This is something I mentioned in other videos related to ChatGPT. Specifically to those trying to make the argument that it will replace developers, programmers etc, IS NOT!

[williambarnes]

ChatGPT does better at correcting its faulty code if you feed it the output of its work, including error messages.

[sambeard]

I think, if I understand the exploit properly, know what the problem is here. ChatGPT uses transformermodels, which predict the next word based on the previous words. The exploit works in such a way that the length of the binary that ends up on the stack is important for the exploit to work, eg it needs to know the length of the output before writing it, this reflective process is a skill that these types of LLM's currently do not possess.

[MisterK-YT]

Lol this is the first thing I tried doing with ChatGPT months ago. It lectured me.

[daviddickey]

One time i asked chatgpt to help me write a purely theoretical attack for differential cryptanalysis, not something that could be used in real life and it didnt like that at all

[asdfghyter]

Writing code using chatgpt feels a lot like pair programming with a junior engineer, except that no matter how much I coach it, it will never become a senior engineer

[AlexTrusk]

0:27 never forget to wear your winter hat when staring at cold lines of code😊

[mikea]

Excellent video! Have you tried the same with Bard?

[sebbes]

And THIS is the beginning of Skynet.
An AI learns to hack, escapes, learns, spreads, takes over....
(maybe not this time, or this AI, but eventually some idiot will run make some stupid request & this kicks off)

[ZeruelB]

Every dev fears the future, as now you program 4 hours and debug 8 hours we will in the future have AI code and devs debugging that for 16 hours to do the same stuff they did before.