Debunking Devin: 'First AI Software Engineer' Upwork lie exposed!

Recently, Devin the supposed "First AI Software Engineer" was announced. The company lied and said that their video showed Devin completing and getting paid for freelance jobs on Upwork, but it didn't show that at all.

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
I'm not kidding! Go watch the original poster's video!

I broke down the Devin Upwork video frame by frame, and here I show what Devin was supposed to do, what it actually managed to do instead, and how bad a job of that it did.

On the whole that's not surprising given the current state of Generative AI, and I wouldn't be bothering to debunk it, except:
1) The company lied about what Devin could do in the video description, and
2) a lot of people uncritically parroted the lie all over the Internet, and
3) That caused a lot of non-technical people to believe that AI might replace programmers soon.

00:00 Intro
00:30 The claim and the problem
03:47 What the job actually would have required
05:50 Requirements that needed to be determined
07:32 How a human compensates for Upwork's lack of RFP process
10:11 What Devin did instead, and how poorly
13:03 Devin seems to be fixing code from Github
14:19 But Devin is actually making up errors, and then fixing them
15:40 All Devin had to do was run the command from the README
16:23 But Devin couldn't figure that out
16:26 So it created this nightmare 'C'-style low-level buffer append loop in Python
17:25 My replication of what Devin tried to do
18:15 It took me about 36 minutes
20:16 It took Devin at least six hours, and maybe more than a day
20:48 more bad Devin code
22:08 List of useless things that make Devin look competent
23:48 Conclusion, and a Plea

Video falsely claiming "Watch Devin make money taking on messy Upwork tasks!" that I'm debunking:

My video about communication being the most important part of software work:

My video about why Devin isn't really a "Software Engineer":

[Geekraver]

99% of AI videos on YT are clickbait garbage. Especially the ones saying AGI is coming next week. Thanks for contributing to the 1%.

[PraveenKumar-bofw]

I really hate how normalized faking it in demos has become

[samwight]

The thought of an AI having to debug its own awful code for six hours in a Sisyphean hell is the funniest thing I've seen all week

[stanleyparks]

AI has no accountability. "We're missing deadlines because the AI isn't doing what we ask it to do!"

[SterileNeutrino]

22:15 Devin has already mastered the Art of Looking Busy 🤣

[Hardwareai]

"Devin is generating its own errors and then debugging and fixing the errors that it made itself." But that's EXACTLY what I do in my job! 😮‍💨

[rayhere]

For crimes against humanity Devin must be sentenced for immediate self deletion.

[EricB]

Quotable: "And if you're just someone who's using the Internet now, please, for the love of all that's holy, be skeptical of everything you see on the Internet or anything you see on the news, especially anything that might possibly be AI related. There's so much hype out there and there's so much stuff that people are bouncing around and saying to each other is true. That's just not true. So please just don't forget to be skeptical. It's important."

[vishalmishra]

You don't mess with a guy with 1000s books in their background!

[isaackoz]

Another thing to keep in mind is that a lot of these tech startups have one goal: to make money. How do they get money? Investors and/or hype. I guarantee this exaggerated demo along with all the articles and influencer videos that came with it convinced investors somewhere to pump money into this— which is exactly what this company's goal is. Companies will lie (if they can get away with it) if it means more money. It's no surprise this company is lying to get hype and investors interested.

Thank you for taking the time to expose them!

[devinrayolsen]

As a developer named Devin.. I feel awkward..

[JT-mrdb]

When I saw the demo, my BS meter was being triggered and I knew someone would come along and closely look at this. Nicely done!

[eugenefritz]

Ahh finally found a non trendy channel.

A real person that actually works in the industry for a long period, can share his experiences in a transparent manner.

That’s a rarity in the internet.

It’s like finding gold in the desert.

If you continue like this Sir, we’ll continue to support you.
If you change we’ll stop it.

Thank you for sharing your insights, you’re valuable.

Cheers.

[apexphp]

Yep, you nailed it. Doesn't matter though, because Cognition Labs reached their goal -- raise tens of millions from investors. That was their goal, not so much the quality of Devin. Not to mention all these AI software engineers are Chat GPT 4 wrappers, hence are probably costing $400/hour in API calls to run.

[dminik]

I remember when Gemini(?) was showcased by google. It was a nice realtime demo showing a person and the AI agent interacting through casual voice chat. Gemini was also apparently looking at a video (also in real time).

I was actually amazed. That was a big jump ahead compared to ChatGPT. I thought that the latency was probably worse, and maybe the video fed in was compressed and missing frames.

Turned out that the whole demo was faked. Instead of a casual conversation, the AI was fed related, but not the same, lines through text. Instead of a video it was an ocasional screenshot, also annotated with a prompt. The voice wasn't even generated by gemini (It may have been voice acted).

I was actually flabbergasted. Like, there wasnt a single second in that video that was truthful. It was like 3 minutes of pure lies and false advertisement. I still dont understand how they thought that was ok.

[andrewyork]

I dont get why people who have no clue what it is a software programmer does insists on making these claims.

[genericgorilla]

the harsh truth is that investors won't really pour money into a fancy copilot. they want to replace ALL labor. this is why stuff like the amazon self-checkout stores are going bust, it's not that the technology is inherently fraudulent or anything like that, but the fact that at some point of the process, it needs human intervention, even if just to check if the computer encountered an edge case or not, makes these concept 100% unviable in the eyes of investors. it's an insane hype cycle that might as well end up on another winter, because it sure as hell won't produce robot butlers.

[halidmohammed]

As a junior developer feeling the pressure, I’m glad I found you early 🙏. Thank you

[PROdotes]

The problem with modern journalism is that, back in the day, someone writing a tech article had some kind of education on it, they would know what the jargon means, they would be able to read a scientific paper on the field they write for, and understand a good part of it...
Now... we just have a bunch of people that get paid per article, so they just find interesting headers on the internet and write a few sentences about it, cause the company they work for pays the same for that, as they would for a well written and researched article... so why bother...

[Justcetriyaart]

yep, us illustrators are also running into this too. too much hype of what the tools can do and the large part of the job and the QnA with clients and interation and fixes. Plus the majority of the AI work is only based on generic work and most of the data are from hobbiest not professionals that make designs for products to sell vs just something pretty for fun