I Tried Every AI Coding Assistant

Which of the AI coding assistants will turn you into a 10x developer? ChatGPT? GitHub Copilot? Tabnine? AWS Code Whisperer? Bard? Sourcegraph Cody? CodiumAI?

Prepping for your frontend interviews? Use code "conner" for a discount on my product FrontendExpert:

Timestamps:
0:00 Tabnine
4:31 ChatGPT
5:37 GitHub Copilot
9:43 Bard
10:49 AWS Code Whisperer
13:51 Sourcegraph Cody
20:40 CodiumAI

[ConnerArdman]

Correction on Sourcegraph Cody:
I wasn't fully using this one correctly. You can download an additional app and link it to your GitHub repository, and it does a _much_ better job of understanding the repo. I'm not sure why the UX doesn't prompt you to do this from the start, but it does fix some of the problems in the video such as over-indexing on node modules. However with all that said, I honestly still wasn't able to get it to perform many tasks that were particularly useful, and the larger the repo I gave it, the less reliable the responses seemed to become. It could definitely be worth a shot in your projects to see if it is useful to you though! 😃

[nikkehtine]

I like that the video starts by jumping right into reviewing the extensions instead of a long intro that ends with a sponsored segment. You get points for that from me

[deanschulze]

The examples he uses (factorial, mergesort) are trivial with simple, well known solutions implemented in almost all languages. No one writes their own mergesort or factorial function.

I'd like to see a comparison of these various tools using something more realistic. For example how well would they implement envelope encryption using various cloud providers? I hope they would produce code that is at least a good starting point for envelope encryption. If I get some time maybe I'll do my own trial using these various tools.

What would be some other realistic examples to test these tools with?

[ridewarior]

I literally haven’t seen your channel before but I gotta comment about how much I appreciate that your video just starts. There isn’t 3-5 minutes of just bs when the topic is literally explained in the title. THANK YOU!

[bytor]

Excellent video. What I love about your presentation is the speed. You speak at a great speed that gets to the point, and gives us exactly the amount that we need without any fluff. So many times when I watch other people's videos they go slowly and I just have to speed up the video to 1.5 speed at least to get through it. This is my first video of your I have watched and I am really glad I did. Thanks.

[mariusj.]

We need a comparison with more non-standard functions/customized requests.

Factorial is too likely to be an example in the training data of the language models.
To determine these tools' capabilities and usefulness for actual work we'd need some tests to get code that solves a problem the tool most likely hasn't seen before, or atleast not with the same constraints or not with the same custom properties (like sort this array of strings alphabetically and add an argument to select whether to treat uppercase and lowercase the same or have all lowercase letters come after all uppercase letters like ASCII code based sorting).

[moralfuxery]

Im only about 8 months into my psuedo coding journey (im in school for InfoSec, and programming takes a bit of a back seat as its not necessary to build a program, only to understand how said code functions so we can break it) but co pilot as been an absolute godsend. Not to code for me, but with lack of any kind of instructor to assist me, its sorta like a mini teacher. I dont let it do it for me, rather help me understand what i did wrong.

[gaweyn]

from wikipedia <170 (number)>: 170 is the largest integer for which its factorial can be stored in IEEE 754 double-precision floating-point format. This is probably why it is also the largest factorial that Google's built-in calculator will calculate, returning the answer as 170! = 7.25741562 × 10306.

[spirit]

This comparison would have been more useful (perhaps) if the selection of tasks posed to each agent were non-trivial, novel, non-encapsulated tasks. Ie: something you couldn't just google "how do I code <x>" type questions to find a definitive answer.

[nikkehtine]

The list is missing Codeium, which I use a lot and is pretty spot-on more often than not in my opinion. Wanted to see how it would rank against Copilot and similarly named CodiumAI

[omnikong]

5:55 It suggested "factorialize" because it's a verb-ized form of the word "factor, " since you're writing a function. If you were writing some property, the noun form would be preferred.

[adampatterson]

Cody works really well in projects since its context comes from your entire project and not the single file you're in. It's also doing this locally if I remember correctly by creating it's models in PostgreSQL.

I'm not sure if the others do this but 'How can I get XYZ from ABC model" has workers really well, or Create Test for this, will often bring in correct model relationships and factories in Laravel.

[falklumo]

This was a nice comparison. Although the test cases were too simplistic, TBH. E.g., you could have asked to complete code to parse a snippet from a text file given within a preceeding comment. Stuff which isn't literally found in text books ...

[lucaslopes.]

I love how this is so straightforward, thank you

[karelmaly]

Also, Cursor IDE and their coding assistant could be mentioned. It works pretty well for me :)

[jimmyj.]

Super interresting comparison. Thanks Conner for this actual topic 🙏

[toterpinguin]

As many other comments have pointed out already, you used very standard functions every computer science student knows about (factorial, mergesort) for testing the AI assistants. The problem is, these functions are most likely occurring many times in all of the models' training data - and obviously computers can memorize and parrot smart-sounding answers / code. If you do a round 2, maybe you could consider tasks that are not really that hard at all, but a little off the beaten path, you can't just google the solution which also means it probably is not in the training data

[patricknelson]

22:36 - What’s interesting is that I think most compilers that perform tail recursion optimization do the same thing (i.e. simply convert them into a loop).

[tekrunner]

My company is probably going to go with tabnine (starting to test it now), mainly because their models can be deployed on-prem. Also because they're modular, one model per language, and they say they'll update them whenever new versions of languages / big frameworks come out.

It's also supposed to have some degree of repo awareness (they told us they stuff the context with the content of files that seem related to the current one), and full repo awareness is on their roadmap (if it does work, it should be a big plus compared to generalist LLMs).

That said, its reasoning capabilities are definitely not great right now.

[SuperMixedd]

in my experience (and I code primarily in R) chatGPT 3.5 gives code that doesn't really work, straight up makes up packages/functions (phantom memories) to solve the issue, and in general is only able to help me 1 out of 10 times