Three Statistical Tests Every Game Developer Should Know

In this 2016 GDC session, Insomniac Games' Elan Ruskin gives a how-to on statistics for answering questions like "does this new camera control scheme make players happier?", "how many players do I need to test this design change on to prove whether it works better?" and "does the framerate really get faster when I do this thing or is it just a fluke of measurement?

GDC talks cover a range of developmental topics including game design, programming, audio, visual arts, business management, production, online games, and much more. We post a fresh GDC video every day. Subscribe to the channel to stay on top of regular updates, and check out GDC Vault for thousands of more in-depth talks from our archives.

[ToriTheChicken]

Some of the GDC talks are very badly presented for YouTube videoes. Not this one. This was great, in just about every way.

[iestynne]

I'm proud to have worked with Elan for several years. As you can tell, he always puts a great deal of effort into preparing for his presentations. Amazingly though, this is actually his normal level of conversational speed, clarity and humor :)

[PrimerBlobs]

"Any actual statisticians are totally cringing." Yep. It's not just pedantry. People will literally not know what their test means, and then they will judge whatever change they make in hindsight anyway.

[_lime.]

13:00, this is a really good one. With Minecraft, Mojang came to a realization that very few players had ever been to the Nether (based on the percent of the population that had the achievement "We need to go deeper!" which is received upon entering the Nether). The ended up realizing that very few non-hardcore players (players that didn't consume game related content outside of the game, like videos, guides, articles, etc...) knew that the Nether existed.

This is why the added obsidian monoliths and broken portals around the Overworld to give you hints.

[infcaat]

wow, he is a fantastic speaker. charismatic, to-the-point, funny and practical.

[ReeseEifler]

Not only is this an amazingly useful talk, it's essentially a perfect presentation. Dope shit.

[kittykittylicization]

As a Biologist (MS)... i was indeed shouting at my screen when you were talking about P values....and then you called it out so im happy now.

[colinstreck]

That was fantastic. Their presentation skills are off the charts.

[Vospi]

As an educator and a grateful listener: that was bril-li-ant.

[ArsenicDrone]

While I wouldn't do everything identically, I didn't have any large complaints, which is not generally what happens listening to quick statistics intros. A good talk.

[Discipol]

Excelent presentation. THIS was simplified? I am afraid of the scenic route xD I wish to know more, and more practical applications on game dev.

[ZZaarraakkii]

@12:53 A thing to note is that in that example, people have been playing the "hard" puzzle before and the "easy" puzzle is a novelty, which may cause players to spend more time on it for the experiment, without it being the better solution long term.

[Brindlebrother]

People are awful at five-star ratings whether that be a game, book, movie, show, item, etc. Basically, people will give 4-5 if the product was at all fun or engaging, or a 1 if there was a problem/complaint/issue or any offense taken.

Good video. Statistics are fun.

[NunSuperior]

Thanks for the talk. I had to learn this stuff on the job at Big Software, Inc. when we started measuring PC boot time impact. There were large variances between each boot.

[maximeflageole]

More interesting and useful presentation about statistics I've ever watched.

[phillipA]

a semester of stats in 30min. thanks guy.

[jonasnockert]

Love this talk! I spent quite some time trying to derive the 8.14 confidence interval in the first example and finally had to install Excel to verify. I couldn't see it at first but the slides actually mix five and six observations. At ~7:38 there are five observations. At 8:19, the confidence interval is calculated using six observations, i.e. T_DIFFMEANS(A2:A7, ...) rather than the 2 x 5 observations shown on the left.

[JohnDoe-mxsq]

This video has existed for almost 4 years and it feels like not a single game dev has ever watched it.

Their sales division has warehouses of supercomputers simulating human brain functions trying to figure out how crap a game can be before you will buy it, and just how much you will spend on DLC just to play the game at all.

[lanord]

The first talk where I needed to decrease the playback speed instead of increasing. Great material! =)

[KillerBearsaw]

Absolutely fantastic presentation, would love to hear him speak more