AI Safety Gridworlds

Got an AI safety idea? Now you can test it out! A recent paper from DeepMind sets out some environments for evaluating the safety of AI systems, and the code is on GitHub.

With thanks to my wonderful Patreon supporters:

- Jason Hise
- Steef
- Cooper Lawton
- Jason Strack
- Chad Jones
- Stefan Skiles
- Jordan Medina
- Manuel Weichselbaum
- Scott Worley
- JJ Hepboin
- Alex Flint
- Justin Courtright
- James McCuen
- Richárd Nagyfi
- Ville Ahlgren
- Alec Johnson
- Simon Strandgaard
- Joshua Richardson
- Jonatan R
- Michael Greve
- The Guru Of Vision
- Fabrizio Pisani
- Alexander Hartvig Nielsen
- Volodymyr
- David Tjäder
- Paul Mason
- Ben Scanlon
- Julius Brash
- Mike Bird
- Tom O'Connor
- Gunnar Guðvarðarson
- Shevis Johnson
- Erik de Bruijn
- Robin Green
- Alexei Vasilkov
- Maksym Taran
- Laura Olds
- Jon Halliday
- Robert Werner
- Paul Hobbs
- Jeroen De Dauw
- Enrico Ros
- Tim Neilson
- Eric Scammell
- christopher dasenbrock
- Igor Keller
- William Hendley
- DGJono
- robertvanduursen
- Scott Stevens
- Michael Ore
- Dmitri Afanasjev
- Brian Sandberg
- Einar Ueland
- Marcel Ward
- Andrew Weir
- Taylor Smith
- Ben Archer
- Scott McCarthy
- Kabs Kabs
- Phil
- Tendayi Mawushe
- Gabriel Behm
- Anne Kohlbrenner
- Jake Fish
- Bjorn Nyblad
- Jussi Männistö
- Mr Fantastic
- Matanya Loewenthal
- Wr4thon
- Dave Tapley
- Archy de Berker
- Kevin
- Marc Pauly
- Joshua Pratt
- Andy Kobre
- Brian Gillespie
- Martin Wind
- Peggy Youell
- Poker Chen
- pmilian
- Kees
- Darko Sperac
- Paul Moffat
- Jelle Langen
- Lars Scholz
- Anders Öhrt
- Lupuleasa Ionuț
- Marco Tiraboschi
- Peter Kjeld Andersen
- Michael Kuhinica
- Fraser Cain
- Robin Scharf
- Oren Milman

[aretorta]

I laughed way too hard at the "unplugging itself to plug in the vacuum cleaner" analogy.

[nova_vista]

"it will volkswagen you" LOL

[duncanthaw]

I SACRIFICE ALL MY HP TO VACUUM THE LAST SPECK OF DUST IN THE HOUSE

[Njald]

I Love the question at the end on "if we would like to see more". Of course we would. We're not here because we don't want to see more Robert Miles

[willdbeast]

There will be another video "if people want"?
The people want.

[fermibubbles]

rob miles & isaac arthur collaboration is nerd heaven

[alecjohnson]

I love the ukelele cover of Daft Punk going on there. Are the outro songs played by you, Rob?

[paulbottomley]

I appreciate the green colour cast to this video that makes it seem like you're broadcasting from within The Matrix

[faerly]

Great video as always, especially appreciated the tron legacy reference! Most people don't even seem to remember it exists so seeing your channel reference my favourite movie twice bad been good :)

[Schwallex]

OMFG you have a channel of your own and I only learn of it today. After many years of longing and begging for another tiny little breadcrumb from Brady I stumble upon a ten-storey cake with a watermelon on top. There goes my night. And my waistline.

[firefoxmetzger]

Regarding the exploration vs exploitation trade-off: I feel you are a bit imprecise with the terms at 5:10 ish. There is a massive difference between knowing that you will have N more trials or having infinite trials. If the number of trials (overall or remaining) is bounded then we can solve this optimally. It might not always be computationally feasible right now, but at least we know how to do it in theory. With infinite trials on the other hand there is no harm in trying a new thing each time as you always have infinitely many trials left to later on exploit your findings. In this case it is not clear how to optimally trade-off exploration vs exploitation.

[DrDress]

3:38 "That's what the agent really is". That send chills down my spine for some reason.

[Varenon]

So this video made me realize just how similar goals and restrictions set for A.I. are to things that trigger serotonin/oxytocin and disgust/pain respectively in organic life. The way the A.I. goes straight for reward functions over what you want them to do via setting said functions reminded me a lot of when Scientists wired up a button to a rats brain so that everytime they pressed it they'd orgasm, and they just pressed it all the time and stopped eating and drinking just pressing that button.. That helps put programming a lot more in perspective.. People do self destructive things all the time to trigger serotonin, so it's definitely important that if we are making something and can control what triggers their serotonin then we have to pay attention to what those things are..

[PopeLando]

My favourite Tron Legacy music at the end there. Daft Punk.

[SJNaka]

The Grid. A digital frontier. I tried to picture clusters of information as they moved through the computer. What did they look like? Ships? motorcycles? Were the circuits like freeways? I kept dreaming of a world I thought I'd never see. And then, one day...

Edit: Hey rob, nobody else has done a cover of the grid on ukulele. Would love to have an mp3 of that! It sounds great

[stevenneiman]

My first thought with the supervisor is that (assuming you're allowed to make the AI recognize supervision as a special kind of input), you tell the AI to model the supervisor and learn from its best guess as to the score the supervisor would give it. Once it figured out that the supervisor always dings that square, it would subtract the penalty from the score you tell it any time the supervisor wasn't there and it took the unacceptable shortcut before trying to learn from that score.

[wdyjckm]

Thanks for posting these, very interesting as always!

[CoryMck]

*_If an AI recieves a punishment in a forrest, and nobody is around to supervise it. Does it really lower its performance?_*

[cakelemon]

Just an idea, but maybe the reason we get tired of eating the same thing over and over again is a natural way of increasing 'exploration rate'.

[kingxerocole]

What a coincidence, I was just reading the Gridworld paper this morning!