The Remote Agent Experiment: Debugging Code from 60 Million Miles Away

Google Tech Talk
February 14, 2012

Presented by Ron Garret.

ABSTRACT

The Remote Agent Experiment: Debugging Code from 60 Million Miles Away

The Remote Agent Experiment (RAX) was an autonomous control system for an unmanned interplanetary spacecraft called New Millennium Deep Space 1 (DS1). In May, 1999, control of the DS1 spacecraft, a $150-million asset, was handed over to the Remote Agent software for three days. It was the first -- and, to date, the last -- time that an interplanetary spacecraft has been under fully autonomous control. RAX was a resounding technological success, but a political disaster. Instead of paving the way for future autonomous missions, RAX is the reason that NASA has not flown an autonomous mission since. This talk is about the lessons learned from an ambitious but ultimately failed attempt to introduce technological change into a large, bureaucratic organization, the limitations of static code analysis, and the unique challenges of debugging code when the round-trip ping time is 45 minutes.


Dr. Ron Garret is a software engineer turned entrepreneur and angel investor. He has co-founded three startups and invested in a dozen others. In a previous life he was an AI and robotics researcher at the NASA Jet Propulsion Lab where he led the development of one of the four major components of the Remote Agent. In 2000 he went to work for what was at the time an obscure little Silicon Valley startup called Google, where he was the lead engineer on the first release of AdWords, and the author of the Google Translation Console. He is currently working on launching a new startup company.

[HueyTheDoctor]

Fascinating presentation, very glad I stumbed upon it, thanks for the upload!

[lawrencebottorff]

What I find most difficult about functional programming politics is how poorly its advocates advocate. They don't very well. Ron Garret's original article "Lisp at JPL" was fairly good at explaining some of the magic about Lisp. P. Graham does some evangelizing, too. But mostly FP people just get cozy in their ivory towers and cry crocodile tears over how filthy the masses are. FP should take a page from Microsoft. MS bends over backwards to assist its developer base.

[ericpmoss]

1:09:35 -- this is why things don't get better. If people only talk about a problem after the recording stops, how will it ever get fixed?

[lawrencebottorff]

(continued from below) FP/Lisp/etc needs to come back out the rabbit hole and do some serious PR work. Garret's take-away is obviously saying the same. Smooze, hand-hold, etc. ... until it catches on. Start with a really good book. I'm not saying they're all bad, but the beginner is often forced to take things on faith, postpone questions -- all the while not understanding the gains, benefits, not able to contrast why the FP approach is better.

[jonathanjohnston]

Interesting from a Nasa / technology POV, some still relevant in 2025.
Also disappointing, as the "debugging using a *remote* REPL" is glossed over took quickly. Glad they were able to find the deadlock & fix it, but there was too little detail about the actual interaction using the REPL. Or, at least too little for a REPL enthusiast.😊

[ericpmoss]

Did I hear this right at 24:50 -- 3 Lisp agents running in the same image but forced because of politics to communicate with each other via a separate buggy student-developed C bottleneck? And people blamed Lisp for problems?!?

[kap]

We at the #Clojure world are trying to bring #Lisp back. Join us!

[aoeu]

Anybody have any Lisp projects that are cool and I can contribute to?

[CDelpino]

REALLY ARE THESE PEOPLE ON DRUGS AS FAR AS THE FCC THE MAX RANGE KNOWN TO REMOTE CONTROL ANY DEVICE IS 65, 000 FEET! ''WTF''

[CDelpino]

REMEMBER 65, 000. FEET AS FAR AS ANYONE IS CONCERNED TO DATE REMOTE CONTROL MAX RANGE TO DATE IT 65, 000 FEET FCC US GOV. WHAT ARE THESE CRAZY CLAIMS OF CONTROL IN SPACE CHECK IT OUT