Thinking While Moving: Deep Reinforcement Learning with Concurrent Control

Показать описание

Classic RL "stops" the world whenever the Agent computes a new action. This paper considers a more realistic scenario where the agent is thinking about the next action to take while still performing the last action. This results in a fascinating way of reformulating Q-learning in continuous time, then introducing concurrency and finally going back to discrete time.

Abstract:
We study reinforcement learning in settings where sampling an action from the policy must be done concurrently with the time evolution of the controlled system, such as when a robot must decide on the next action while still performing the previous action. Much like a person or an animal, the robot must think and move at the same time, deciding on its next action before the previous one has completed. In order to develop an algorithmic framework for such concurrent control problems, we start with a continuous-time formulation of the Bellman equations, and then discretize them in a way that is aware of system delays. We instantiate this new class of approximate dynamic programming methods via a simple architectural extension to existing value-based deep reinforcement learning algorithms. We evaluate our methods on simulated benchmark tasks and a large-scale robotic grasping task where the robot must "think while moving".

Authors: Ted Xiao, Eric Jang, Dmitry Kalashnikov, Sergey Levine, Julian Ibarz, Karol Hausman, Alexander Herzog

Links:

Рекомендации по теме

Комментарии

I think comparing the grasp success with blocking and non-blocking directly is unfair towards the continuous agent. As the continuous agent has to learn a lot more.

Instead I think we should compare the grasp success at a fixed wall clock duration.

herp_derpingson

Thinking While Moving: Deep Reinforcement Learning with Concurrent Control

Thinking While Moving: Deep Reinforcement Learning with Concurrent Control

Presentation of Thinking While Moving: Deep Reinforcement Learning with Concurrent Control

Thinking While Moving: Deep Reinforcement Learning with Concurrent Control

AI Learns to Walk (deep reinforcement learning)

Joelle Pineau: Reproducibility, Reusability, and Robustness in Deep Reinforcement Learning ICLR 2018

Speeding up Deep Reinforcement Learning via Transfer and Multitask Learning

Physics-Informed Deep Reinforcement Learning for Power System Optimization and Control

Raia Hadsell - Deep Reinforcement Learning & Real World Challenges

MIT 6.S094: Deep Reinforcement Learning for Motion Planning

Google's DeepMind AI Just Taught Itself To Walk

MIT 6.S094: Deep Reinforcement Learning

Season 1 Ep 1 Foundations of Deep Reinforcement Learning with Pieter Abbeel

A friendly introduction to deep reinforcement learning, Q-networks and policy gradients

Tim Lillicrap - Data efficient deep reinforcement learning for continuous control

MIT 6.S191: Reinforcement Learning

773: Deep Reinforcement Learning for Maximizing Profits — with Prof. Barrett Thomas

Reinforcement Learning: AlphaGo

AI Learns to Use Stairs (deep reinforcement learning)

Deep Reinforcement Learning in the Real World -Sergey Levine

DeepMind x UCL RL Lecture Series - Introduction to Reinforcement Learning [1/13]

Reinforcement Learning: Crash Course AI #9

Deep Reinforcement Learning

SDS 551: Deep Reinforcement Learning — with Wah Loon Keng

TensorFlow and deep reinforcement learning, without a PhD (Google I/O '18)