Nonlinear Control: Hamilton Jacobi Bellman (HJB) and Dynamic Programming

This video discusses optimal nonlinear control using the Hamilton Jacobi Bellman (HJB) equation, and how to solve this using dynamic programming.

This is a lecture in a series on reinforcement learning, following the new Chapter 11 from the 2nd edition of our book "Data-Driven Science and Engineering: Machine Learning, Dynamical Systems, and Control" by Brunton and Kutz

This video was produced at the University of Washington

[higasa]

This is the first time I've ever seen the explanation of HJB-DP in a intuitive and fashionable way, not by following the text book lines one by one. Thank you so much for the great talk.

[charlescai]

Very good explanation to derivative of HJB equation. But there's a point I may have to add that I think there may be a typo in 'DERIVING HJB EQUATION': In dV/dt, minimizing the integral of L(x, u), the lower limits of integral should be t instead of 0. Only by the case, we can conclude in the second last equation that -L(x(t), u(t)) can be obtained from the time derivative of integral of function L(x, u)...

[alanzhus]

Can't believe serious topic as this can have thousands of views hours after release. Youtube is really a magic place.

[hfkssadfrew]

Hey Steve, on 9:11 it should be integration from t to t_f, then that’s where the - comes from.

[ailsani]

I am a follower from his 'control bootcamp' series. Just trying to tell everyone new here that his video is life-saving.

[blitzkringe]

Please do more of this content. Thank you.

[prantel]

At 11:47 the bounds of the integral should be from “t” to “tf”; not from 0 to tf. If you make that change then the derivative of the integral wrt to t will be -L(., .)

[ecologypig]

Excellent. Can see a lot of connections with Control and how the essence of Bellman equation are all over the place in different fields. Thanks Prof. Brunton!

[amaarquadri]

Wow it's so cool that these concepts from reinforcement learning apply so perfectly to nonlinear control.

[aiwithhamzanaeem]

Thanks Professor Steve, Finally I completed the playlist.

[leventguvenc]

Very nice video. In deriving the HJB equation, the lower limit of the integral should be t instead of 0.

[ramanujanbose]

Steve I follow all of your lectures. Being a mechanical engineer I really got amazed by watching your turbulence lectures. I personally worked with CFD using scientific python and visualization and computation using python and published a couple of research articles. I'm very eager to work under your guidance in the field of CFD and Fluid dynamics using Machine learning specifically simulation and modelling of turbulence fluid flow field and explore the mysterious world of turbulence. How should I reach you for further communication?

[rajanisingh]

@Eigensteve, Thanks for such a nice and interesting videos. I've seen all your videos on reinforcement learning. It would be really helpful if you could do a lecture on how dynamic games (either discrete or continuous time) can be solved using reinforcement learning with a walkthrough example. For now, the theoretical concepts on reinforcement learning are clear from your videos, but how it's actually implemented to solve problems is still unclear. Also if you can recommend some resource that would be bonus!

[djredrover]

it would be lovely if you could do a MATLAB demo of an ONC using HJB for a hovercraft/drone with full 6-DOF model.

[qiguosun]

Great Lecture, could you think about discusing HJB with variational inequality? thanks!

[tuptge]

More on non linear control please! Im trying to make up my mind on topics for my postgrad thesis!

[geonheelee]

A Great Lecture. I hope the next lecture will open asap. In particular, I'm interest in detailed relationship between RL and optimal control.

[julienriou]

that's weird not to talk about Pontryagin Maximum Principle in an introduction to optiaml control

[qejacwa]

This is a fantastic video on the derivation. However, there are quite some typos in the video. Hopefully, Steve can correct them. For example, the lower limit in the integral is supposed to be t instead of 0 in the derivation of HJB equation.

[junhyeongjunhyeong]

nice introduce to HJB. 12:25 why do we take an action at xn(the terminal state)?it is not intuitively clear to me. if cost function L is given, we can get action at xn. it is the action that minimize the cost function at xn. but it is obviously an unnecessary action when i think about it