Paper Notes: Survey of Model-Based Reinforcement Learning
I've been studying a survey paper these days and took some notes. Title: Survey of Model-Based Reinforcement Learning: Applications on Robotics
[Incomplete, in progress.....]
Need: Adaptive robot: Adaptable Robot
Challenge: Reinforcement learning requires interactive training between the robot and the environment, a process that may damage the robot and the environment, especially in human-robot collaboration scenarios.
Belief: Although there are many difficulties right now, RL will make robots more intelligent.
RL / Model-based / Model-free
The difference between RL and other deep learning methods is that RL involves interaction between the agent and the environment, and is mainly used to solve decision-making problems.
A robot's state can be represented as continuous or discrete values. The robot controller is in state s and generates action a (motor action) through policy π. In a deterministic policy, a is unique. In the stochastic case, depending on a random variable ε, a probability distribution π(a|s, ε) over a is produced.
The goal of RL algorithms is to find the maximum expected value of the reward function r.
Most RL research is model-free, but recently model-based RL approaches are increasing.
Model-based RL has certain advantages over model-free when dealing with real-world use cases.
The interaction between the robot and the environment is an MDP Markov decision process. The difference between model-based and model-free lies in the model of robot-environment interaction. Model-free has no environment or model, and is entirely based on trial and error.
This paper classifies model-based methods. The criteria are as follows:
Value function definition Transition policy optimization methods Transition model and learning methods Reward function: reward function, gives the policy
Return function: return function, gives the expected return
Two Major Types of RL Approaches
Value Function V or Value-action function Q Bellman equation:
- DP: Dynamic Programming: iterative algorithm. Policy iteration first evaluates the current policy, computing the state or value function; then it improves the policy. transition dynamics
- MC: Monte Carlo method: based on sampling
- TDL: Temporal Difference Learning: accounts for the value difference between states
- DDP: Differential Dynamic Programming Policy search methods: more suitable for solving high-dimensional, multi-DOF robot problems
- Gradient based methods
- Expectation Maximization methods
- Information Theory: Exploit concepts such as entropy
- Bayesian optimization methods
- Evolutionary computation Transition Models
- Stochastic
- Deterministic Return Functions
- Discounted
- Averaged
Policy search methods
Compared with value function methods, which determine the policy by maximizing the Value, policy search methods parameterize the policy and search for the optimal policy.
5 main categories:
Gradient descent methods Adjust the weights of a neural network. Usually trained first in simulation, then transferred to a real robot. Requires setting the step size and may suffer from slow convergence. The Probabilistic Inference for Learning Control (PILCO) framework is currently the best method — it needs few training samples and is fast. Its transition dynamics model handles uncertain inputs well, performs well in trajectory prediction, and has high error tolerance. Sampling-based methods Bayesian methods Evolutionary methods Information Theory
Transition models
Applications in Robotics
UAV Hovering Maneuvering UGV Docking Navigation UUV Tracking Swimming Bipedal Kicking Walking Balancing Robotic Manipulators Surgical Motion control Ball-hitting
Key focus: Robotic Manipulator applications
The state space is continuous, containing the position and velocity or acceleration of each joint.
The action space is also continuous, 3-dimensional, containing torque.
This field includes low-DOF robots as well as high-DOF robots such as PR2, and obstacle avoidance tasks.
Conclusion
The above discussed the successful experiences of model-based RL. But in fact these applications are simplified tasks or game tasks, such as table tennis, badminton, pendulums, etc. Although there are some "success stories" in autonomous driving, there is still a lack of successful evidence in scenarios where reliability and robustness matter more, such as service and industrial robots. Not just reinforcement learning — the entire deep learning field lacks success cases. However, in human-robot collaboration scenarios where the environment or manipulated objects change frequently, reinforcement learning can still help.
Model-based RL requires fewer environment interactions and is therefore more suitable for robot learning than model-free.
Fewer environment interactions mean less wear and damage to the robot. So through this paper we analyze how to apply model-based approaches to service and industrial robots, such as the very common and highly challenging pick and place operation. Production line assembly falls into this category.
To learn an adaptive pick and place operation, the tool needs sensors to perceive the environment and the object being manipulated, and to identify obstacles. Obstacle avoidance is crucial. The reward can be defined as the distance between the current state and the target state. Trajectory smoothing is achieved through policy optimization methods. Loss is defined as unexpected states such as collisions.
Another goal is fast convergence of the policy learning process.
Value function iteration methods are not suitable for large continuous state and action spaces, because their V or Q functions must be computed continuously without a termination condition.
Sampling and TD learning methods also have limitations because they lack a prior model of the transition model.
DDP comes from value function methods but is feasible. DDP's constraints on the initial trajectory limit the system's autonomy.
Information Theory algorithms also depend on the initial trajectory. Therefore, policy search methods are more suitable for collaborative robot applications, because they can reduce the dimensionality of the policy learning problem by parameterizing the policy function. The most promising of these algorithms are gradient- and sampling-based methods.
The transition model is very important for the efficiency of learning algorithms.
The current robot market features low-cost, adaptable robots. This makes robots safer and easier to buy. But it also brings some problems:
Elastic collaborative joints are extremely difficult to physically model. Rethink's robots are an example. Cheap internal sensors bring a lot of noise. Some robots like UR have no torque sensors and use other methods instead.
