Torque Control of an Elbow Joint Using a Markov Decision Process and Deep Q-Network: A Simulation Study
Abstract
Upper-limb rehabilitation focuses on restoring strength, mobility, and functional movement after injury, illness, or surgery, with robotic assistive systems providing controlled motion support during recovery. However, accurate torque control of the elbow joint remains challenging due to nonlinear dynamics and uncertainties in human-robot interaction. This study presents a simulation-based torque control approach for an elbow joint using a Markov Decision Process (MDP) and Deep Q-Network (DQN). The elbow joint is modeled as a nonlinear single-degree-of-freedom system, while the DQN agent learns an optimal torque policy for tracking a desired joint trajectory. Results show that the DQN-based controller is capable of achieving stable and effective elbow joint torque control.
Keywords: Markov Decision Process, Deep Q-Network, Reinforcement Learning, Elbow Joint, Torque Control, Rehabilitation Robotics, Numerical Simulation.
1. Introduction
The elbow joint plays an important role in upper-limb movement, enabling daily activities such as reaching, lifting, and object manipulation. Impairments in elbow function are commonly associated with neurological disorders such as stroke and spinal cord injury, often resulting in reduced motor control and limited mobility that require rehabilitation therapy.
In recent years, rehabilitation robotics has gained attention as an effective approach for assisting upper-limb recovery through controlled and repetitive therapeutic motion. Robotic exoskeletons developed for elbow rehabilitation can provide adjustable support and improve motor recovery outcomes. However, achieving accurate and stable torque control remains a major challenge due to the nonlinear dynamics of the human arm and uncertainties associated with human-robot interaction.
Conventional control techniques such as proportional-integral-derivative (PID) and model-based computed torque control have been widely applied in robotic systems. Although these methods can provide satisfactory performance under ideal conditions, they often depend on accurate mathematical models and may experience performance degradation in the presence of nonlinear dynamics, external disturbances, and patient-specific variations.
These limitations have motivated increasing interest in data-driven and learning-based control strategies. One promising solution is reinforcement learning (RL), in which an agent improves its control decisions through repeated interaction with the environment. The learning problem is commonly represented as a Markov Decision Process (MDP), where actions are selected based on the current system state in order to maximize long-term reward. Recent developments in deep reinforcement learning, especially Deep Q-Networks (DQN), have shown good performance in robotic control problems by learning effective control policies directly from system interaction.
2. Elbow Joint Dynamic Model
The elbow joint is commonly modeled as a single-degree-of-freedom (1-DOF) rotational system representing flexion and extension motion in the sagittal plane. This simplification is widely adopted in rehabilitation robotics and manipulator modeling because it captures the essential dynamics of upper-limb motion while maintaining analytical simplicity.
The nonlinear equation of motion describing the elbow dynamics is given by:
where theta is the elbow joint angle, theta_dot and theta_ddot denote the angular velocity and angular acceleration respectively, I is the moment of inertia of the forearm-hand segment, b is the viscous damping coefficient, m represents the mass of the forearm-hand segment, g is the gravitational acceleration, l is the distance from the elbow joint to the center of mass, and tau is the applied control torque.
The nonlinear gravitational component mgl*sin(theta) arises from the Euler-Lagrange formulation commonly used in robotic manipulator dynamics. This term introduces state-dependent nonlinearity into the system, making the model more representative of real biomechanical motion and human-robot interaction conditions.
State-Space Representation
For reinforcement learning implementation, the nonlinear dynamic model is expressed in state-space form by defining the state variables as:
The resulting state-space representation becomes:
x2_dot = (tau - b*x2 - mgl*sin(x1)) / I
3. Markov Decision Process Formulation
The elbow joint torque control problem is represented using a Markov Decision Process (MDP), a framework commonly used for sequential decision-making problems involving uncertainty. An MDP is defined by the tuple (S, A, P, R, gamma), where S represents the state space, A is the action space, P denotes the state transition probability, R is the reward function, and gamma is the discount factor.
3.1 State Space
The state of the elbow joint system is defined as:
This state representation captures both the physical dynamics of the joint and the control objective of trajectory tracking.
3.2 Action Space
The action corresponds to the torque applied at the elbow joint. For DQN implementation, the continuous torque input is discretized into finite action levels:
3.3 Reward Function
The reward function is designed to encourage accurate trajectory tracking while penalizing excessive torque usage:
where the first term penalizes tracking error, the second term penalizes excessive control effort, and alpha and beta are weighting factors that balance accuracy and energy efficiency.
4. Deep Reinforcement Learning Method
The torque control problem formulated as an MDP is solved using a Deep Q-Network (DQN) approach. In reinforcement learning, the controller improves its decision-making ability through repeated interaction with the environment instead of depending entirely on an exact mathematical description of the system dynamics.
4.1 Q-Function Definition
The action value function, or Q-function, is defined as:
The objective of the DQN agent is to approximate the optimal action-value function Q*(s, a), which satisfies the Bellman optimality condition.
4.2 Deep Q-Network Architecture
In the DQN framework, a neural network is used to approximate the Q-function: Q(s, a; theta). The network takes the system state as input and outputs estimated Q-values for all possible torque actions.
Network input:
- Elbow joint angle, theta
- Angular velocity, theta_dot
- Tracking error, e
The output layer represents discrete torque actions applied to the elbow joint. Fully connected hidden layers with ReLU activation functions are used to learn the relationship between system states and control actions.
4.3 DQN Learning Update
The DQN parameters are updated using the Bellman equation:
L(theta) = E[(y - Q(s, a; theta))^2]
4.4 Exploration Strategy
An epsilon-greedy policy is employed to balance exploration and exploitation. With probability epsilon, the agent selects a random torque action to explore the environment, while with probability 1-epsilon, the action with the highest predicted Q-value is selected. The exploration rate is gradually reduced during training to improve convergence toward an optimal policy.
4.5 Experience Replay and Target Network
To improve training stability, experience replay is employed by storing previous interactions (s, a, R, s') in a replay memory buffer. During training, randomly selected mini-batches from the replay memory are used to reduce dependence between consecutive samples.
A separate target network is also used to stabilize Q-value updates. The target network parameters are periodically updated using the parameters of the main network, helping to reduce oscillations during training.
5. Simulation Setup
The study is conducted entirely in a numerical simulation environment developed in Python using PyTorch, allowing safe, repeatable, and controlled evaluation of reinforcement learning performance without the need for physical hardware implementation.
5.1 System Parameters
| Parameter | Symbol | Value |
|---|---|---|
| Moment of inertia | I | 1.0 kg*m^2 |
| Damping coefficient | b | 2.0 N*m*s/rad |
| Forearm-hand mass | m | 1.0 kg |
| Distance to COM | l | 0.20 m |
| Gravitational acceleration | g | 9.81 m/s^2 |
5.2 DQN Configuration
| Parameter | Value |
|---|---|
| Algorithm | Deep Q-Network (DQN) |
| Training episodes | 300 |
| Steps per episode | 300 |
| Learning rate | 0.001 |
| Discount factor (gamma) | 0.99 |
| Batch size | 64 |
| Replay buffer capacity | 10,000 |
| Exploration strategy | epsilon-greedy |
| epsilon decay rate | 0.995 |
| Minimum epsilon | 0.05 |
5.3 Evaluation Metrics
Performance is evaluated using:
- Mean Squared Tracking Error (MSTE): MSTE = (1/N) * sum((theta - theta_desired)^2)
- Cumulative control effort: sum(|tau|)
- Visual analysis: learning curves, trajectory tracking plots, torque signal plots, tracking error plots
6. Results and Discussion
6.1 Learning Performance
The learning performance of the proposed controller is evaluated using the average cumulative reward obtained during training.
The learning curve shows the evolution of the average reward as training progresses. At the early stage of training, the reward values fluctuate significantly due to the exploration behavior of the epsilon-greedy policy. As the number of episodes increases, the average reward gradually improves and becomes more stable, indicating that the agent successfully learns an effective torque control policy for the elbow joint system.
6.2 Elbow Joint Trajectory Tracking
The trained DQN controller is tested on the nonlinear elbow joint model using a desired joint trajectory of theta_desired = pi/4 rad.
The trajectory tracking result demonstrates that the DQN-based controller is able to drive the elbow joint toward the desired angle with stable behavior. The response exhibits smooth convergence with minimal oscillatory behavior, indicating that the learned torque policy can effectively compensate for the nonlinear dynamics of the elbow system.
6.3 Torque Control Signal
The torque profile shows that the controller applies varying torque levels during the motion. Larger torque values are observed during the initial stages when the tracking error is relatively high, while smaller corrective torques are applied as the joint approaches the desired trajectory. The discrete nature of the torque signal results from the discretized action space used in the MDP formulation.
6.4 Tracking Error Analysis
The tracking error decreases progressively as the simulation evolves, indicating that the controller gradually improves the alignment between the actual and desired elbow motion. Although small residual fluctuations are present due to the discrete action space and nonlinear system dynamics, the overall tracking error remains bounded and stable throughout the simulation period.
The Mean Squared Tracking Error (MSTE) obtained during simulation was 0.0021 rad^2, indicating accurate trajectory tracking performance.
6.5 DQN Training Loss
The loss curve illustrates the convergence behavior of the neural network during training. At the beginning of training, larger fluctuations are observed due to random exploration and unstable Q-value estimation. As training progresses, the loss gradually decreases and becomes more stable, indicating improved approximation of the optimal action-value function.
6.6 Discussion
The simulation results demonstrate that the proposed MDP-DQN framework can successfully learn torque control policies for nonlinear elbow joint motion. One major advantage of the proposed method is that the controller learns suitable torque actions directly from interaction with the environment, reducing dependence on precise inverse dynamic modeling of the elbow system.
This makes the approach potentially suitable for rehabilitation robotics applications where system dynamics may vary across patients or operating conditions. The results also highlight some limitations: since the torque action space is discretized, the generated torque signal exhibits discrete switching behavior, which may introduce small oscillations in the tracking response. In addition, the study is limited to a simplified single-degree-of-freedom elbow model implemented entirely in simulation.
7. Conclusion
This study presented a simulation-based torque control approach for an elbow joint using a Markov Decision Process (MDP) framework and a Deep Q-Network (DQN). The elbow joint was modeled as a nonlinear single-degree-of-freedom system, and a reinforcement learning agent was trained to learn an optimal torque policy for trajectory tracking. Results from numerical simulations showed that the proposed method can achieve stable learning behavior, effective trajectory tracking, and bounded control effort. The study demonstrates the potential of reinforcement learning-based controllers for nonlinear rehabilitation systems.
Future work will focus on extending the approach to continuous action spaces and validating the method on more complex upper-limb models.
References
- R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed. MIT Press, 2018.
- V. Mnih et al., "Human-level control through deep reinforcement learning," Nature, vol. 518, pp. 529-533, 2015.
- S. Lillicrap et al., "Continuous control with deep reinforcement learning," ICLR, 2016.
- J. Schulman et al., "Proximal policy optimization algorithms," arXiv:1707.06347, 2017.
- D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," ICLR, 2015.
- O. Deisenroth, G. Neumann, and J. Peters, "A survey on policy search for robotics," Foundations and Trends in Robotics, vol. 2, pp. 1-142, 2013.
- M. W. Spong, S. Hutchinson, and M. Vidyasagar, Robot Modeling and Control, 1st ed. Wiley, 2006.
- R. Siciliano et al., Robotics: Modelling, Planning and Control. Springer, 2010.
- D. E. Winter, Biomechanics and Motor Control of Human Movement, 4th ed. Wiley, 2009.
- A. Maciejasz et al., "A survey on robotic devices for upper limb rehabilitation," J. NeuroEngineering and Rehabilitation, vol. 11, p. 3, 2014.