2023-03-15-IMF-人工智能与宏观经济建模_RBC模型中的深度强化学习（英）_31页_1mb

Summary of IMF Working Paper on AI and Macroeconomic Modeling with Deep Reinforcement Learning

Introduction

• Purpose: The paper explores the integration of deep reinforcement learning (DRL) using the DDPG algorithm into a real business cycle (RBC) macroeconomic model to create an AI-based simulator.
• Scope: It aims to demonstrate that this approach can generate realistic macroeconomic dynamics comparable to rational expectations models while allowing for bounded rationality in agents.

Methodology Implementation

• Algorithm Choice: DDPG (Deep Deterministic Policy Gradient) is employed for its ability to handle continuous action spaces, high-dimensional state spaces, and non-stationary problems.
• Model Setup: An RBC model is specified with households and firms optimizing utility and profits, respectively. Parameters follow standard settings, and the RL agent learns through self-interaction in either deterministic or stochastic environments.
• Key Components: State variables include capital, productivity shock, and past actions, while actions involve labor, investment, and bond adjustments. Reward is based on logarithmic utility.

Experimental Findings

• Deterministic Environment: The agent starts with random actions and gradually converges to the steady-state optimal decisions. Learning shows non-monotonic convergence in mean squared distance metrics, indicating volatility but overall improvement after many episodes.
• Stochastic Environment: The agent learns faster and makes decisions closer to optimal due to richer state exploration. Volatility is higher, but convergence occurs earlier (around 125 episodes) compared to deterministic scenarios, likely from increased experience with shocks.
• Challenges: Learning can be unstable, leading to "learning traps" where suboptimal behaviors persist. Parameter tuning, such as exploration rates and neural network settings, significantly affects stability and convergence.

Limitations and Future Directions

• Current Constraints: The basic RBC model is limited to productivity shocks and simplified labor-leisure decisions.
• Enhancements: Future work should add more variables, sectors, or use alternative DRL algorithms (e.g., TD3 or SAC) for better stability and broader applications.

Conclusion

• Reinforcement learning can successfully replicate rational expectations behavior in RBC models, providing a flexible tool for macroeconomic analysis.
• However, challenges in learning stability require careful parameter calibration, and further research could expand this approach for real-world policy simulations.