CHIMERA is a deep reinforcement learning agent trained to play a trick-based card game using Proximal Policy Optimization (PPO) with Generalized Advantage Estimation (GAE) and self-play.
The agent uses a shared feature extractor followed by two heads: a policy network (action selection) and a value network (state evaluation).
Residual blocks improve stability:
\[ h_{l+1} = \mathrm{ReLU}\Big(\mathrm{LayerNorm}(x_l + f(x_l))\Big) \]
The agent is trained using Proximal Policy Optimization (PPO), a policy gradient method that prevents destructive updates by constraining how much the policy can change each iteration. The PPO objective is:
\[ L^{\text{CLIP}}(\theta) = \mathbb{E}_t \Big[ \min \big( r_t(\theta) A_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) A_t \big) \Big] \]
where:
Advantages are computed using Generalized Advantage Estimation:
\[ \delta_t = r_t + \gamma V(s_{t+1}) - V(s_t) \] \[ A_t = \sum_{l=0}^{\infty} (\gamma \lambda)^l \delta_{t+l} \]
This reduces variance while maintaining low bias in policy gradient updates.
The value network is trained using a mean-squared error loss:
\[ L_V = \mathbb{E}_t \Big[ (V(s_t) - R_t)^2 \Big] \]
where \( R_t = A_t + V(s_t) \) is the Monte Carlo return estimate.
Training is performed via self-play within a multi-agent environment. The same policy controls all agents, but each player receives separate observations and rewards. This induces a competitive learning signal without requiring external opponents.
For the betting net, rewards are assigned at the round level, meaning each batch of four bets followed by the 24-action game played by the main nets corresponds to one completed round. For the main net, rewards are assigned at the trick level, meaning each batch of four actions corresponds to one completed trick.
The training process alternates between collecting gameplay experience and updating the policy using PPO.
The following visualizations show the exported ONNX computation graphs.