Risk penalties are common in reinforcement-learning trading, but they are rarely the thing an experiment is about. My bachelor thesis in Computer Science at the University of Amsterdam makes them the main variable: what happens to two trading agents when each is told, more and more strongly, to avoid its own risk?
The full paper is at the end of this page. The two posts I wrote along the way are Making Trading Agents Care About Risk and A Controlled Study of Risk Aversion.
The setting
Two agents share one limit order book in JaxMARL-HFT, a GPU simulator that replays historical order messages:
- A market maker posts bid and ask quotes. Its risk is inventory: after uneven fills it holds a position it did not choose.
- An execution agent has to buy or sell 600 shares in 64 decision steps. Its risk is the quantity still left to trade.
Each agent has its own recurrent policy and its own reward, and they are trained together with independent PPO in 4,096 parallel environments. The data is Amazon (AMZN) LOBSTER order-book data, split by date into 74 training days, 25 validation days and 24 held-out test days.
The penalty
Only one thing changes between training runs. The market maker’s reward loses a multiple of its squared inventory, and the execution agent’s reward loses a multiple of its squared remaining quantity:
The same quadratic form sits behind two classical models: Avellaneda and Stoikov’s inventory term for market making, and Almgren and Chriss’s variance term for execution. To make the coefficients comparable, each one is scaled by a reference value estimated on validation episodes. I then vary the two penalties separately and together in a grid, and check each learned policy against a fixed counterparty.
What the runs show
Execution: lower exposure through urgency
The execution agent becomes more urgent. At the strongest penalty, mean squared remaining quantity is 59% lower (88,505 to 36,593 shares²), completion rises from 95.3% to 99.9%, and the order finishes about 20 decision steps earlier. Passive placements fall from 51% to 26%.


Market making: lower inventory can hide withdrawal
The market maker first controls risk, then stops trading. A moderate penalty (a quarter of the reference value) lowers mean squared inventory by 48% while the agent keeps quoting about as often. From half the reference value upward, the no-trade rate is above 95%. The inventory is low because the agent has left the market, not because it manages its position better.

The joint sweep
Each penalty mainly moves its own agent. In the joint grid, the market maker’s inventory follows its own coefficient and the execution agent’s exposure follows its own.

The lesson I take from it: the same penalty leads to different behaviour because the two tasks offer different ways to avoid risk. An execution agent cannot avoid its risk by doing nothing, because unfinished quantity stays costly. A market maker can. A risk measure therefore has to be read next to a check that the task is still being done.
Limits
Each coefficient was trained once, so the results describe these runs and do not locate a point that would hold across seeds. The study uses one stock and one period, the replayed order flow does not react to the agents, and the 4,096 test windows come from only 24 days. The next step is to repeat the key coefficients with several seeds and report day-level uncertainty.