Excellent, look forward to giving this a go.

I was looking at: https://arxiv.org/abs/2506.18254 but your approach is even more general.

Might end up being some confusion with the RULER benchmark from NVIDIA given the (somewhat shared) domain: https://github.com/NVIDIA/RULER

EDIT: by shared I only mean the adjacency to LLMs/AI/ML, RL is a pretty big differentiator though and project looks great

Very cool. Do you do anything to mitigate ordering bias in the evaluation function, or do you just expect it to average out over time?

how does o3 on the customer support agent task so dreadfully underperform qwen?

Love these write ups!

Dope