One thing I notice about the weave-agent is it confabulates a lot, is weirdly ingenious when it comes to solving problems but has basic awareness fails and prematurely marks tasks complete, undermining debugging. I suspect playing around generates the data to solve this. minihf.com/posts/2024-0...

Part of the solution is basic redesign. I'm thinking before the agent can mark a task complete it needs to pass a test suite, and instead of making local evaluations the evaluation stage should contribute to a namespace of tests for that part of the task. This inhibits confabulated completions.