It probably makes a huge difference because the models that are RLed on Codemode were trained on JavaScript. So you are already off the beaten path. But for the best in class models, it won’t be very noticable.
maybe i misunderstood what you meant by optimization but i think because of python being the sandbox lang, models have to do almost no escaping for nested json and there are no rejected tool calls. they seem to default to using """-style strings for everything.
What i mean is the following. If you trigger a tool named “execute” via OpenAI that has an argument called “code” that contains some JavaScript code, the tool call gets encoded as such:
So you already are going through one layer of JSON encoding. To avoid that issue, on OpenAI you can create a freeform tool (basically you define our own grammar, worst case a regex that is /.+/) and then it encodes as such:
(Note that in practice on the closed weights models it probably has a <|constrain|>regex:.+ or whatever).
On Anthropic you don’t need to do anything, because it uses XML tool calls (antml) toplevel so the first level of code never goes through JSON to begin with. But the more complex wrapping is going on, the harder it is for smaller LLMs to follow.
Claude Code had a lot of hacks in the harness to deal with the effects of bad JSON encoding that was happening when the harness issued writes where it had to through JSON.
So the first level of problem is the tool call layer itself.
It probably makes a huge difference because the models that are RLed on Codemode were trained on JavaScript. So you are already off the beaten path. But for the best in class models, it won’t be very noticable.
score 1 · canonical
maybe i misunderstood what you meant by optimization but i think because of python being the sandbox lang, models have to do almost no escaping for nested json and there are no rejected tool calls. they seem to default to using
"""-style strings for everything.score 1 · canonical
What i mean is the following. If you trigger a tool named “execute” via OpenAI that has an argument called “code” that contains some JavaScript code, the tool call gets encoded as such:
<|start|>assistant<|channel|>commentary to=functions.execute <|constrain|>json<|message|>{"code":"python code here"}<|call|>So you already are going through one layer of JSON encoding. To avoid that issue, on OpenAI you can create a freeform tool (basically you define our own grammar, worst case a regex that is
/.+/) and then it encodes as such:<|start|>assistant<|channel|>commentary to=functions.execute <|message|>python code here<|call|>(Note that in practice on the closed weights models it probably has a
<|constrain|>regex:.+or whatever).On Anthropic you don’t need to do anything, because it uses XML tool calls (antml) toplevel so the first level of code never goes through JSON to begin with. But the more complex wrapping is going on, the harder it is for smaller LLMs to follow.
Claude Code had a lot of hacks in the harness to deal with the effects of bad JSON encoding that was happening when the harness issued writes where it had to through JSON.
So the first level of problem is the tool call layer itself.
score 1 · canonical