I struggle to comprehend how I could conceivably gain value by having up to 60 different color coded elements on my screen

You don’t. That’s how how this system works at all. You can run Pi yourself and you will discover that Pi is very conservative when it comes to colors.

score 1 · canonical

For many months, Pi annoyingly insisted on a light/dark theme duo

I’m confused you say that because Pi did not. You can set a basic theme.

score 1 · canonical

How would you address it?

score 1 · canonical

Disclaimer here: this is from our company blog and it abstractly relates to a coding agent but it really doesn’t have anything to do with AI.

Interestingly what I learned from this is that there is no good contrast algorithm available today so we are trying to get people to do the survey so that we can get some good real world data points to get an open source algorithm for the contrast that is freely usable by anyone.

Survey is here if you are willing to take it: https://contrastsurvey.earendil.com/

score 5 · canonical

What i mean is the following. If you trigger a tool named “execute” via OpenAI that has an argument called “code” that contains some JavaScript code, the tool call gets encoded as such:

<|start|>assistant<|channel|>commentary to=functions.execute
<|constrain|>json<|message|>{"code":"python code here"}<|call|>

So you already are going through one layer of JSON encoding. To avoid that issue, on OpenAI you can create a freeform tool (basically you define our own grammar, worst case a regex that is /.+/) and then it encodes as such:

<|start|>assistant<|channel|>commentary to=functions.execute
<|message|>python code here<|call|>

(Note that in practice on the closed weights models it probably has a <|constrain|>regex:.+ or whatever).

On Anthropic you don’t need to do anything, because it uses XML tool calls (antml) toplevel so the first level of code never goes through JSON to begin with. But the more complex wrapping is going on, the harder it is for smaller LLMs to follow.

Claude Code had a lot of hacks in the harness to deal with the effects of bad JSON encoding that was happening when the harness issued writes where it had to through JSON.

So the first level of problem is the tool call layer itself.

score 1 · canonical

They are related but Claude currently exposes not a lot of capabilities in the dynamic workflow system. My expectation is though that it will be upgraded to a general codemode tool eventually.

score 1 · canonical

How are you doing benchmarking in Pi?

Badly.

How much effort is spent to co-evolve Pi with different models?

More than you think. But at lot of this right now is vibes based and looking at user reports. A big issue with all of this is that almost anything works with SOTA models (except when it overfits) but with the Chinese open weight models you have all kinds of crazy regressions.

There are some harnesses that go really far patching around some of the failure conditions. For instance smaller Qwen models go into loops like crazy and some harnesses spend a lot of effort to detect loops and break out of them. We don’t do any of this because our expectation is that models fix this and we don’t want to carry nonsense like this in the core.

would Chinese LLMs benefit from this as well, or are they very similar from a harness perspective?

Not all Chinese models are the same here but outside of benchmarks there is a huge difference between US SOTA models and Chinese open weight models still. But we do not try to benchmaxx or fine tune for a particular model. I know that some are, but it’s very hard to evaluate that this actually works well for all situations.

score 1 · canonical

It probably makes a huge difference because the models that are RLed on Codemode were trained on JavaScript. So you are already off the beaten path. But for the best in class models, it won’t be very noticable.

score 1 · canonical

As somebody who does not use Pi I am really curious what led to this decision.

Combination of conservatism and basic benchmarking across different LLMs. Codemode works very poorly for some LLMs due to how tool calls are encoded. Anthropic special cases toplevel string arguments in antml to be mostly escaping free, and OpenAI enables something similar with freeform tools. So codemode itself can be escaping free on those models. But then the tools within it, lose that optimization because now you’re doing JSON escaping.

Some smaller models really suffer from this but they still reach for codemode too much after a while. This will need more investigation before we subject people to that as a default experience.

Codemode also is quite beefy in the system prompt itself, for the agent to make sense of it.

score 1 · canonical

I think this is a great change. At least it recognizes there are more languages than just English :)

score 1 · canonical