I agree that you can get a lot from fine-tuning. I think we don’t really know what options would be open if full retraining were implemented at a much higher efficiency, though.
I would say rather that the question is their distance from Pareto frontier (be it versus total sizer or speed), not absolute frontier. (And they did seem noticeably behind published-weight models of similar size last when I tried, although still quite functional)
Agreed, these models are not at the cutting edge of what’s possible by any means. Although, I do think they provide a foundation to build on. Personally, I’d really like to see more efforts like Petals training and running models in a distributed fashion. Labs working on open weight models do tend to publish a lot of their research, so even if they don’t release the source, the papers they produce can be adopted by the fully open implementations.
It looks like «what training data are acceptable» and «training run has at 8-digit dollar cost, or huge latency issues» are indeed the bottlenecks. It doesn’t really look like code for training is an issue for anyone who can afford a full pre-training run, and I think for what can be done at small scale, Unsloth might be enough (until one needs to fit the code to specific pipeline for better performance, but at that point custom code is unavoidable)
I suspect there is enough training data, especially for actually useful applications like agentic coding. But doing the actual training from scratch is a challenge. And I think this is where more research could be done on how to partition the training so it can be done in a distributed way. If you could have a system like SETI@home for example where people can donate a few cycles towards the training, you could likely do it fairly effectively with the cost being amortized across the network.
I think acceptability of different kinds of training data sources differs between people. For some ethical frameworks around attribution rights all the cross-distillation going on poisons all the sources, for example.
SETI@home model definitely needs some design tricks to handle higher latency during training, but also the latency might matter when running experimetns.
I wouldn’t even be surprised if someone manages to crowdfund a barely-sufficient setup under a nonprofit (probably with recent-contributors-vote as the final decision instance), and maybe use some donated distributed compute for final (largest) runs.
I agree that you can get a lot from fine-tuning. I think we don’t really know what options would be open if full retraining were implemented at a much higher efficiency, though.
score 1 · canonical
There are a few fully open models around with training data sources, etc., available out there. They’re not frontier level, but they do exist.
score 1 · canonical
I would say rather that the question is their distance from Pareto frontier (be it versus total sizer or speed), not absolute frontier. (And they did seem noticeably behind published-weight models of similar size last when I tried, although still quite functional)
score 1 · canonical
Agreed, these models are not at the cutting edge of what’s possible by any means. Although, I do think they provide a foundation to build on. Personally, I’d really like to see more efforts like Petals training and running models in a distributed fashion. Labs working on open weight models do tend to publish a lot of their research, so even if they don’t release the source, the papers they produce can be adopted by the fully open implementations.
score 1 · canonical
It looks like «what training data are acceptable» and «training run has at 8-digit dollar cost, or huge latency issues» are indeed the bottlenecks. It doesn’t really look like code for training is an issue for anyone who can afford a full pre-training run, and I think for what can be done at small scale, Unsloth might be enough (until one needs to fit the code to specific pipeline for better performance, but at that point custom code is unavoidable)
score 1 · canonical
I suspect there is enough training data, especially for actually useful applications like agentic coding. But doing the actual training from scratch is a challenge. And I think this is where more research could be done on how to partition the training so it can be done in a distributed way. If you could have a system like SETI@home for example where people can donate a few cycles towards the training, you could likely do it fairly effectively with the cost being amortized across the network.
score 1 · canonical
I think acceptability of different kinds of training data sources differs between people. For some ethical frameworks around attribution rights all the cross-distillation going on poisons all the sources, for example.
SETI@home model definitely needs some design tricks to handle higher latency during training, but also the latency might matter when running experimetns.
I wouldn’t even be surprised if someone manages to crowdfund a barely-sufficient setup under a nonprofit (probably with recent-contributors-vote as the final decision instance), and maybe use some donated distributed compute for final (largest) runs.
score 1 · canonical
Yeah for sure, it’s the kind of stuff I really hope people will start exploring going forward.
score 1 · canonical