I suspect there is enough training data, especially for actually useful applications like agentic coding. But doing the actual training from scratch is a challenge. And I think this is where more research could be done on how to partition the training so it can be done in a distributed way. If you could have a system like SETI@home for example where people can donate a few cycles towards the training, you could likely do it fairly effectively with the cost being amortized across the network.
I think acceptability of different kinds of training data sources differs between people. For some ethical frameworks around attribution rights all the cross-distillation going on poisons all the sources, for example.
SETI@home model definitely needs some design tricks to handle higher latency during training, but also the latency might matter when running experimetns.
I wouldn’t even be surprised if someone manages to crowdfund a barely-sufficient setup under a nonprofit (probably with recent-contributors-vote as the final decision instance), and maybe use some donated distributed compute for final (largest) runs.
I suspect there is enough training data, especially for actually useful applications like agentic coding. But doing the actual training from scratch is a challenge. And I think this is where more research could be done on how to partition the training so it can be done in a distributed way. If you could have a system like SETI@home for example where people can donate a few cycles towards the training, you could likely do it fairly effectively with the cost being amortized across the network.
score 1 · canonical
I think acceptability of different kinds of training data sources differs between people. For some ethical frameworks around attribution rights all the cross-distillation going on poisons all the sources, for example.
SETI@home model definitely needs some design tricks to handle higher latency during training, but also the latency might matter when running experimetns.
I wouldn’t even be surprised if someone manages to crowdfund a barely-sufficient setup under a nonprofit (probably with recent-contributors-vote as the final decision instance), and maybe use some donated distributed compute for final (largest) runs.
score 1 · canonical
Yeah for sure, it’s the kind of stuff I really hope people will start exploring going forward.
score 1 · canonical