How can a project realistically tell if a contributor used an open-weight model and self hosted it without it being a de facto ban on AI for anyone not heavily involved in the project?
Open-weight models are still giving up control. For one, all LLM outputs are generally recognized to not be copy writable which undermines copy-left license. Additionally, unlike open source software it’s a lot harder to look at what represented in these models. A model can easily be designed to make specific plausible mistake as intentional backdoors, act worse on tasks relevant to an industry a country / maker cares about (like Fable has been designed), etc.
Very good points, the last part is especially interesting and probably something that will play out more over the years. Models with built in failsafes to sabotage work based on certain criteria, I imagine the usa/China have already started working on weaponizing this type of behavior.
So why help by giving training data and give a unnerving lever of control of open source projects to a handful of companies?
Use open weight Chinese models that don’t collect your data.
How can a project realistically tell if a contributor used an open-weight model and self hosted it without it being a de facto ban on AI for anyone not heavily involved in the project?
Open-weight models are still giving up control. For one, all LLM outputs are generally recognized to not be copy writable which undermines copy-left license. Additionally, unlike open source software it’s a lot harder to look at what represented in these models. A model can easily be designed to make specific plausible mistake as intentional backdoors, act worse on tasks relevant to an industry a country / maker cares about (like Fable has been designed), etc.
Very good points, the last part is especially interesting and probably something that will play out more over the years. Models with built in failsafes to sabotage work based on certain criteria, I imagine the usa/China have already started working on weaponizing this type of behavior.