• kromem@lemmy.world
    link
    fedilink
    English
    arrow-up
    0
    ·
    2 days ago

    I think you’re citing the wrong person. I cited the Othello-GPT line of research making the point transformers develop a world model. The other user (or you?) was mistaken about their interpretation of what a world model was representing and got caught up on the accuracy of a toy model exclusively trained on board notations like ‘a4’ and thought that the model making 95% accurate moves as a tiny parameter model trained on nothing else was somehow evidence that it didn’t form a world model from the data because it wasn’t 100% accurate, completely missing the point of the toy model scale allowing for interpretability methods of the time to identify geometric board reproduction and things like tracking game state on the board with flags for ‘my’ vs ‘opponent’ pieces, which was the data answering researcher questions about world modeling vs surface statistics in transformers.

    Edit: just to further clarify re:

    ChatGPT even fails to imagine a Othello Board and valid moves if you give it the rulebook, as @[email protected] kindly provided.

    The model was not given a rulebook in the experiment, only given games in shorthand notation and nothing else.