• 234 Posts
  • 320 Comments
Joined 3 years ago
Cake day: June 30th, 2023
  • brianpeiris@lemmy.caOPtoTechnology@lemmy.world•ARC-AGI-3 news: In the past two weeks multiple groups have scored at least 99.9% on the public challenges at relatively low costs.English
    2·
    13 hours ago

    ARC Prize maintains multiple tracks around their benchmarks. They have “verified” leaderboards, “community” leaderboards, and they also run the ARC Prize competition.

    They update the “verified” leaderboards when they test raw LLMs without sophisticated harnesses. They seem to update this sporadically and only occasionally do press releases or blog posts about new scores. For example, the latest score from Claude Opus 5 (High) is 30% at $20,000, and they didn’t post about that as far as I know. Again, this just the raw LLM without an agentic or world-model harness.

    The ARC Prize competition has a harder set of criteria. Participants have to use smaller, open models with a limited compute budget, with open source code, and of course the solutions are verified by ARC Prize at the end of the competition.

    The “community” leaderboards, which is what this post is about, are self-reported and not verified by ARC Prize. There are no restrictions on what model is used or limitations on compute. So naturally they aren’t going to make official news releases about those, unless they decide to verify them at some point.

    The only reason I chose to post this is that the top solutions seem legitimate, with source code released, and two of them have associated papers.

  • brianpeiris@lemmy.caOPtoTechnology@lemmy.world•ARC-AGI-3 news: In the past two weeks multiple groups have scored at least 99.9% on the public challenges at relatively low costs.English
    2·
    13 hours ago

    It think it’s still unwise to talk about these topics in broad terms like AGI and even “intelligence”. We still have to pick the capabilities apart to have useful discussions about them. I agree these games are better tests than many benchmarks, but it’s also important to note that these solutions use a combination of well-designed deterministic harnesses, as well as LLMs. So it’s inaccurate to say that “LLMs have achieved AGI” (not sure if that’s what you were getting at). This feels like an important milestone, but we’ll have to continue to probe for failure cases in other categories of problems.

    Aside from emotional intelligence, experience, embodiment, etc., these ARC-AGI-3 solutions all rely on the sandbox being a safe environment to fail. The solutions iterate through the problem thousands of times before coming to a final solution. Many real-world human problems cannot be re-tried safely or efficiently.

  • brianpeiris@lemmy.catoProgrammer Humor@programming.dev•Zig language's policy on AIEnglish
    52·
    28 days ago

    I kinda agree with this, except the machine learning field should bear some responsibility for begetting LLMs. In particular, they got very used to the idea of scraping the internet for huge amounts of data needed for all types of models, and paid less and less attention to how much energy their training and inference was costing versus the value the models were providing. The seeds of the problems with LLMs existed before they landed on the scene.