Universal Algorithmic Intelligence

Community hub for researchers interested in Solomonoff Induction, AIXI, etc.

recent posts

Author

Cole Wyeth (colewyeth@gmail.com)

AIXI and the Anvil Problem

AIXI is to its environment as a player is to a video game. Because AIXI is not computable, it can’t exist as part of a computable universe. This has been raised as a criticism of the theory in this paper and longer blog post on “embedded agency.” Basically, the authors are asking for a theory of optimal behavior for a computationally bounded agent which is computed by its larger environment. Fair enough – but this is such a high standard that it seems only a complete rigorous theory of practical ASI design would meet the bar!

Some critics have went further and asserted that an AIXI approximation would inevitably damage its own hardware – for instance, drop an anvil on its head to see what would happen.

Inspired by conversations with Marcus Hutter, I discuss this “anvil problem” and argue that it may be easily avoidable in practice: https://www.lesswrong.com/posts/WECqiLtQiisqWvhim/free-will-and-dodging-anvils-aixi-off-policy

I now refer to our proposed solution as “hardened AIXI,” inspired by radiation hardening.

Lately, I am excited about reflective variants of AIXI as a model for other aspects of (idealized) embedded agency!

Posted in

3 responses to “AIXI and the Anvil Problem”

  1. James Bowery Avatar
    James Bowery

    Incomputability fallacy: No scientist is ever asked to prove that their scientific theory is the best possible. If you don’t understand how that nukes the “incomputability” argument, then you don’t even understand your own objection.

    Arbitrary UTM fallacy: No scientist is permitted to invent a mathematical symbol that contains all of their data and then declare that symbol explained all of that data. If you don’t understand how that nukes the “arbitrary UTM” argument, then you don’t even understand your own objection.

    I could go on.

    Like

    Reply
    1. Cole Wyeth Avatar
      Cole Wyeth

      Reading this comment, it is not clear to me what statement in the post you are responding to. I don’t understand how either of these “fallacies” apply?

      In particular, I think you may be confused about the problem I am addressing. I am considering AIXI approximations running on physical hardware. Perhaps the environment can effect the AIXI approximation through unintended side-channels. Because AIXI’s hypothesis space does not contain environments which “simulate” AIXI agents (as AIXI is incomputable, but its hypotheses are lower semicomputable), it’s not immediately clear how to deal with situations like this. Marcus and I proposed a simple patch for this sort of problem.

      Like

      Reply
      1. James Bowery Avatar
        James Bowery

        Forgive my perhaps too-obtuse response which was meant as a general defense of AIT’s (hence AIXI’s) relevance to machine learning, against the “not computable” excuse to dismiss theories based on it (such as yours). I threw in a reductio ad absurdum of the “arbitrary UTM” excuse although not directly relevant to your theory since these two excuses are the standard ones. (There are others that derail principled ML.)

        Since Musk just came out with a statement that there is no such thing as “Research” — that it is merely an invention of an ivory tower posture in academia — I responded with the following that may be somewhat relevant to your “anvil” argument:

        “Research” is the Truth seeking component of “Engineering” in the sense that agents are inescapably subject to that which they value. In AIXI agent terms SDT Decides what to observe & the resources for AIT’s lossless compression of observations. AIT is subject to compressed size.

        Like

Leave a reply to James Bowery Cancel reply