On Awareness
Awareness is important.
Being aware of a specific quality is a necessary prerequisite to change. For example, once I started truly tracking how much time I spent on chess/videogames (and when I finally started becoming aware of the fact that I wasn't having fun playing videogames) on a weekly basis - i.e. once I became aware of my wasteful time spending, I made efforts to change that. Similarly, after reading Taubes' book on sugar and being made aware of the damaging effects of sugar1, I've since taken steps in my life to inhale less sugar.
The ability to be self-aware (e.g. the ability to recognize that the dual in the mirror is a reflection) is classically thought of as a benchmark for intelligence. Moreover, I think at some core level, all definitions of "true intelligence" (i.e. AGI) must invoke some notion that an agent that is intelligent should be able to understand some concept [x], the existence of a meta-[x] and therefore the meta-[x] itself, and so on.
As an example of what I mean, consider the game of chess. At some level, if you were to play a robot, the only thing that would matter is the game situation -- the pieces on the board, the time on both agents' clocks, the move number, etc. In reality however, as anyone who plays chess knows, opening choices are not random and are usually premeditated; for example, I may start off with 1. Nf3 over 1. e4 if I've played more games that started out with 1. Nf3 as I have more confidence in my game there. Thus, while the "true" evaluation of the position may be equal (i.e. if you asked an oracle, they would say that under optimal play White can never force a win) no matter which opening move I pick, by recognizing that I'm better in the lines that go 1. Nf3 rather than 1. e4, I'm already recognizing and playing chess on some meta-level as my thinking goes beyond the pieces; rather than "yes both positions are objectively equal and thus there is no difference in which position you play", I recognize that while both positions are objectively equal, since I have familiarity in one position and not the other, the more familiar position will be easier for me to play (which helps me with my ultimate goal of winning a game). Thus, by recognizing that not all opening moves are equal, one inadvertently recognizes the existence of a meta-level of chess.23
As such, I think awareness, the ability to be aware, and the ability to direct your attention onto things that matter to you are key hallmarks of what constitutes intelligence. The recognition of a meta-level (or a meta / meta-game in video games) should follow naturally from (in some sense) 'true' awareness as the whole idea of the meta is built upon being aware. For example, why are some characters in Super Smash Bros 'better' than others? Simply because people started becoming aware that these characters win more than average and if the goal is to win, these characters might be worth selecting over others.
Besides the book's thesis which is that sugar bad because sugar is correlated to obesity while almost nothing else is (which suggests that something sugar or sugar-adjacent causes obesity), the psychological effects of sugar also don't seem pretty good (eating sugary snacks/treats ~ more prone to instant gratification is what I've noticed in my life).↩
Of course, with the meta there can be the meta meta and then the meta meta meta, etc. As an example, when I play a tournament and I observe my future opponent play some opening line, I may run that line through an engine and find the best response to it so that when I do end up playing my opponent, while they may choose their preferred opening for familiarity (playing chess on the meta-level), I may follow their opening and their line as I recognize that in their familiar line, I hold the advantage as I've seen their weaknesses (meta meta chess as I recognize that they want to play the lines they're comfortable with so I prepare to play a line they're comfortable with).↩
The meta level of chess can also be much more broad -- as an example, the Sicilian (1. e4 c5) is historically connoted with dynamic chess games while the Petrov (1. e4 e5 2. Nf3 Nf6) has gotten a reputation of being a drawing line. As a result, when deciding what opening to play, more than deciding which opening will lead to the biggest positional advantage after the opening, one will also base their openings on whether the opening is quiet/sharp, positional/tactical, mainline/sideline, etc. Inadvertently thus, the player that recognizes these dynamics (and deviations) may gain information (meta-information?) that might lead them to victory. As an example, if someone who plays the Sicilian (sharp, tactical, well-understood opening) suddenly plays the Petrov against me, this signals to me that I should be either wary or on the offensive as their opening choice suggests that either they aren't looking for a fight (in which case if I fight I may be more likely to win as they're not as alert) or they have some hidden preparation in this line and perhaps I'm better off playing safe rather than aggressively lest I fall into their engine chess preparation.↩