Rationality  volume 3

Rationality: From AI to Zombies - Volume 3

The third volume of the Less Wrong book Rationality: From AI to Zombies. This book would not have been possible without the help of our Kickstarter backers.

It contains the following sequences:

  • Fake Preferences
  • Value Theory
  • Quantified Humanism
  • Yudkowsky’s Coming of Age
  • Challenging the Difficult
  • The Craft and the Community

Please send any feedback to support+the.sequences@castify.co.


Not For The Sake Of Happiness (Alone)

Play f114902fd0cfda50800be7fbb69752c4badad535050adaaeb6927f96d31eb7c8

  1. Not For The Sake Of Happiness (Alone)

    Followup to: Terminal Values and Instrumental Values When I met the futurist Greg Stock some years ago, he argued that the joy of scientific discovery would s

  2. Fake Selfishness

    Followup to: Fake Justification Once upon a time, I met someone who proclaimed himself to be purely selfish, and told me that I should be purely selfish as w

  3. Fake Morality

    Followup to: Fake Selfishness God, say the religious fundamentalists, is the source of all morality; there can be no morality without a Judge who rewards and

  4. Fake Utility Functions

    Continuation of: Fake Fake Utility Functions Every now and then, you run across someone who has discovered the One Great Moral Principle, of which all other v

  5. Detached Lever Fallacy

    Followup to: Humans in Funny Suits This fallacy gets its name from an ancient sci-fi TV show, which I never saw myself, but was reported to me by a reputable s

  6. Dreams Of Ai Design

    Followup to: Anthropomorphic Optimism, Three Fallacies of Teleology After spending a decade or two living inside a mind, you might think you knew a bit about

  7. The Design Space Of Minds In General

    Followup to: The Psychological Unity of Humankind People ask me, "What will Artificial Intelligences be like? What will they do? Tell us your amazing story

  8. Where Recursive Justification Hits Bottom

    Followup to: No Universally Compelling Arguments, Passing the Recursive Buck, Wrong Questions, A Priori Why do I believe that the Sun will rise tomorrow? Becau

  9. My Kind Of Reflection

    Followup to: Where Recursive Justification Hits Bottom In " Where Recursive Justification Hits Bottom", I concluded that it's okay to use induction to reason about the probability that induction will work in the future, given that it's worked in the past; or to use Occam's Razor to conclude that the simplest explanation for why Occam's Razor works is that the universe itself is fundamentally simple.

  10. No Universally Compelling Arguments

    Followup to: The Design Space of Minds-in-General, Ghosts in the Machine, A Priori What is so terrifying about the idea that not every possible mind might agre

  11. Created Already In Motion

    Followup to: No Universally Compelling Arguments, Passing the Recursive Buck Lewis Carroll, who was also a mathematician, once wrote a short dialogue called What the Tortoise said to Achilles. If you have not yet read this ancient classic, consider doing so now.

  12. Sorting Pebbles Into Correct Heaps

    Followup to: Anthropomorphic Optimism Once upon a time there was a strange little species-that might have been biological, or might have been synthetic, and pe

  13. 2 Place And 1 Place Words

    Followup to: The Mind Projection Fallacy, Variable Question Fallacy I have previously spoken of the ancient, pulp-era magazine covers that showed a bug-eyed m

  14. What Would You Do Without Morality?

    Followup to: No Universally Compelling Arguments To those who say "Nothing is real," I once replied, "That's great, but how does the nothing work?" Suppose you

  15. Changing Your Metaethics

    Followup to: The Moral Void, Joy in the Merely Real, No Universally Compelling Arguments, Where Recursive Justification Hits Bottom, The Gift We Give To Tomorrow, Does Your Morality Care What You Think?, Existential Angst Factory, ... If you say, "Killing people is wrong," that's morality.

  16. Could Anything Be Right?

    Followup to: Where Recursive Justification Hits Bottom, Rebelling Within Nature Years ago, Eliezer1999 was convinced that he knew nothing about morality. For a

  17. Morality As Fixed Computation

    Followup to: The Meaning of Right Toby Ord commented: Eliezer, I've just reread your article and was wondering if this is a good quick summary of your positi

  18. Magical Categories

    Followup to: Anthropomorphic Optimism, Superexponential Conceptspace, The Hidden Complexity of Wishes, Unnatural Categories 'We can design intelligent machine

  19. The True Prisoner's Dilemma

    It occurred to me one day that the standard visualization of the Prisoner's Dilemma is fake. The core of the Prisoner's Dilemma is this symmetric payoff matrix

  20. Sympathetic Minds

    Followup to: Humans in Funny Suits "Mirror neurons" are neurons that are active both when performing an action and observing the same action-for example, a neuron that fires when you hold up a finger or see someone else holding up a finger.

  21. High Challenge

    Followup to: Not for the Sake of Happiness (Alone), Existential Angst Factory There's a class of prophecy that runs: "In the Future, machines will do all the

  22. Serious Stories

    Every Utopia ever constructed-in philosophy, fiction, or religion-has been, to one degree or another, a place where you wouldn't actually want to live. I am no

  23. Value Is Fragile

    Followup to: The Fun Theory Sequence, Fake Fake Utility Functions, Joy in the Merely Good, The Hidden Complexity of Wishes, The Gift We Give To Tomorrow, No Un

  24. The Gift We Give To Tomorrow

    Followup to: Thou Art Godshatter, Joy in the Merely Real, Is Morality Given?, Rebelling Within Nature How, oh how, did an unloving and mindless universe, coug

  25. Scope Insensitivity

    Once upon a time, three groups of subjects were asked how much they would pay to save 2000 / 20000 / 200000 migrating birds from drowning in uncovered oil ponds

  26. One Life Against The World

    Followup to: Scope Insensitivity "Whoever saves a single life, it is as if he had saved the whole world." -- The Talmud, Sanhedrin 4:5 It's a beautiful thou

  27. The Allais Paradox

    Followup to: But There's Still A Chance Right?, Beautiful Probability Choose between the following two options:1A. $24,000, with certainty.1B. 33/34 chance

  28. Zut Allais!

    Continuation of: The Allais Paradox Huh! I was not expecting that response. Looks like I ran into an inferential distance. It probably helps in interpreting

  29. Feeling Moral

  30. The "Intuitions" Behind "Utilitarianism"

    Followup to: Circular Altruism. Response to: Knowing your argumentative limitations, OR "one [rationalist's] modus ponens is another's modus tollens." (Stil

  31. Ends Don't Justify Means (Among Humans)

  32. Ethical Injunctions

  33. Something To Protect

    Followup to: Tsuyoku Naritai, Circular Altruism In the gestalt of (ahem) Japanese fiction, one finds this oft-repeated motif: Power comes from having somethin

  34. When (Not) To Use Probabilities

    Followup to: Should We Ban Physics? It may come as a surprise to some readers of this blog, that I do not always advocate using probabilities. Or rather, I d

  35. Newcomb's Problem And Regret Of Rationality

    Followup to: Something to Protect The following may well be the most controversial dilemma in the history of decision theory: A superintelligence from another galaxy, whom we shall call Omega, comes to Earth and sets about playing a strange little game.

  36. The Twelve Virtues Of Rationality

  37. My Childhood Death Spiral

    Followup to: Affective Death Spirals, My Wild and Reckless Youth My parents always used to downplay the value of intelligence. And play up the value of-effort

  38. My Best And Worst Mistake

    Followup to: My Childhood Death Spiral Yesterday I covered the young Eliezer's affective death spiral around something that he called "intelligence". Eliezer 1996, or even Eliezer 1999 for that matter, would have refused to try and put a mathematical definition-consciously, deliberately refused. Indeed, he would have been loath to put any definition on "intelligence" at all.

  39. Raised In Technophilia

    Followup to: My Best and Worst Mistake My father used to say that if the present system had been in place a hundred years ago, automobiles would have been outl

  40. A Prodigy Of Refutation

    Followup to: My Childhood Death Spiral, Raised in Technophilia My Childhood Death Spiral described the core momentum carrying me into my mistake, an affective death spiral around something that Eliezer 1996 called "intelligence". I was also a technophile, pre-allergized against fearing the future.

  41. The Sheer Folly Of Callow Youth

    Followup to: My Childhood Death Spiral, My Best and Worst Mistake, A Prodigy of Refutation "There speaks the sheer folly of callow youth; the rashness of an ignorance so abysmal as to be possible only to one of your ephemeral race..."

  42. That Tiny Note Of Discord

    Followup to: The Sheer Folly of Callow Youth When we last left Eliezer1997, he believed that any superintelligence would automatically do what was "right", and

  43. Fighting A Rearguard Action Against The Truth

    Followup to: That Tiny Note of Discord, The Importance of Saying "Oops" When we last left Eliezer 2000, he was just beginning to investigate the question of how to inscribe a morality into an AI. His reasons for doing this don't matter at all, except insofar as they happen to historically demonstrate the importance of perfectionism.

  44. My Naturalistic Awakening

    Followup to: Fighting a Rearguard Action Against the Truth In yesterday's episode, Eliezer2001 is fighting a rearguard action against the truth. Only graduall

  45. The Level Above Mine

    Followup to: The Proper Use of Humility, Tsuyoku Naritai (At this point, I fear that I must recurse into a subsequence; but if all goes as planned, it really w

  46. The Magnitude Of His Own Folly

    Followup to: My Naturalistic Awakening, Above-Average AI Scientists In the years before I met that would-be creator of Artificial General Intelligence (with a

  47. Beyond The Reach Of God

    Followup to: The Magnitude of His Own Folly Today's post is a tad gloomier than usual, as I measure such things. It deals with a thought experiment I invented to smash my own optimism, after I realized that optimism had misled me.

  48. My Bayesian Enlightenment

    Followup to: The Magnitude of His Own Folly I remember (dimly, as human memories go) the first time I self-identified as a "Bayesian". Someone had just asked

  49. Tsuyoku Naritai! (I Want To Become Stronger)

    In Orthodox Judaism there is a saying: "The previous generation is to the next one as angels are to men; the next generation is to the previous one as donkeys

  50. Tsuyoku Vs. The Egalitarian Instinct

    Hunter-gatherer tribes are usually highly egalitarian (at least if you're male)-the all-powerful tribal chieftain is found mostly in agricultural societies, rarely in the ancestral environment. Among most hunter-gatherer tribes, a hunter who brings in a spectacular kill will carefully downplay the accomplishment to avoid envy.

  51. Trying To Try

    "No! Try not! Do, or do not. There is no try." -Yoda Years ago, I thought this was yet another example of Deep Wisdom that is actually quite stupid.

  52. Use The Try Harder, Luke

    Followup to: Trying to Try "When there's a will to fail, obstacles can be found." -John McCarthy I first watched Star Wars IV-VI when I was very young. Se

  53. On Doing The Impossible

    Followup to: Use the Try Harder, Luke "Persevere." It's a piece of advice you'll get from a whole lot of high achievers in a whole lot of disciplines. I didn

  54. Make An Extraordinary Effort

    Followup to: Trying to Try, Tsuyoku Naritai "It is essential for a man to strive with all his heart, and to understand that it is difficult even to reach the

  55. Shut Up And Do The Impossible!

    Followup to: Make An Extraordinary Effort, On Doing the Impossible, Beyond the Reach of God The virtue of tsuyoku naritai, "I want to become stronger", is to a

  56. Final Words

    Sunlight enriched air already alive with curiosity, as dawn rose on Brennan and his fellow students in the place to which Jeffreyssai had summoned them. They sa

  57. Raising The Sanity Waterline

    To paraphrase the Black Belt Bayesian: Behind every exciting, dramatic failure, there is a more important story about a larger and less dramatic failure that m

  58. A Sense That More Is Possible

    Previously in series: Raising the Sanity WaterlineFollowup to: Teaching the Unteachable To teach people about a topic you've labeled "rationality", it helps f

  59. Epistemic Viciousness

    Previously in series: A Sense That More Is Possible Someone deserves a large hattip for this, but I'm having trouble remembering who; my records don't seem to

  60. Schools Proliferating Without Evidence

    Previously in series: Epistemic Viciousness Robyn Dawes, author of one of the original papers from Judgment Under Uncertainty and of the book Rational Choice i

  61. 3 Levels Of Rationality Verification

    Previously in series: Schools Proliferating Without EvidenceFollowup to: A Sense That More Is Possible I strongly suspect that there is a possible art of rati

  62. Why Our Kind Can't Cooperate

    Previously in series: Rationality Verification From when I was still forced to attend, I remember our synagogue's annual fundraising appeal. It was a simple en

  63. Tolerate Tolerance

    Followup to: Why Our Kind Can't Cooperate One of the likely characteristics of someone who sets out to be a "rationalist" is a lower-than-usual tolerance for f

  64. Your Price For Joining

    Previously in series: Why Our Kind Can't Cooperate In the Ultimatum Game, the first player chooses how to split $10 between themselves and the second player, a

  65. Can Humanism Match Religion's Output?

    Previously in series: Your Price for Joining Perhaps the single largest voluntary institution of our modern world-bound together not by police and taxation, no

  66. Church Vs. Taskforce

    Previously in series: Can Humanism Match Religion's Output?Followup to: Is Humanism a Religion-Substitute? I am generally suspicious of envying crazy groups o

  67. Rationality: Common Interest Of Many Causes

    Previously in series: Church vs. Taskforce It is a non-so-hidden agenda of this site, Less Wrong, that there are many causes which benefit from the spread of r

  68. Helpless Individuals

    Previously in series: Rationality: Common Interest of Many Causes When you consider that our grouping instincts are optimized for 50-person hunter-gatherer ban

  69. Money: The Unit Of Caring

    Previously in series: Helpless Individuals Steve Omohundro has suggested a folk theorem to the effect that, within the interior of any approximately rational,

  70. Purchase Fuzzies And Utilons Separately

    Previously in series: Money: The Unit of Caring Yesterday: There is this very, very old puzzle/observation in economics about the lawyer who spends an hour vo

  71. Bystander Apathy

    The bystander effect, also known as bystander apathy, is that larger groups are less likely to act in emergencies - not just individually, but collectively. Put an experimental subject alone in a room and let smoke start coming up from under the door. 75% of the subjects will leave to report it.

  72. Collective Apathy And The Internet

    Previously in series: Beware of Other-Optimizing Followup to: Bystander Apathy Yesterday I convered the bystander effect, aka bystander apathy: given a fixed problem situation, a group of bystanders is actually less likely to act than a single bystander.

  73. Incremental Progress And The Valley

    Followup to: Rationality is Systematized Winning Yesterday I said: "Rationality is systematized winning." "But," you protest, "the reasonable person doesn't a

  74. Bayesians Vs. Barbarians

    Previously in series: Collective Apathy and the InternetFollowup to: Helpless Individuals Previously: Let's say we have two groups of soldiers. In group 1,

  75. Beware Of Other Optimizing

    Previously in series: Mandatory Secret Identities I've noticed a serious problem in which aspiring rationalists vastly overestimate their ability to optimize o

  76. Practical Advice Backed By Deep Theories

    Once upon a time, Seth Roberts took a European vacation and found that he started losing weight while drinking unfamiliar-tasting caloric fruit juices. Now supp

  77. The Sin Of Underconfidence

    There are three great besetting sins of rationalists in particular, and the third of these is underconfidence. Michael Vassar regularly accuses me of this sin,

  78. Go Forth And Create The Art!

    Previously in series: Well-Kept Gardens Die By PacifismFollowup to: My Way I have said a thing or two about rationality, these past months. I have said a thi

This audiobook wouldn’t have been possible without our Kickstarter backers. Here they are:

  • Taymon
  • Chris Kleinknecht
  • Tom Dickinson
  • John Salvatier
  • Daniel Laplante
  • William S
  • Mark Arbogast
  • Eric Rogstad
  • Ka Woomba
  • Logan Airey
  • Kenny Evitt
  • Michael Plotz
  • Mintareja Budi Dharma
  • h
  • Mikhail Garvey
  • Drew Benedetti
  • Charles Tucker
  • Alex Demarsh
  • Matthew Fallshaw
  • Chris Carr
  • Alexander Steffen
  • Pete
  • Gregory Perkins
  • Clinton Pong
  • Jessica Chung
  • Lawrence Chan
  • Michal Bak
  • Svein Ove Aas
  • Daniel Morgan
  • Andrew Mehler
  • James Bowles
  • Jonas Nystrom
  • Julien Couvreur
  • ori
  • Joseph Gnehm
  • Dan Nuffer
  • Alex Cardy
  • Kelp Farmer
  • Eric M Vincent
  • Scott Garrabrant
  • Eric Lin
  • Michael Drake
  • Chris Hawkins
  • Pat Connelly
  • john philpot
  • Mick Porter
  • William Everett
  • Craig Lewin
  • Quintrino
  • Brian Kirby
  • Alan Williamson
  • Nigel Leslie
  • Kazi Siddiqui
  • Gavan Wilhite
  • Jared Pierce
  • Daniel Yokomizo
  • Pavel Savara
  • Malcolm Ocean
  • Jared Trezise
  • Peter Burns
  • Nolan
  • Jordan Onuma
  • Ben
  • Taryn East
  • Jim Spofford
  • Malcolm Reid Jr
  • Brian Brunswick
  • Todd
  • Julian Wicks
  • Nick Mathewson
  • Coeurl
  • Andrew Pearson
  • Manish Mathai
  • Nope Then
  • MarsColony_in10years
  • nightmare1zero1
  • Charlie Dake
  • Arran Stirton
  • Edan Maor
  • Richard Neilsen
  • Gavin Smith
  • Wes Winham
  • Kazi Siddiqui
  • Elizabeth Edwards-Appell
  • Adrian Stymne
  • Tianyi Cui
  • Andrew