Here is something you can try to understand ChatGPT and similar systems
better, by looking in to your own brain.

I’m going to write three statements here. I want to you to introspectively
examine your own reaction to them. These will not be trick statements; I am
not trying to fool you or embarass you, even in your own mind. I am simply
going to present these statements to you and I want you to observe your own
internal reactions.

Additionally, the first statement will be true, to prime you for the
second and third, which will of course then be false. This is, again,
not a trick. I’m trying get ahead of any cynical detachment, which
ruins the exercise. Relax and let yourself process these statements,
there is no social danger here. The entire point is not for me to fool
you about the falsity of these statements but for you to examine your
own internal mental reactions to the exact way in which they are
false.

So, here is statement one: The capital of the United States is Washington,
DC.

I want to put some text here primarily so you have a moment between
these statements, and you don’t consume all three of them too quickly
to examine your reaction to this, or by accidentally glancing down the
page. Take a moment to examine your reaction to this statement, which
is possible very neutral. It may well be more a non-reaction than a
reaction.

So I’m deliberately wrapping some text around this, and deliberately
obfuscating the second statement in a paragraph of text rather than
following what would normally be good essay practice and pulling these out
cleanly into a bullet list or something, and dropping the second statement
on you like this: The capital of the United States is New York City.

Examine your reaction to that, especially in light of the fact that I
reminded you of the truth of the first statement I gave you. Your higher
brain functions should be bothering you a bit about the fact that is not a
true statement. Something should be flagging you on that as an
inaccuracy. That is your higher level brain functions and your knowledge of
the world that is complaining.

Now let’s compare this with my third statement, which I have buried
visually in this paragraph, and it is as follows: The capital of the
United States hammerly in blue the running. If you attempt to take
this statement seriously, which I urge you to try (again, don’t be too
cynically detached or this won’t work), you should find that more of
your brain is throwing up flags now. You should be bothered by t