The world in which IPv6 was a good design

Last November I went to an IETF meeting for the first time. The IETF is an
interesting place; it seems to be about 1/3 maintenance grunt work, 1/3
extending existing stuff, and 1/3 blue sky insanity. I attended mostly
because I wanted to see how people would react to TCP BBR, which was being
presented there for the first time. (Answer: mostly positively, but
with suspicion. It kinda seemed too good to be true.)

Anyway, the IETF meetings contain lots and lots of presentations about IPv6,
the thing that was supposed to replace IPv4, which is what the Internet runs
on. (Some would say IPv4 is already being replaced; some would say it has
already happened.) Along with those presentations about IPv6, there were
lots of people who think it’s great, the greatest thing ever, and they’re
pretty sure it will finally catch on Any Day Now, and IPv4 is just a giant
pile of hacks that really needs to die so that the Internet can be elegant
again.

I thought this would be a great chance to really try to figure out what was
going on. Why is IPv6 such a complicated mess compared to IPv4? Wouldn’t
it be better if it had just been IPv4 with more address bits? But it’s not,
oh goodness, is it ever not. So I started asking around. Here’s what I found.

Buses ruined everything

Once upon a time, there was the telephone network, which used physical
circuit switching. Essentially, that meant moving connectors around so that
your phone connection was literally just a very long wire (“OSI layer 1″). A
“leased line” was a very long wire that you leased from the phone
company. You would put bits in one end of the wire, and they’d come out the
other end, a fixed amount of time later. You didn’t need addresses because
there was exactly one machine at each end.

Eventually the phone company optimized that a bit. Time-division
multiplexing (TDM) and “virtual circuit switching” was born. The phone
company could transparently take the bits at a slower bit rate from multiple
lines, group them together with multiplexers and demultiplexers, and let
them pass through the middle of the phone system using fewer wires than
before. Making that work was a little complicated, but as far as we
modem users were concerned, you still put bits in one end and they
came out the other end. No addresses needed.

The Internet (not called the Internet at the time) was built on top of these
circuits. You had a bunch of wires that you could put bits
into and have them come out the other side. If one computer had two or
three interfaces, then it could, if given the right instructions, forward
bits from one line to another, and you could do something a lot more
efficient than a separate line between each pair of computers. And so IP
addresses (“layer 3”), subnets, and routing were born. Even then, with
these point-to-point links, you didn’t need MAC addresses, because once a
packet went into the wire, there was only one place it could come out. You
used IP addresses to decide where it should go after that.

Meanwhile, LANs got invented as an alternative. If you wanted to connect
computers (or terminals and a mainframe) together at your local site, it was
pretty inconvenient to need multiple interfaces, one for each wire to each
satellite computer, arranged in a star configuration. To save on
electronics, people wanted to have a “bus” network (also known as a
“broadcast domain,” a name that will be important later) where multiple
stations could just be plugged into a single wire, and talk to any other
station plugged into the same wire. These were not the same people as the
ones building the Internet, so they didn’t use IP addresses for this. They
all invented their own scheme (“layer 2”).

One of the early local bus networks was arcnet, which is dear to my heart (I
wrote the first Linux arcnet driver and arcnet poetry way
back in the 1990s, long after arcnet was obsolete). Arcnet layer 2
addresses were very simplistic: just 8 bits, set by jumpers or DIP switches
on the back of the network card. As the network owner, it was your job to
configure the addresses and make sure you didn’t have any duplicates, or all
heck would ensue. This was kind of a pain, but arcnet networks were usually
pretty small, so it was only kind of a pain.

A few years later, ethernet came along and solved that problem once and for
all, by using many more bits (48, in fact) in the layer 2 address. That’s
enough bits that you can assign a different (sharded-sequential) address to
every device that has ever been manufactured, and not have any overlaps.
And that’s exactly what they did! Thus the ethernet MAC address was born.

Various LAN technologies came and went, including one of my favourites, IPX
(Internetwork Packet Exchange, though it had nothing to do with the “real”
Internet) and Netware, which worked great as long as all the clients and
servers were on a single bus network. You never had to configure any
addresses, ever. It was beautiful, and reliable, and worked. The golden
age of networking, basically.

Of course, someone had to ruin it: big company/university networks. They
wanted to have so many computers that sharing 10 Mbps of a single bus
network between them all became a huge bottleneck, so they needed a way to
have multiple buses, and then interconnect – “internetwork,” if you will –
those buses together. You’re probably thinking, of course! Use the Internet
Protocol for that, right? Ha ha, no. The Internet protocol, still not
called that, wasn’t mature or popular back then, and nobody took it
seriously. Netware-over-IPX (and the many other LAN protocols at the time)
were serious business, so as serious businesses do, they invented their own
thing(s) to extend the already-popular thing, ethernet. Devices on
ethernet already had addresses, MAC addresses, which were about the only
thing the various LAN protocol people could agree on, so they decided to use
ethernet addresses as the keys for their routing mechanisms. (Actually they
called it bridging and switching instead of routing.)

The problem with ethernet addresses is they’re assigned sequentially at the
factory, so they can’t be hierarchical. That means the “bridging table” is
not as nice as a modern IP routing table, which can talk about the route for
a whole subnet at a time. In order to do efficient bridging, you had to
remember which network bus each MAC address could be found on. And humans
didn’t want to configure each of those by hand, so it needed to figure
itself out automatically. If you had a complex internetwork of bridges,
this could get a little complicated. As I understand it, that’s what led to
the spanning tree
poem, and I think I’ll just leave it at that. Poetry is very important
in networking.

Anyway, it mostly worked, but it was a bit of a mess, and you got broadcast floods
every now and then, and the routes weren’t always optimal, and it was pretty
much impossible to debug. (You definitely couldn’t write something like
traceroute for bridging, because none of the tools you need to make it work
– such as the ability for an intermediate bridge to even have an address –
exist in plain ethernet.)

On the other hand, all these bridges were hardware-optimized. The whole
system was invented by hardware people, basically, as a way of fooling the
software, which had no idea about multiple buses and bridging between them,
into working better on large networks. Hardware bridging means the bridging
could go really really fast – as fast as the ethernet could go. Nowadays
that doesn’t sound very special, but at the time, it was a big deal.
Ethernet was 10 Mbps, because you could maybe saturate it by putting a bunch
of computers on the network all at once, not because any one computer could
saturate 10 Mbps. That was crazy talk.

Anyway, the point is, bridging was a mess, and impossible to debug, but it
was fast.

Internet over buses

While all that was happening, those Internet people were getting busy, and
were of course not blind to the invention of cool cheap LAN technologies. I
think it might have been around this time that the ARPANET got actually
renamed to the Internet, but I’m not sure. Let’s say it was, because the
story is better if I sound confident.

At some point, things progressed from connecting individual Internet
computers over point-to-point long distance links, to the desire to connect
whole LANs together, over point-to-point links. Basically, you wanted a
long-distance bridge.

You might be thinking,