[ih] Karl's post from Friday: Re: Interop as part of Internet History (was Re: Fwd: Fwd: List archives (Was: Exterior Gateway Protocol))

[Joseph Touch]

Subject: Re: [ih] Interop as part of Internet History (was Re: Fwd: Fwd: List archives (Was: Exterior Gateway Protocol))
Date: Fri, 11 Sep 2020 14:56:57 -0700
From: Karl Auerbach <karl at cavebear.com> <mailto:karl at cavebear.com>
To: Dan Lynch <dan at lynch.com> <mailto:dan at lynch.com>, Jack Haverty <jack at 3kitty.org> <mailto:jack at 3kitty.org>
CC: internet-history at elists.isoc.org <mailto:internet-history at elists.isoc.org>

On 9/10/20 9:26 PM, Dan Lynch via Internet-history wrote:
> “I know it works. I saw it at Interop.”
> 
> That became our tag line in our early marketing of Interop.

Interop had two major parts, maybe three.

- There was the "show net".

- There were the vendors (who had to attach to the show net)

- There were presentations/papers.

The show was one of the critical drivers that forced vendors to have working products.

Interop was a loud bell that rang once or twice a year. And lord help the vendor whose products were not ready when that bell tolled.

(Another major driver of vendors was the now largely forgotten Air Force ULANA procurement.)

There were a lot of companies who thought they were interoperable but discovered otherwise at the show. We were not shy about turning off vendors that were so non-interoperable that they were causing problems.

The importance of Interop in forcing early Internet products to work together can not be overstated.

For fifteen or so years I was part of the core volunteer "NOC Team" that designed, installed, and operated that net.

It was not a small net - tens of thousands of hosts, lots of internal routing issues (we had massive redundancy), multiple external providers (hence a lot of BGP juggling), big DNS, heavy traffic loads, external attacks, etc

Many of the early Interop show networks were designed at my house in Santa Cruz.

(I also met my wife via the show net; she was working at Dan's ACE [Another Cute Employee] and charge of the volunteer core team.)

Here's a video or us installing the '93 net (either San Francisco or DC) that Dan Lynch had made (by Linda Fefferman). (I spoke to her a few years back and she still had the raw betamax tapes from which this video was made.)

https://youtu.be/SMkKIaHee4c <https://youtu.be/SMkKIaHee4c>

Here's a few, very few, early photos:

https://www.cavebear.com/archive/interop/ <https://www.cavebear.com/archive/interop/>

We had a class A /8 (45.x.x.x) to play around with. We subnetted it heavily.

In the early days we also carried DECnet, Netware/IPX, as well as IPv4 (including IP multicast). We also did IPv6 fairly early.

We (John Romkey, Simon Hackett, and I) showed the first two Internet Toasters (yes there were two) at the 1989 show in San Jose. Unfortunately on the first day we forgot to bring bread - so we had to toast and re-toast the same slice.

In 1997 during one of the shows I called from the show floor over an early IP based phone into an NTIA conference call - for most of the participants that was the first IP based phone call they had ever heard.

And during that same show I used a wi-fi and IP multicast battery powered camera (mounted on a hard hat) to broadcast interviews on the show floor - I kinda looked like a mad bomber, my wife called it the "husband cam" because she could see who I was looking at as I wandered the show floor.)

We quickly developed a rib-and-spine approach for convention centers. In the San Jose and DC shows we used lots and lots of routers - Cisco, Wellfleet, and 3COM.

Management of that was a pain, so we quickly developed two management networks - a so called "spy" net which used remote controlled optical mirrors so that we could create direct bi-directional ethernet paths onto any part of the show network. We used that for monitoring and getting to disconnected parts. We also had a separate management net that led us to the console ports of all the routers (and switches.)

One of our important developments was the ability to pre-wire a convention center in a warehouse - by the time we were doing the Atlanta shows it took 45 fully loaded trucks to haul our gear from our warehouse to the convention center. Dave Bridgham and I considered bying a C 130 to get some of the gear around quickly. And we really stressed out the local Frys.

We developed a rubber-bungee strap system so that we could dangle the booth drops from the convention center ceiling, but above the roofs of the trucks that drive around the show floor during setup. After the show the bungees would pull the drops back up out of the way of the trucks. We also learned how to co-exist on friendly terms with the local unions - many of workers were not then familiar with network tech and were more than willing to work with us to learn.)

We also developed a portable fiber plant using a massive "pink" cable with some serious (and expensive) quick-connect plug/socket devices - we bought out the entire national supply of those (and discovered that we had depleted the US national military stockpile of 'em.)

We also got really good at moving a /8 around the world - sometimes in a matter of days. During hook-up we would stress-out most providers BGP implementations due to the up/down cycles during hook-up. That's when many providers learned about route flapping and damping.

Our core team evolved over the years - at the start it was small, with people like Simon Hackett (https://www.cavebear.com/archive/interop/Internode <https://www.cavebear.com/archive/interop/Internode>); John Romkey, Dave Bridgham, Stev Knowles, Peter deVris (FTP software); Geoff Baehr (Sun); Stuart Vance (TGV); me (Epilogue Technology, Empirical Tools and Toys); etc.

The first year or two was mostly yellow-hose ethernet.

Then David Systems and Synoptics turned us onto 10-Base-T. Which we mis-wired with stranded-wire connectors and had to re-connect the entire show floor (by hand) overnight.

Eventually we got good with fiber, managed by David Steele (who did some of the fiber plans for the 777 aircraft) and Merike Kaeo (Merike also handled our early FDDI deployment - during which we found flaws in the specifications and vendors were deploying fixes during the show.) And we got really good at both local routing and multi-homed external routing. Our three router goddesses were Cindi Jung (3COM), Robin Littlefield (Wellfleet), and Chris Pecina (Cisco?)

We weren't beyond pulling a few legs. A surprising number of people (and tech journalists) believed this press release we issued:

https://www.cavebear.com/archive/cavebear/catalogue/maypressrelease.html <https://www.cavebear.com/archive/cavebear/catalogue/maypressrelease.html>

One year when we had something like 90 routers on the show floor we bought a bunch of spacey-looking TV dish antennae, put one on each router, and aimed them at a disco ball on the convention center ceiling - and told people that it was a low-earth orbit geosynchronous satellite.

I gave the first "tour" of our show net at the first DC show to Jack Haverty and Vint Cerf. That tour evolved into a regular part of the show. (That may have been the year when we got snowed into the convention center.)

Over the years our group grew - there are a few hundred members by now - and we got really good a "commando networking". We could wire up a big convention center (like Atlanta) plus hotels (in Atlanta we ran illicit cables through active train tunnels.) We got good at running laser aimed links between hotel room windows and remote rooftops.

Much of what we learned made its way out to the ISP world. I developed test and diagnostic tools that found their way into products by Fluke and others.

Other ideas vanished - like my demo of inserting words into VoIP calls or our demo of a RAID 5 array made of USB flash drives via iSCSI over wi-fi.

--karl--