[ih] Interop historical material?

[Jack Haverty]

Very nice, Karl!   Good stories.  Interop was a crucial part of Internet 
History.   Building an Internet involves a lot more than writing RFCs.

I found the description of the Interop "spy net" interesting.  I never 
knew how you guys managed to corral that beast of wires and boxes 
scattered across a convention hall.

That reminded me of a discussion that occurred back in the late 70s 
timeframe, when TCP V2 was undergoing open heart surgery to recast it as 
TCP/IP V4.   The debate centered on whether or not there should be "out 
of band control" for the various pieces of the Internet, so that 
debugging problems could be done remotely even when the basic Internet 
service itself wasn't working for whatever reason.

We never got very far on describing the "spy net", and eventually 
concluded it would just make the Internet mechanisms too complex and 
slow things down.   So things like SNMP and ICMP were defined on top of 
the IP service instead of some separate "spy net" (call it a control 
plane if you like...), and we hoped it would be OK.

Of course today's boxes, at least the consumer-level ones that I have as 
an end-user, don't have RS232 console ports any more, so the Interop 
technique wouldn't work now.   It just dawned on me that I *did* put in 
a rudimentary independent "control plane" in the equipment running now 
in my house.   My internet boxes are plugged into "smart home" outlets, 
that can be independently controlled through radio channels separate 
from Wifi (I use Zwave mostly).

That doesn't have as much functionality as an RS232 console, but it does 
provides the basics.  When something's wrong, a simple tap on my phone 
turns off the Internet boxes and turns them back on.

First rule of network troubleshooting -- try rebooting!

Jack Haverty

PS - speaking of ACE, see attached image; that's been on my desk as 
landing pad for my coffee mug for close to 40 years.


On 12/1/21 12:58 PM, Karl Auerbach wrote:
> On 11/22/21 6:26 PM, Jack Haverty via Internet-history wrote:
>> Speaking of Interop....
>>
>> Has any of the material from the Interops been preserved? There were 
>> administrative pieces, like lists of vendors, floor maps, titles of 
>> sessions, speakers' presentations, attendee lists, and such as well 
>> as mountains of presentation slides, papers, vendor handouts, and 
>> promotional material.
>
> I was a leader of the team that designed and deployed the Interop nets 
> from the outset in the late 1980's.  Many of the early show nets were 
> designed at my house.  My now wife worked at Dan Lynch's company ACE 
> ("Another Cute Employee") trying to manage the herd of cats that was 
> the shownet team.
>
> A lot of material from Interop exists.  There is an active body of 
> folks who designed, deployed, and operated the show net over the years.
>
> My own website has some ancient materials 
> (https://cavebear.com/archive/interop/) including the handout from the 
> tours we gave of the net infrastructure.  (BTW, the very first of 
> those tours was the one I gave to you and Vint at one of our DC shows.)
>
> There was also the Linda Feferman film from an early show - 
> https://youtu.be/SMkKIaHee4c   I'd sure like to get a better print of 
> that, or better the original, uncut footage.
>
> Those of us who worked on the Interop net have masses of materials 
> cached away, much of it online.
>
> But some was definitely not preserved - we never talked about our 
> movie nights when we'd take over some comfortable vendor booth at 
> night when the show was closed and watch movies and pass around a lot 
> of single malt - most of which we bought using Dan Lynch's credit card.
>
> And more than one of us met our then-to-be spouses while working on 
> the show net.
>
> The show had a rule - connect and try to interoperate.  We were not 
> nice to vendors who tried to be islands.  And when we did find a 
> vendor that was doing bad things (whether by intent or accident) we 
> were not slow to cut them off.
>
> We had to make things work - this was practical networking at its 
> finest and most stressful.  We had to cover every detail from 
> connectors and splicing to multi-homed BGP to IP multicast to ATM 
> VPI/VCI routing to dragging wire through parking lots and coring 
> buildings.
>
> There were several dimensions to the show net.  The most obvious was 
> the topology.  We had 45/8 as our address space that we dragged around 
> from place to place.  Not long after we started we multi-homed that 
> address block, which often severely stressed the route damping of our 
> external connectivity providers as we bounced up and down in the days 
> when we brought up that block in a new location.
>
> What was not seen was the evolution from a star configuration to a 
> rather massive rib-and-spine physical topology.  We had to get the 
> basic infrastructure into a convention center fairly quickly before 
> the trucks came onto the floor.  We had to invent a system of 
> pull-down drops with bungie cords to stay out of the way of the 
> trucks.  And within minutes after closing we had to release those 
> drops back to the ceiling to get out of the way of the trucks removing 
> the vendors stuff.  And there was a lot of other network cable 
> infrastructure that went out to areas other than the show floor.  For 
> example, we often had laser scopes on building rooftops to provide 
> links to other locations.
>
> We had everything.  In the early shows we had TCP/IP, ISO/OSI, DECnet, 
> and Netware.  We had lots and lots of routers - we had pairs of 
> Cisco's and Wellfleets and I think some Proteon and 3COM routers.  We 
> had lots of media types.  The early shows used yellow hose ethernet 
> but we jumped onto Synoptics and David Systems twisted pair ethernet 
> the moment we first saw it.  We also had FDDI (and we found bugs in 
> the specifications) and ATM and just about everything else, including 
> ethernet-over-barbed wire.  We were also fairly intent on making IP 
> multicast work (and it did.)
>
> Rarely seen were the warehouses where we built the intrastructure 
> before the show and did a lot of testing (including full 
> power-off/restore testing.)  We would build the net and pack it onto 
> large trucks for shipping to the convention centers.  I remember us 
> loading 46 large trucks on one occasion.  Dave Bridgham (FTP Software) 
> and I were considering buying a used C130 as a way to move some of our 
> gear, especially when we had a fast shift, such as from Las Vegas to 
> Paris or Tokyo in a week or two.
>
> The convention centers were large - it was physically impossible to 
> get around when things went awry.  So we deployed an entirely separate 
> physical network that we called the "spy network".  We used this for 
> several things.  One was to get to terminal servers that attached to 
> the RS232 serial ports of all our our infrastructure devices.  That 
> way we could get control from our NOC.  We also had systems of mirrors 
> mounted on piezo-electric crystal steering devices so that we could 
> switch the fiber optic links of the spy network so that we could drop 
> a packet sniffer anywhere we wanted.  (We had passive splitters on 
> many of the fiber links.)
>
> We also used a lot of rather heavy grade fiber cables - we ended up 
> depleting the US military's stockpile of certain many-strand, 
> quick-connect fiber connectors.
>
> I developed the first Internet "butt set" as a tool to get out on the 
> floor and begin diagnosing problems within seconds after arrival.  
> (Parts of that still exist in some of the tools now sold by companies 
> such as Fluke and one of my fellow designers of that went on to form 
> other diagnostic tool companies, such as Air Magnet.)
>
> We had a lot of fun doing the show net.  For instance we had very 
> early VoIP - I remember being on a call with NTIA about the yet-to-be 
> formed ICANN and mentioned that I was calling over the net (from the 
> show floor) and hearing some very surprised sounds from people who did 
> not know that such a thing was even possible.
>
> And on another occasion I used some early RTP/RTCP based audio-video 
> software from Precept Software - I had a camera+microphone duct-taped 
> onto a hardhat and carried a laptop with a stack of batteries 
> duct-taped on.  I sort of looked like a terrorist-in-training.  I 
> interviewed people on the show floor. My wife called it the "husband 
> cam" because she could see everyone I looked at - and I looked a lot: 
> in those days the vendor booths often had what we referred to as 
> "booth bunnies".
>
> The show was a target for attack.  The very first attack was rather 
> mild - Carl Malamud and I were setting up some NCD Xterms and suddenly 
> a foreign desktop appeared on all of them.  Someone was trying to 
> steal username/password pairs via faux login screens.
>
> The net at the shows was just a thin layer over the work to get that 
> net deployed.  We had to deal with immature technologies, new 
> implementations, and constraints that were often far from technical.  
> For instance, we had senior union electricians who figured they could 
> bend, cut, and splice our coax cables - or worse, our fiber optic 
> cables. We learned how to deal with the unions - from peeling off 
> bills from thick wads of twenties, to pointing out that fiber optic 
> cables are pipes for light and perhaps could be better handled by 
> people from the plumbers' unions, to simply bringing along our own, 
> and trusted, union electricians.)  We weren't always within the law - 
> like when we pulled fiber optics through the active railroad tunnels 
> at the Atlanta convention center.
>
> We were a smelly bunch - convention centers are not air conditioned 
> while the trucks are going in/out.  And in Las Vegas that meant 
> working in 100F+ heat.  And we would work 24 hours a day for days on 
> end.  We had to mandate sleep and shower periods.   (But even then we 
> were a smelly bunch - for instance when we got snowed in in DC and 
> trucks started dumping manure for a garden show days before we were 
> able to get out networking gear out.)
>
> In addition to the tech a lot of other stuff happened.  For instance I 
> first met a friend by climbing through her car window as we went to 
> blow off steam doing some white water rafting in western 
> Pennsylvania.   About 30 of us (all wearing Motorola radios that kept 
> squawking) got mugged directly in front of the White House.  (Although 
> the muggers had guns I think our group was better armed - one does not 
> realize how useful to networking a good knife can be.)  We had 
> parties, such as when we rented the Air and Space museum in DC, the 
> roof of the the La Defense arch in Paris, and the Howard Hughes Suite 
> (two floors) at the now gone Desert Inn in Las Vegas.
>
>         --karl--
>