[ih] NIC 7104 (ARPANET Protocol Handbook)

[Jack Haverty]

On Fri, 2006-04-28 at 09:07 -0400, Craig Partridge wrote:
> So how did mailers handle this? Did they try MLFL and
>     when it bounced with a 5xx message say, "oh $%!, its a Multics
>     machine?" and try a USER ANONYMOUS command.  Or did they do
> something
>     smarter, or did Multics just not get all the mail coming to it?

Yep, all sorts of tricks were necessary.  

I wrote the mail daemon running on MIT-ITS, which talked a lot with
MIT-MULTICS back in the mid/late 70s.  The daemon was written in Muddle
(a home-grown language that looked somewhat like LISP).  This daemon
tried to process messages as structured data; today we would call it a
relational database, but that technology hadn't been invented yet.  

There was a machine online at CCA called "The DataComputer" that was
probably the first online network database.  It provided a very
primitive (by today's standards) interface for storing and querying data
- SQL did not yet exist.   The idea was to have an online repository
into which important messages could be placed and made available for
posterity to search and retrieve - we called this a "message archive".

To make the messages in the Datacomputer useful, it was important to get
the data precise.  E.G., a field like "To: PDL, Cerf at ISIA" was ambiguous
was "PDL" really "PDL at ISIA" (picking up the host from the end of the
line)?  Or was it "PDL at MIT-DMS" (picking up the host from the "From:
JFH at MIT-DMS" elsewhere in the header)? 

Various mail programs adopted different such "abbreviations" which drove
me crazy.  This is in addition to the quirks in the interactions over
the net (e.g., the Multics problem with USER). 

To handle all of this protocol chaos, I wrote (and rewrote, and tweaked)
a sizable (for a LISPish world) chunk of code to try to deduce the
precise meaning of each message header contents and semantics based on
where the message came from.  Different mail programs had different
ideas about the interpretation of fields in the headers.

That code first tried to figure out where an incoming message had come
from.  This was not so obvious as it might seem because of
redistribution and forwarding of messages, and differences in behavior
of various versions of the other guy's software.   So it wasn't enough
to just look to see if you were talking to MIT-MULTICS.   I remember
having conditional clauses that in essence said "If I see a pattern like
such-and-such in the headers, this is probably a message from version
xx.yy of Ken Pogran's Multics mailer."   With enough such tests, it
formed an opinion about which mail daemon it was talking with, and which
mail UI program had created a message.

Having hopefully figured out the other guy's genealogy (and therefore
protocol dialect), the code then acted based on a painfully collected
set of observations about how that system behaved.   This is where
things like "It looks like Multics, let's try USER ANONYMOUS" came into
play.   That particular one was easy.   Figuring out many of the "naked
usernames" was very difficult.

Of course, as mail usage grew this nonsense got more and more painful
and unreliable.  So eventually, for this and other reasons (e.g., poor
query language of the Datacomputer), we abandoned the notion of trying
to archive messages in a structured way in an online database.   It's
really too bad - it would be nice today to be able to search
HEADER-PEOPLE and other archives that way.

The root of the problem was the conflict between people who viewed the
protocol as a human-friendly environment, where someone could read the
textual parts of the protocol interactions and do the right thing, and
people (me) who were trying to view the protocol as a computer-friendly
environment.

I tried pushing the envelope toward computer-friendly by writing up RFCs
713 and 722, but there weren't many other people trying to do things
like using the Datacomputer, so human-friendly (and therefore slightly
fuzzy) protocols prevailed.

Error messages (like the 5xx responses) were similarly unreliable.   I
remember one anecdote...  One day Ken Pogran asked me why I had just
sent him a message with comments about some document (probably an ARPA
proposal) that we had finished a year prior.  I hadn't sent such a
message, so he showed it to me, and sure enough it was my message -
except that the "Date:" field showed that I sent it about a year
earlier, but he didn't get it until that day.  So, where had it been all
year....?

Well, it was a short message, something like: "Ken - take a look at
this:  FILE NOT FOUND  /Jack"   We both dug into our respective year-old
pile of mailer daemon logs to figure it out.

Well, my mail composition program had a feature where you could "attach"
a file.  This wasn't attachments like we have today since the protocol
didn't allow for that.  But when I composed a message, rather than
inserting the file into the editor, I could tell the mail daemon to
insert the file on-the-fly as it sent it out over the network.

My wayward message had a large attached file - probably a new version of
the document.   But when the message got to Ken, instead of the
document, it just said FILE NOT FOUND.

It turned out that just before Ken got that message, I had a rare
episode of cleaning out old disk files, and I deleted the document -
hence the FILE NOT FOUND.

In the logs, we found the evidence.  Dutifully, mindlessly, stubbornly,
persistently, my mailer daemon had been connecting to Multics, every few
minutes or so, for *a year*, and sending the whole message with large
document.   Multics slurped it all in, but had an unpublished policy of
not accepting messages bigger than a certain size.   So it responded to
the (lengthy) MAIL (or MLFL, I forget) command with an error, something
like: "XXX Message exceeds size limits, please send smaller messages."

I always have wondered what percentage of the overall Arpanet traffic
that year was attributable to that one message.....

The problem?   Well, the Multics mailer used an XXX code which was one
of the "temporary error, try again later" flavors, and my mail daemon
wasn't clever enough to read/understand the ascii text, nor suspicious
enough to set some kind of "time-to-live" counter on outgoing queued
messages.   So it tried again, and again, and again, .....

We really really should have looked at the logs more often.

When I finally deleted the document, the message suddenly became small
enough to please the Multics bureaucracy, ... and the mail went through.

So, another few lines of code in the mail daemon's "it looks like
Multics" section went in that day.

Then there was the day that three people's mailing lists got
accidentally configured into a loop and created an out-of-control data
firehose spanning the continent .... but that's another story.

By the way, those incidents were the motivation for putting the
"MESSAGE-ID" field in the headers, so you could write code to detect
such behavior.   But I doubt there are (m)any mailer daemons today that
do anything "smart" with that information.

Yep, lots of tweaks, lots of heuristics.   But smart?   Your call...

/Jack