[ih] Distributed file systems [was: As Flag Day approaches at CMU]
Brian E Carpenter
brian.e.carpenter at gmail.com
Sat Sep 6 18:55:37 PDT 2025
"MIT wasn't the only place where the IMP became the de facto local area net."
Which reminded me of Scrapbook at NPL. It was an early hyperlinked system but was also a (small scale) distributed file system by the mid 1970s. It was not widely known and is badly documented.
I happened to meet and interview one of the Scrapbook team last year:
https://www.cs.auckland.ac.nz/research/groups/CDMTCS/researchreports/download.php?selected-id=884
The Andrew File System and its descendants, like NFS, only came along in the 1980s. AFS in particular *must* have been influenced by the Unix United paper, which had examples like:
cd /../unix2/user/brian
quicksort a > /../unix1/user/brian/b
(where unix1 and unix2 were host names, and brian wasn't me, it was Brian Randell.)
Regards/Ngā mihi
Brian Carpenter
On 07-Sep-25 12:05, Steve Crocker wrote:
> Jack,
>
> I spent a year and half in the MIT-AI lab (Feb 1967 to May 1968), the next three years (May 1968 to May 1971)at UCLA working on the Arpanet protocols, the next three years (July 1971 to August 1974) at (D)ARPA primarily overseeing the AI portfolio but also paying attention occasionally to Arpanet issues. As part of this work I saw the multiple operating system and architecture projects at the various sites, including in particular the C.mmp and other efforts at CMU.
>
> Later, around 1990 or so, Jerry Popek at UCLA and I at Trusted Information Systems worked together on extending his shared Unix file structure to work over the Internet using secured file transfer. That project was called Truffles. I share these biographical details as a preface to my remarks embedded below.
>
> On Sat, Sep 6, 2025 at 7:24 PM Jack Haverty via Internet-history <internet-history at elists.isoc.org <mailto:internet-history at elists.isoc.org>> wrote:
>
> I doubt anything was published. It just wasn't that big a deal, and
> wasn't part of the research mission.
>
>
> Heh, heh. This is an example of some really great work done at MIT-AI lab that was IMO underplayed. Documentation and publication of the MIT-AI work was almost non-existent. From the perspective of sitting at my desk at DARPA, it was pretty challenging figuring out what to write each year. Further, the small hack you're describing could have been the basis for a more common network service. However, it would have required a fair amount of work to deal with the differences across the operating systems and, of course, the access control problems. The ITS designers, at least in the early days, took pride in avoiding heavy duty access control. It was a delight to use ITS, and I fully admired the MIT-AI hackers. But Tenex had stronger controls and quite usable software, so it won the day. (DEC had its own operating system, TOPS-10, which was much weaker. They eventually licensed Tenex from BBN and brought it out as TOPS-20.)
>
>
> The "distributed file system" I described was "a hack" probably coded
> overnight when someone realized that it could be easily done by using
> the mechanisms that already existed inside ITS - like the JOB/BOJ
> pseudo-devices. Such work wasn't part of any official project, but was
> done primarily to mitigate the scarcity of CPU cycles, memory, or disk
> space. It might have been mentioned in the annual report of the AI Lab
> work, but I doubt there was ever a paper published.
>
>
> Yeah. See above. It would have been an excellent and substantial contribution to the Internet if the work had been pursued. It took a very long time before there were semi-sensible solutions to the file-sharing problem. The early commercial offerings were ridiculously expensive. Two of us founded Shinkuro to provide an essentially free way to share files across the net based on everyone having a local copy of each shared file, i.e. no central servers. Dropbox, et al eventually won the day.
>
>
> With several ITSes on the ARPANET, and the hassle of dealing with FTP to
> move files, someone likely noticed that a "remote disk" capability would
> be useful and was an easy thing to try. I recall someone once
> wondering why there was a lot of traffic between hosts on the MIT IMP,
> where other sites traffic patterns were mostly long distance. MIT used
> the IMP as a poor man's LAN, and made it as easy as possible by coding
> up things like remote disks.
>
>
> MIT wasn't the only place where the IMP became the de facto local area net. And it wasn't long before there was a broad realization that the majority of the communication in the Internet was local.
>
> My favorite example was at UCSB where they had been trying to add interprocess communication to OS/MVT on their IBM 360/75 to make it possible for two partitions, i.e. separate jobs running concurrently, to communicate. I don't believe they ever got it working. However, when they connected their machine to the IMP and got their host-host protocol software working...
>
>
> If anyone else had "done it first" I never heard about it. There was no
> network yet, so information about other projects was not readily
> available. Professors might have read journals, but hackers mostly
> wrote code.
>
> Most of the OS changes to ITS were done by the MIT AI Lab. But their
> focus was AI, and changes to the OS were often done to help with some AI
> project. In DM, we changed software as needed, mostly focussed on
> research on use of the network such as email.
>
> AI changed their PDP-10 hardware when that was useful, e.g., by adding a
> new instruction to ROTate memory in a counterclockwise direction, which
> was helpful to the AI Chess program. I recall someone at some point
> made some hardware changes (might have been on the DM machine) that
> enabled a program to be run "in reverse" for a bit. That was helpful
> in debugging to figure out how the OS code actually got to some weird
> place, e.g., in some data structure, where it crashed because data as
> "instructions" made little sense.
>
> ITS was a lot like Unix, in the sense that it was not an official
> project to research issues of operating systems. That was more Multics
> territory. ITS was just a tool to be used and modified as needed to
> help with the actual research topics of AI, DM, and later ML (MathLab)
> and MC (Macsyma Consortium).
>
>
> The MIT-AI lab, along with the AI labs at CMU, Stanford, BBN, et al produced some great systems work.
>
>
> The DEC field service techs used to hate coming to ITS land, but also
> liked it because they always learned something.
>
> Jack
>
>
> On 9/6/25 14:27, Brian E Carpenter wrote:
> > I've never looked into the early history of distributed file systems.
> > Was that work at MIT ever published? Was it pioneering or did someone
> > else do it first?
> >
> > My favourite paper in that area is the "Unix United" paper [1] from 1982.
> >
> > [1] https://doi.org/10.1002/spe.4380121206 <https://doi.org/10.1002/spe.4380121206> (paywalled) or
> > http://homepages.cs.ncl.ac.uk/brian.randell/Papers-Articles/399.pdf <http://homepages.cs.ncl.ac.uk/brian.randell/Papers-Articles/399.pdf>
> > Regards/Ngā mihi
> > Brian Carpenter
> >
> > On 07-Sep-25 08:04, Guy Almes via Internet-history wrote:
> >> Jack,
> >> Thanks very much.
> >> So this was in place by the mid-70s, right?
> >> -- Guy
> >>
> >> On 9/6/25 3:15 PM, Jack Haverty via Internet-history wrote:
> >>> ITS at MIT circa early 1970s used a naming convention for files --
> >>> <device>:<directory>;<name1> <name2> So, for example, I logged in to
> >>> MIT-DM as JFH. My files on disk were things like DSK:JFH;THESIS TJ6
> >>> File names were limited to alphanumerics of 6 characters or less
> >>> (motivated by what you could encode into a 36-bit PDP-10 memory
> >>> location).
> >>>
> >>> Once the ARPANET and NCPs appeared, the 'net was a new toy, so
> >>> people of
> >>> course experimented with how to use it. I don't remember the
> >>> details or
> >>> timing (sometime in early 1970s), but at but at some point the Message
> >>> Of The Day announced a new capability - you could use files on some
> >>> other ITS machine just by using a different <device> to specify the DSK
> >>> on some other ITS machine.
> >>>
> >>> So, for example, from the MIT-AI machine a user could get to my file on
> >>> the DM machine by specifying DM:JFH;THESIS TJ6.
> >>>
> >>> Similarly, from my account on MIT-DM, I could get to another machine's
> >>> files by using a name such as AI:TK;NEWS ITS to get at Tom Knight's
> >>> file
> >>> on the AI machine.
> >>>
> >>> This provided more flexibility than FTP. You could use a remote
> >>> file in
> >>> any program that knew how to use files on devices. To the program, the
> >>> remote disk looked and behaved like a local disk. (More or less -
> >>> problems of "global LANs" were still be be surfaced)
> >>>
> >>> I don't recall at all how this worked, or who implemented it. IIRC, it
> >>> took advantage of an interprocess communication capability called the
> >>> "JOB/BOJ device", which enabled one program to open a JOB device, and
> >>> another program to open the corresponding BOJ (JOB reversed) device,
> >>> and
> >>> send whatever they liked back and forth. But I don't remember details.
> >>>
> >>> We also had the ability for one process (aka "job") to map some or all
> >>> of another process' address space into its own address space. I can't
> >>> recall if anyone got motivated to get that working across the ARPANET
> >>> though. If so, it would probably have been done using the same
> >>> internal mechanisms that got the remote file systems capability.
> >>>
> >>> However, for anyone curious, the ancient ITS system is online and has
> >>> even been resurrected so you can look at the code or even run it on
> >>> your
> >>> modern computer - see https://github.com/PDP-10/its <https://github.com/PDP-10/its>
> >>>
> >>> Jack Haverty (JFH at MIT-DM in the 70s)
> >>>
> >>> On 9/6/25 09:28, Guy Almes via Internet-history wrote:
> >>>> Noel,
> >>>> So this was a real networked file system (and not just lots of
> >>>> FTP)?
> >>>> Very interesting,
> >>>> -- Guy
> >>>>
> >>>> On 9/6/25 11:35 AM, Noel Chiappa via Internet-history wrote:
> >>>>>
> >>>>> > From: Guy Almes
> >>>>>
> >>>>> > There are probably a number of ARPAnet sites where the
> >>>>> ARPAnet
> >>>>> > served this LAN role in the pre-Ethernet days.
> >>>>>
> >>>>> Notably MIT, where the 4 ITS machines shared their file systems
> >>>>> over the
> >>>>> ARPANET.
> >>>>>
> >>>>> Noel
> >>>>> --
> >>>>> Internet-history mailing list
> >>>>> Internet-history at elists.isoc.org <mailto:Internet-history at elists.isoc.org>
> >>>>> https://urldefense.com/v3/__https://elists.isoc.org/mailman/listinfo/ <https://urldefense.com/v3/__https://elists.isoc.org/mailman/listinfo/>
> >>>>> <https://urldefense.com/v3/__https://elists.isoc.org/mailman/listinfo/ <https://urldefense.com/v3/__https://elists.isoc.org/mailman/listinfo/>>
> >>>>> internet-history__;!!KwNVnqRv!
> >>>>> C8xpr0pcWUCRUGu5ny4SaIVDwdiMWrntxKhWopeJnt_Ni81FeTKeUj6hb30-W92d1QCtfI-
> >>>>>
> >>>>> NdmswzrIufFLHCtXfTXzJRA$
> >>>>> <https://urldefense.com/v3/__https://elists.isoc.org/mailman/listinfo/ <https://urldefense.com/v3/__https://elists.isoc.org/mailman/listinfo/>
> >>>>>
> >>> internet-history__;!!KwNVnqRv!
> >>> C8xpr0pcWUCRUGu5ny4SaIVDwdiMWrntxKhWopeJnt_Ni81FeTKeUj6hb30-W92d1QCtfI-
> >>> NdmswzrIufFLHCtXfTXzJRA$> >>
> >>> <https://urldefense.com/v3/__https://elists.isoc.org/mailman/listinfo/internet-history__;!!KwNVnqRv!C8xpr0pcWUCRUGu5ny4SaIVDwdiMWrntxKhWopeJnt_Ni81FeTKeUj6hb30-W92d1QCtfI-NdmswzrIufFLHCtXfTXzJRA$ <https://urldefense.com/v3/__https://elists.isoc.org/mailman/listinfo/internet-history__;!!KwNVnqRv!C8xpr0pcWUCRUGu5ny4SaIVDwdiMWrntxKhWopeJnt_Ni81FeTKeUj6hb30-W92d1QCtfI-NdmswzrIufFLHCtXfTXzJRA$>>
> >>>>>> -
> >>>>> Unsubscribe:https://urldefense.com/v3/__https://app.smartsheet.com/b/ <https://urldefense.com/v3/__https://app.smartsheet.com/b/>
> >>>>> <https://urldefense.com/v3/__https://app.smartsheet.com/b/ <https://urldefense.com/v3/__https://app.smartsheet.com/b/>>
> >>>>> form/9b6ef0621638436ab0a9b23cb0668b0b?
> >>>>> The*20list*20to*20be*20unsubscribed*20from=Internet-history__;JSUlJSU!!
> >>>>>
> >>>>> KwNVnqRv!C8xpr0pcWUCRUGu5ny4SaIVDwdiMWrntxKhWopeJnt_Ni81FeTKeUj6hb30-
> >>>>> W92d1QCtfI-NdmswzrIufFLHCtVmthkWew$
> >>>>> <https://urldefense.com/v3/__https://app.smartsheet.com/b/form/9b6ef0621638436ab0a9b23cb0668b0b?The*20list*20to*20be*20unsubscribed*20from=Internet-history__;JSUlJSU!!KwNVnqRv!C8xpr0pcWUCRUGu5ny4SaIVDwdiMWrntxKhWopeJnt_Ni81FeTKeUj6hb30-W92d1QCtfI-NdmswzrIufFLHCtVmthkWew$ <https://urldefense.com/v3/__https://app.smartsheet.com/b/form/9b6ef0621638436ab0a9b23cb0668b0b?The*20list*20to*20be*20unsubscribed*20from=Internet-history__;JSUlJSU!!KwNVnqRv!C8xpr0pcWUCRUGu5ny4SaIVDwdiMWrntxKhWopeJnt_Ni81FeTKeUj6hb30-W92d1QCtfI-NdmswzrIufFLHCtVmthkWew$>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>
>
> --
> Internet-history mailing list
> Internet-history at elists.isoc.org <mailto:Internet-history at elists.isoc.org>
> https://elists.isoc.org/mailman/listinfo/internet-history <https://elists.isoc.org/mailman/listinfo/internet-history>
> -
> Unsubscribe: https://app.smartsheet.com/b/form/9b6ef0621638436ab0a9b23cb0668b0b?The%20list%20to%20be%20unsubscribed%20from=Internet-history <https://app.smartsheet.com/b/form/9b6ef0621638436ab0a9b23cb0668b0b?The%20list%20to%20be%20unsubscribed%20from=Internet-history>
>
>
>
> --
> Sent by a Verified
>
> sender
More information about the Internet-history
mailing list