[ih] notable "bakeoffs" Re: internet-history Digest, Vol 84, Issue 4

[Dave Crocker]

On 5/21/2014 10:16 AM, Suzanne Woolf wrote:
> Occasionally I try to explain Internet history and processes to people outside of engineering culture. In that context, what we mean by "interoperability" and its role in usable standards is hard to explain, but keeps turning out to be important….


I would assume two kinds of difficulty, for folks not familiar with
interop testing:

   1. There's a spec; implement it.  The assumption would be that all
that's needed is to follow the spec.  Of course, this misses a) the
ambiguities of (all) specs, which produces individual interpretations
and hence individual variations and hence non-interoperability; and b)
the complexity of specs and the inevitable array of bugs.  Interop
testing isn't exhaustive, but it makes sure that the basic code paths
(inter-)work.

   2. Conformance testing is sufficient.  This assumes that an
independent test engine will suffice.  What it misses is the
impressively unpredictable interaction problems that can occur between
any two, independent implementations.  Conformance testing can be useful
for shaking out the 'normal' bugs, but never assures actual
interoperability.


There are two other benefits of interop testing that are easily missed:

   1. Cost.  Compared with more formal testing disciplines, interop
events can be remarkably inexpensive, especially given the level of
their efficacy.

   2. Community.  Especially for the early stages of a new technology,
an interop event usually turns the independent implementers into a
collaborative community.  I've tended to claim that it moves adoption
forward by at least 6 month, compared with only doing ad hoc testing.


d/
-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net