[ih] low latency traffic (was UDP Length Field?)

[John Gilmore]

Jack Haverty via Internet-history <internet-history at elists.isoc.org> wrote:
> From what I can anecdotally see today, 40 years later, low-latency
> datagram service on the Internet is not on anyone's radar.  I helped a
> friend investigate his attempts to use a gaming-type app over the
> Internet last year, and our experiments discovered that packet loss
> rate was surprisingly (to me) measured at 0%, latency was on average
> in the hundreds of milliseconds, but had "tails" of data points out to
> 30 seconds.  The Internet "IP datagram service" today seems to be very
> connection-oriented, delivering every packet but with noticeable very
> long delays.  I suspect this may be a cause of the anomalies we often
> see today in TV interviews conducted using the Internet.

Actually, there is a very active and somewhat successful effort (active
queue management) to improve Internet latency.

Given technological gains in semiconductor speed and power, and optical
fibers, the Internet industry quickly learned that it was cheaper to buy
higher bandwidth than it was to try to prioritize categories of traffic.
This also helped with latency since faster networks tend to have lower
latencies as well as higher overall thruput.  It also offered
significant "headroom" in deployed networks, which allowed the Internet
to cope smoothly when e.g. the whole world suddenly started using 30-40%
more realtime Internet traffic during the Covid shutdowns.

A countervailing trend was caused by TCP's insistence on increasing its
offered load until a packet is actually dropped, combined with the
decreasing cost of RAM buffering throughout the network.  It was
inconceivable for 1980s routers to have more than a few packets' worth
of buffering, so back then, packets dropped immediately after a flow was
not sustainable.  TCP's flow control was designed to respond to that
signal.  Now routers and even network interface chips can have many
seconds' worth of RAM, so 10 to 20 years of designs caused packets to be
queued rather than dropped.  The resulting interaction is called
Bufferbloat.  It was publicized by Jim Gettys, who investigated it for
One Laptop Per Child.  It particularly happens at pinch points like
wireless gateways, where high speed offered traffic backs up trying to
get into a low speed link.  The queue of pending traffic clobbers the
latency of every packet that follows, and TCP persists in continuing to
offer more traffic to top up the queue whenever it shrinks.  This is
easily visible when doing a large bulk upload from a home machine, such
as a remote backup, simultaneous with a latency-sensitive download.
This causes the downstream traffic to significantly stutter, because its
acks suffer high latency behind the queue of upstream bulk packets.

Significant work by Van Jacobson and Kathleen Nichols produced a
"controlled delay" algorithm, CoDel, that could generally, reliably, and
cheaply condition a router's queues without manual tuning.  More years
of work, largely by Dave Taht, produced, tested and deployed a Linux
kernel implementation, fq_codel.  That has been deployed into millions
of end user access points via the OpenWRT ecosystem, and into servers
via ordinary Linux distributions.  The problem is not "solved", but it
can usually be ameliorated by upgrading local networking routers or
firmware.

Interestingly, BitTorrent earlier went through a similar evolution and
came up with a different amelioration.  When it became popular, ISPs
complained that Torrent users were eating all their bandwidth.  Actually
what was happening is that BT nodes were opening simultaneous TCP
connections to many sites, and feeding all of them at whatever rate TCP
would allow.  Latency for everyone suffered at the in-ISP pinch points.
Once BT designer Bram Cohen realized that TCP's non-delay-sensitive flow
control was the issue, he replaced it with a UDP-based protocol that
handled its own flow control in a latency-sensitive manner, bypassing
TCP and its bugs entirely (without changing the user interface at all).
ISPs rapidly stopped complaining, and BitTorrent is now accepted as a
useful and non-disruptive network application.

Historically, TCP has mostly been tuned only when its performance was
intolerable, such as in 1986 on the 56k NSFnet before its late 1988
upgrade to T1 links, and again before the 1991 T3 upgrades.  The pace of
backbone bandwidth improvement since then has prevented much further
tuning for decades.

I believe the ultimate cure for high latency would be to fix endpoint
TCP implementations to be latency-sensitive and not just packet-drop
sensitive.  But that effort has not succeeded so far, because the
resulting implementations produce lower thruput when facing competing
traffic from the old algorithm that floods its own traffic ahead of them
in router queues.  It's a "nice guys finish last" tragedy of the commons
problem that causes major implementers to not switch to the "slower"
algorithm.  This is why bufferbloat has had to be fixed with CoDel in
middleboxes, rather than in TCP at the endpoints, somewhat damaging the
"dumb network core" principle that has allowed the Internet to scale.
There were hints of this in the early Router Requirements RFCs that said
routers forced to drop packets should punish end nodes that were not
backing off their offered load, such as section 5.3.6 of RFC 1812 in
1995, but this was still very much a research topic back then (as it is
today).

See:

  https://en.wikipedia.org/wiki/Bufferbloat
  https://en.wikipedia.org/wiki/CoDel

	John