Subject: Re: Raising the bar on the MBone

From: [email-masked]
Date: Fri, 1 Aug 1997 16:42:32 -0700 (PDT)

I know I addressed this once before, so, pardon me, I apologize in 
advance.

Louis Mamakos said:
> Demand for what?  I suspect there's unlimited demand for free Internet
> connectivity, yet that's not compelling enough to build a network to
> offer free service.  There must be demand for a credible multicast-IP
> based service offering.

I really don't want to let this part of the discussion die.
Consider that N receivers want to receive a particular datastream.
They have a choice of receiving it unicast or multicast.

In the unicast case, the source has to send N streams to each
of the N receivers.  The ISPs connection to the source has
to carry N copies, and the subsequent links have to carry
some fractions thereof as the paths diverge.  

Now in the multicast case, the ISP has only to carry one stream
from the source, an increasing number of streams up to the
final destinations.  Now, in no case does the link ever have
to carry more than one copy.

Therefore, for a given set of receivers, the ISPs WAN traffic can 
only be reduced, sometimes dramatically for large numbers of receivers.  
Therefore, in a receiver-pays-for-connection-bandwidth pricing model, 
the ISP gets a "bonus" of reduced internal traffic for the same 
delivered traffic.  Therefore, in this model, multicast would seem 
to be a Good Thing for the ISP.

So, I must assume that some other pricing model is being assumed
by those who assign large unrecovered costs from providing
multicast service.  I would really like to understand what
assumptions about costs/pricing are being made.

Or, the other possibility is that all types of multimedia 
are troubling, but in that case, I wonder what the reaction
is to unicast audio/video streaming, or, the Web, which
also increases (often gratuitous) back-back transmissions
of image data, etc.  If the problem is that the peak/average
traffic ratio is being reduced (that is, the average traffic
is a much bigger percentage of peak than it used to be), 
I'm afraid unicast got there first.

So, it would certainly be helpful to me (personally) if some
discussion from ISPs took place regarding what they are
expecting in the future wrt costs and pricing.

-Hugh LaMaster

  Hugh LaMaster, M/S 258-5,     ASCII Email:  [email-masked]
  NASA Ames Research Center     Or:           [email-masked]
  Moffett Field, CA 94035-1000  No Junkmail:  USC 18 section 2701
  Phone:  415/604-1056          Disclaimer:   Unofficial, personal *opinion*.

---

From: "John M. Zwiebel" <[email-masked]>
Date: Fri, 01 Aug 1997 19:29:06 -0700

Hugh said:    

    

Everyone knows t
is.  Everyone accepts this.

Some things that I think keep getting ignored:

  --the additional state that the routers may
	have to carry depending on the protocol being used.

  --if the ISP's use closest exit routing, one ISP that has a lot of 
	sources can cause other ISPs to carry all of the multicast traffic.
	If the only way to charge for multicast is that the source pays,
	those ISP's with no sources get stuck with no income and a lot of
	traffic.  Sources pay for it with advertising.  How do receivers
	pay?

  --Current routing protocols are flat, no heirarchy.  Lot's of routes
	use lots of memory.  No BGP.

  --There aren't good ways of setting policy for who carries what.  No
	BGP for multicast.

  --There is little experience with multicasting over a large area.  The
	MBONE isn't a large area because its all tunnels.  (yes it covers
	the whole globe, so what.)

  --current protocols don't scale to cover the entire world.
	PIM densemode and DVMRP periodically flood everywhere even if they
	prune off quickly, which sometimes doesn't happen, there is a
	significant use of bandwidth in getting things to a stable condition.
	Where do you propose placing the RP for PIM sparsemode?  Can you
	imagine the dykstra that MOSPF would have to calculate?  Every router
	would have to be a cray.

  --There are a lot of mrouted machines running on unix boxes.  They can't
	handle all the routes in the internet.  How do you propose keeping
	these systems up and running?  

At first glance, one only sees the advantages of using multicast.  Its the
details that hurt.

This is just an opinion, but if I hold my breath till I'm blue (ie, if 
I constantly demand, which seems to be the operative word here) I'm not
going to get to ride in the space shuttle.  So, users can demand multicast
on the internet all they want, it isn't going to happen until the tools arrive.

They are coming.

------------------------------------------------------------------------------
	John Zwiebel                       Phone: 408-526-5303
	Cisco Systems Inc.                 
	Multimedia Group                   

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

          Never ascribe to malice what can be explained by incompetence.

				-- Paul Traina

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

---

From: "Heiko W. Rupp" <[email-masked]>
Date: Sat, 2 Aug 1997 12:05:20 +0200

> only be reduced, sometimes dramatically for large numbers of receivers.  

Or for applications that need large masses of bandwidth.
Suppose you want to transfer very huge files to three sites behind a 
specific link. With a file size of say 2MB you can saturate a T1
for about eleven seconds. With three receivers and unicast for 30 secs.
I think this already pays here.

> the ISP gets a "bonus" of reduced internal traffic for the same 

Not only as in the above mentionned case the need for larger pipes
can be delayed.

-- 
NTG Netzwerk und Telematic GmbH   /          Heiko W.Rupp
Geschaeftsbereich Xlink         \/           phone: +49 721 9652 0
Vincenz-Priessnitz-Str. 3       /\ link      fax:   +49 721 9652 210
D-76131 Karlsruhe, Germany     /_______      email: [email-masked]
[ NTG Netzwerk und Telematic GmbH, Sitz Chemnitz                          ]
[ Kreisgericht Chemnitz/Stadt HRB 4217, Geschaeftsfuehrer: Michael Rotert ]

---

From: Martin Hamilton <[email-masked]>
Date: Sat, 02 Aug 1997 22:05:23 +0100

-----BEGIN PGP SIGNED MESSAGE-----

Hmm...  Everyone seems so hung up on audio and video :-)

What about multicast based News distribution ?  Now that INN is a
going concern again, perhaps it would be possible to get the code that
people have been writing rolled into the core distribution...

We're starting to connect WWW proxy caches together in peer groups via
multicast ICP.  That's part of the market leading 'product' - Squid.

And how about a drive to get major WWW content providers to replicate
their pages by multicasting the content (or change notices) out to
replica servers ?

Just a thought, or two...

Cheerio,

Martin

---

From: Luca Bertagnolio <[email-masked]>
Date: Sun, 03 Aug 1997 10:25:22 +0200

>Hmm...  Everyone seems so hung up on audio and video :-)

Agree.  I am thinking also of the possible benefits of turning push
technlogies like PointCast to multicast, at least for distribution over the
higly overpriced WAN links (well, overpriced here in Europe!), with some
local caching for the dialup users that can't be forced to wait online
until their requested info comes in via multicast.

Ciao, Luca
Milan, Italy

---

From: "Heiko W. Rupp" <[email-masked]>
Date: Sun, 3 Aug 1997 15:15:23 +0200

> people have been writing rolled into the core distribution...

We are doing this just here on our backbone .. Software will be available
real soon now.

  Heiko

---

From: [email-masked] (Ran Atkinson)
Date: Mon, 4 Aug 1997 09:36:51 -0700

John Zwiebel's list makes a fine starting point for a discussion
of the issues with widespread deployment of IP multicasting.

Another item that is a significant concern to some of us is:
	congestion avoidance

When those N users each grab their content via Pointcast (or
equivalent), they generally are using TCP.  TCP has well understood
and good quality congestion avoidance properties.

TCP does not currently work over IP multicast, so multicast apps
use UDP.  UDP does not have any congestion avoidance properties.

So one legitimate concern is that a significant increase in use of
IP multicast applications could cause overall network throughput
to degrade because mcast apps generally do not automatically back off
in the presence of congestion.

This topic is one being taken up within the IRTF Reliable Multicast
RG, but is a research issue at this time IMHO.

Ran
[email-masked]

---

From: [email-masked] (Ran Atkinson)
Date: Mon, 4 Aug 1997 10:18:12 -0700

I am among those who think that running NNTP/UDP/IP Multicast
is interesting.  Some form of web content push via IP Multicast
could also be interesting.

Others interested in this topic should read the USENIX paper below:
      Author: Kurt J. Lidl
      Author: Josh Osborne
      Author: Joe Malcolm
       Title: Drinking from the Firehose: Multicast USENET News
       Pages: 33-45
   Publisher: USENIX
 Proceedings: USENIX Conference Proceedings
        Date: Winter 1994
    Location: San Francisco, CA
 Institution: UUNET Technologies, Inc.

I believe a copy of the paper is online at ftp.uu.net (along with
their software), but I do not recall the URL.

Ran
[email-masked]

---

From: Steve Deering <[email-masked]>
Date: Mon, 4 Aug 1997 10:28:57 -0700

> use UDP.  UDP does not have any congestion avoidance properties.

This is orthogonal to multicast -- the same issue arises with unicast UDP
apps, or with hacked or misused TCPs.  What is required is mechanisms in
routers (such as statistical approximation of fair queueing, or RED-with-
penalty-box) to prevent non-congestion-sensitive flows from hogging links.
We are no longer in a world where we can trust that all IP sources will be
"well- behaved", not even TCP sources.  And this has nothing to do with
multicast.

Steve

---

From: "Heiko W. Rupp" <[email-masked]>
Date: Mon, 4 Aug 1997 19:36:10 +0200

> Others interested in this topic should read the USENIX paper below:

I think their code is not completly available.

Please also refer to

http://www-old.xlink.net/~hwr/jenc8.ps
and 
draft-rfced-rupp-exp-02.txt (from http://ds.internic.net/internet-drafts/)

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 4 Aug 1997 20:15:23 +0200 (MET DST)

Ran Atkinson writes:
> is interesting.

A research note that I wrote in October 1995 about a workable system
can be found at http://user.cs.tu-berlin.de/~nilss/som/newscast.htm

This system has been in actual use since April 1996, although its
deployment has been hindered by the lack of a portable version of the
underlying multicast protocol MTP-2 (the participants of the
experiment need to run a Solaris box right now), problems with channel
feeds in INN 1.5, and, more importantly, by certain heisenbugs in a
widely deployed implementation of one multicast routing protocol.

Wide deployment of the newscaster system would have to wait for
implementation of the security system that is hinted at in the
research note.  If you want to play with the code or take part in the
experiment anyway, contact me.

Gruesse, Carsten

---

From: [email-masked] (Ran Atkinson)
Date: Mon, 4 Aug 1997 11:17:59 -0700

> use UDP.  UDP does not have any congestion avoidance properties.

% This is orthogonal to multicast -- the same issue arises with
% unicast UDP apps, or with hacked or misused TCPs.

	In a non-operational abstract sense, they could be viewed
as orthogonal.

	However, in an operational pragmatic sense, they are not
orthogonal.  There are very few unicast UDP applications in use,
even though they require no special support from the infrastructure.
The most commonly used applications (e.g. http, ftp, telnet) use TCP
and consequently have reasonable congestion avoidance properties.
I actually have a Very Good idea of what the traffic looks like on
our network, as it happens.  It is possible that other networks
have different characteristics, though other operationally-minded
folks seem to have similar perceptions.

	A "hacked TCP" is broken, pure and simple.  However, the
majority of the TCP implementations that are deployed _do_ implement
slow-start and exhibit reasonable (if not ideal) congestion avoidance
properties.  When I read Vern Paxson's recent paper on TCP
implementations, I did take the time to look into the problematic
code he identified.  Thanks in part to his efforts, it turns out that
the deployed state of things in TCP-land remains pretty good just
at the moment.

% What is required is mechanisms in routers (such as statistical
% approximation of fair queueing, or RED-with-penalty-box) to prevent
% non-congestion-sensitive flows from hogging links.

	When those features are available in a stable router release,
I'm sure that ISPs will deploy them with glee.  Until then,
we are where we are.

	If one wants native IP multicast to be more widely deployed,
one might reasonably spend effort working on making the above mechanisms
more widely available to ISPs and other network operators.  Of course,
folks who build routers are in a better than average position to
influence things in this regard.

% We are no longer in a world where we can trust that all IP sources
% will be "well-behaved", not even TCP sources.

	Perhaps.

	However, for the present, TCP-based sources are behaving,
while a number of deployed IP multicast applications lack any
form of congestion avoidance and do offer excessive bandwidth load
under congested conditions.

	The multicast games tend to be the worst offenders, perhaps
because the vat/vic/wb users have largely been educated to rate-limit
their offered load and the vat/vic/wb implementers have reasonable
default settings.

Regards,

Ran
[email-masked]

---

From: [email-masked] (Ran Atkinson)
Date: Mon, 4 Aug 1997 11:26:19 -0700

Heiko Rupp is mistaken.  The UUNET code is freely available.
It isn't perfect, but it is a fine starting point.  No,
I had nothing to do with that code or their work.

Moreover, Heiko's code is a fine example of the kind of multicast
application that lacks congestion avoidance properties and
is dangerous to the overall network health if widely deployed.
His approach (and document) has other serious issues (e.g. security),
but that isn't really a good topic for this mailing list.

I suspect that the IETF will have a BOF on the topic of USENET
over IP Multicast this winter.  In the mean time, folks interested
in this topic should probably read the Lidl, et. al. paper for
additional background.

Ran
[email-masked]

---

From: [email-masked]
Date: Mon, 04 Aug 1997 21:08:53 +0200

> and consequently have reasonable congestion avoidance properties.

Some of the UDP applications in use are rather important (eg. DNS).
However DNS has rather special characteristics (short transactions/
flows).

Another important UDP application is NFS - but as far as I know it's
not much used in WANs. Also, some NFS/UDP implementations have
congestion avoidance properties similar to TCP.

Steinar Haug, Nethelp consulting, [email-masked]

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 4 Aug 1997 21:15:18 +0200 (MET DST)

> congestion avoidance properties similar to TCP.

That's not the point. It's a fundamental problem: As the Internet
approaches ubiquoutous usage you cannot imply the well behavior of
end systems anymore. It doesn't help to know umpteens of well behaved
examples anymore.

Toerless

---

From: Jamshid Mahdavi <[email-masked]>
Date: 04 Aug 1997 15:17:35 -0400

> "well- behaved", not even TCP sources.  And this has nothing to do with

Perhaps not entirely.  In the unicast world, congestion control
algorithms are fairly well understood, and it is just the
{naivity,ill-will} of implementers of UDP applications (and wayward
TCPs) which causes these applications to run without congestion
control.

In the multicast world, techniques for congestion control are still an
open research question.  I wouldn't expect multicast application
developers to "do the right thing" at this stage of the game, because
the "right thing" is still being figured out in the IRTF.

So that no one misinterprets me:

1)  Unicast applications without congestion control are bad.
Application programmers who write them should be sent to bed without
their dinner.

2)  Router mechanisms for detecting "evil applications" are good and
are clearly needed, because some programmers don't mind going to bed
without their dinner.

--Jamshid

---

From: "Heiko W. Rupp" <[email-masked]>
Date: Mon, 4 Aug 1997 21:24:14 +0200

> Heiko Rupp is mistaken.  The UUNET code is freely available.

Hu? Lasttime I checked, the sender was not. Ok.

> application that lacks congestion avoidance properties and

But this does the uunet code also do.

> His approach (and document) has other serious issues (e.g. security),

This is - if you look at it - resolved in the actual version of
the draft. And I can't add this now to a paper that was submitted in
february (the jenc one).

> over IP Multicast this winter.  In the mean time, folks interested

*sigh*  If there is anyone interested, we could aready meet for a pizza
in Munich

---

From: [email-masked] (Ran Atkinson)
Date: Mon, 4 Aug 1997 12:48:42 -0700

On Aug 4 21:08, <[email-masked]> wrote:

% Some of the UDP applications in use are rather important (eg. DNS).
% However DNS has rather special characteristics (short transactions/
% flows).

Absolutely.  Hence DNS flows don't represent an operational issue
wrt congestion avoidance.

% Another important UDP application is NFS - but as far as I know it's
% not much used in WANs. Also, some NFS/UDP implementations have
% congestion avoidance properties similar to TCP.

	True.

	Moreover, modern NFS implementations (NFSv3 and later)
support NFS/TCP.  Because NFS/TCP works much better than NFS/UDP
over wide-area links, folks using NFS in the wide area tend to
use NFS/TCP anyway.

Ran

---

From: [email-masked] (Ran Atkinson)
Date: Mon, 4 Aug 1997 12:58:46 -0700

> application that lacks congestion avoidance properties and

% But this does the uunet code also do.

	Actually, the Lidl code is less dangerous than yours
with respect to congestion avoidance.  Its not perfect, but
it is less dangerous.  Further, Kurt Lidl has been open and willing
to listen to other inputs on evolving their code.

% This is - if you look at it - resolved in the actual version of
% the draft.

	I disagree, and I am not alone in that view.

	I *HAVE* been following your drafts -- as my unicast email to you
along the way should have made clear to you.  Since you've sent some
private replies to me, I have reason to believe that you knew a priori
that I have been reading your drafts.

Ran

---

From: Steve Deering <[email-masked]>
Date: Mon, 4 Aug 1997 14:58:57 -0700

> even though they require no special support from the infrastructure.

Are all of the Internet Telephony apps and unicast video apps built on
TCP rather than UDP?  (That's a question, not a challenge -- I don't know
the answer myself.)

> properties.

I keep hearing about mailing lists on which PC users tell each other how
to twiddle their TCP parameters to "improve throughput" (essentially by
making them more aggressive than "stock" TCP).  But maybe that's just an
urban legend in the end-to-end community.  There certainly is the
Netscape example of opening multiple concurrent TCP connections to get
around TCP's per-connection congestion control.

> influence things in this regard.

I agree (with both of those observations).

Steve

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 5 Aug 1997 00:44:34 -0700

>> to be a Good Thing for the ISP.

    This assumes there is no fanout in that ISP. But if there is significant
    fanout in the ISP, there have been arguments that indicate it may be
    no better than unicast.

Dino

---

From: Jon Crowcroft <[email-masked]>
Date: Tue, 05 Aug 1997 08:34:28 +0100

>RG, but is a research issue at this time IMHO.

so is TCP congestion control - 

but we actually have at least 3 fairly reasoanble congestion control
mechanisms for multicast
1/ the rate limit in the mbone
2/ use lorenzo's scheme
3/ some of the apps have source based rate limiters.....

while these don't _adapt_ as well as TCP yet, if they are capped at
some percentage of your net, you can make a policy decision based in
the capacirty you save from them being multicast, for what cap to
set....

so i don't agre that it is much more a _research_ issue than good TCP
congestion control
(e.g. on mobile/wireless, or on very hi basndwidth links or with
explicit congestion indication) is...

what we don't have is a widely used application with working code and
consensus, and an  RFC/IETF standard (in that order) which i think we
will see in the next 2 years...

what level of assurance do people want for congestion controlled (i.e.
adaptive tcp-friendly) multicast schemes? do you want the "Van" stamp
approval, or a Rich Steven's RFC or what?

what is the metric for acceptable multicast source (or receiver based
flow) behaviour?
 jon

osal doesn't seek to connect 
up everybody with a PC to "the Mbone" today.  Apparently
you don't think anything less than 10^6 households is
an "interesting scale".  But, for some purposes, 1000
corporate and educational sites with T1's-T3's could 
also be "interesting".  

"The Mbone" is an experiment, and, as an experiment,
it works.  The next step is to go on to expand the base
of "early adopters".  In the meantime, roadblocks that 
prevent multicast from being ubiquitously deployed will
have to be eliminated.  Automobiles and telephones were 
frivolously expensive for ordinary people, when they were 
first deployed, too, and it wasn't really until the 
mid-1980's, 100 years after the development of the
internal-combustion-engine-automobile, that cars became
reasonably pollution-free, non-balky, mostly trouble-free,
relatively safe "appliances".  Computers are now 50 years
into their development, and IP-based networks about 16 years.
The pace of development has accelerated considerably since 
1885, but, you still don't normally expect any new technology 
to go from "experimental" stage to the fully-developed state
overnight.  In this case, we have a successful experiment,
"The Mbone", and some technology suitable for "early adoption".  
Ubiquitous deployment will come later.

> Now what was it that is so attractive about it??

More efficient use of WANs.  The ability to provide
real-time information to large numbers of users 
simultaneously.

>         -mo
> 
> PS - this does not mean the technology cannot be salvaged for
> some useful purpose, but it's not exactly what people seem
> to have in mind.

I'm not sure what you think I have in mind.  However, I'm not
afraid to be explicit about it.  I think that eventually the 
unicast routing world and the multicast routing world can be 
made to coincide mostly, and that to do so will enable more 
efficient use of WANs.

I'm not under the delusion that all the problems have already
been solved.  There are lots of interesting problems to be
solved, although, obviously, I think the technology is more 
mature than you do.

In a subsequent post, "Mike O'Dell" <[email-masked]> writes:

> Ah yes,  you have come to the nub....
> 
> We don't talk about unannounced products
>
>         cheers,
>         -mo

If the word is "products", perhaps you think that some
part of the experiment was a success after all?

-Hugh LaMaster

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 5 Aug 1997 00:44:34 -0700

Dino

---

From: Jon Crowcroft <[email-masked]>
Date: Tue, 05 Aug 1997 08:34:28 +0100