Subject: packet loss between mbone2.UU.Net & SanFrancisco.mci.net

From: Van Jacobson <[email-masked]>
Date: Mon, 10 Feb 97 14:37:28 PST

Together with about 50 others, I'm trying to listen to the San
Francisco NANOG meeting.  There's fairly high loss on both audio
& video (~20%).  The attached mtrace says all the loss is
happening between mbone2.uu.net and dec3800-2-fddi-0.SanFrancisco.mci.net.
Since mbone2.uu.net is a cisco with 26 mbone tunnels, it seems
reasonable that it would drop lots of packets whenever there's
non-trivial load on the mbone but perhaps the problem is
somewhere else.

I guess that problems like this are only going to get worse
(over the past 2 months, the mean packet rate through the
backbone mrouters has gone from 120 p/s to over 250 p/s)
since every college in the country seems to think it's very cool to
broadcast their local radio station on the mbone.  Today I count
at least 3 of them active -- CMU, University of Missouri-Kansas
City & University of Missouri-Columbia.  At a time when there's
new, original, content starting to appear on the mbone, it's a
shame to have the bandwidth taken away by yet another low
quality rendition of CDs anyone can buy at the corner store.  Oh
well, perhaps one day the parties involved will realize they're
not actually demonstrating a very high level of coolness.

 - Van

-----------------

Mtrace from 206.171.148.21 to 131.243.1.67 via group 224.2.251.171
Querying full reverse path... 
  0  vs.ee.lbl.gov (131.243.1.67)
 -1  ir40gw.lbl.gov (131.243.1.1)  DVMRP  thresh^ 0  
 -2  mr1.lbl.gov (131.243.64.184)  DVMRP  thresh^ 1  
 -3  lbl-mr1-f.es.net (198.128.16.13)  DVMRP  thresh^ 48  
 -4  llnl-mr2.es.net (134.55.12.229)  DVMRP  thresh^ 8  
 -5  mbone.nsi.nasa.gov (192.203.230.241)  DVMRP  thresh^ 64  
 -6  dec3800-2-fddi-0.SanFrancisco.mci.net (204.70.158.61)  DVMRP  thresh^ 64  
 -7  MBONE2.UU.NET (137.39.246.98)  DVMRP  thresh^ 64  
Round trip time 201 ms; total ttl of 67 required.

Waiting to accumulate statistics... Results after 9 seconds:

  Source        Response Dest    Overall     Packet Statistics For Traffic From
   * * *        224.0.1.32       Packet      206.171.148.21 To 224.2.251.171
     v       __/  rtt  190 ms     Rate       Lost/Sent = Pct  Rate
137.39.2.254   
137.39.246.98   MBONE2.UU.NET  
     v     ^      ttl   65       225 pps       53/251  = 21%  27 pps
204.70.158.61   dec3800-2-fddi-0.SanFrancisco.mci.net 
     v     ^      ttl   66        96 pps        0/198  =  0%  22 pps
192.203.230.241 mbone.nsi.nasa.gov 
     v     ^      ttl   67       259 pps        0/198  =  0%  22 pps
134.55.12.229   llnl-mr2.es.net 
     v     ^      ttl   67       254 pps        5/198  =  3%  22 pps
134.55.12.101  
198.128.16.13   lbl-mr1-f.es.net 
     v     ^      ttl   67       253 pps       -3/193  = -1%  21 pps
131.243.64.184  mr1.lbl.gov    
     v     ^      ttl   67       251 pps       -2/196  =  0%  21 pps
131.243.64.1   
131.243.1.1     ir40gw.lbl.gov 
     v      \__   ttl   67       253 pps        ?/198         22 pps
131.243.1.67    131.243.1.67
  Receiver      Query Source

---

From: "Paul 'Shag' Walmsley" <[email-masked]>
Date: Mon, 10 Feb 1997 20:35:07 -0600 (CST)

As I am one of the individuals who is currently multicasting one of the
radio stations that Van mentioned (KCOU), I think that some discussion on
this topic will be at the worst informative and at best entertaining.  

On Mon, 10 Feb 1997, Van Jacobson wrote:

> & video (~20%).  

The obvious implication (that this is a bad thing) is something that I
completely agree with.

>                  The attached mtrace says all the loss is

.. and you (apparently) have found the cause: a multicast router that is
supporting too many tunnels.  (Although this may not in fact be the cause
of the loss at that point, since as of 01:27 11 Feb 97 UTC, 13 of those 26
tunnels are down.  Still, a likely culprit.) 

> I guess that problems like this are only going to get worse

As a separate assertion, this is certainly likely, since more and more
sites are attaching to the MBONE, however ...

> City & University of Missouri-Columbia.  

... this is certainly contradictory ("every college in the country" != 3),
but it also seems potentially misleading.  I think it is misleading since
there seem to be other much more serious problems confronting the MBONE
other than college radio stations, like, for example, the existence of
multicast routers with tunnels that they cannot support during
"non-trivial loads" without lossage; or the fact that there are still
non-pruning multicast routers attached to the MBONE; or perhaps the
continuing existence of duplicate tunnels crossing the same links.

(Re non-pruning MBONE routers: unfortunately, Piete Brooks' fine web site
<URL:http://www.cl.cam.ac.uk/mbone/>, sporting, among other fine
information, a version histogram of the current MBONE routers, is
apparently malfunctioning right now; otherwise, I would quote mrouter
version statistics.)

Ultimately, if the MBONE gains widespread adoption, there will be lots of
content on the MBONE, certainly much more than there is now.  Some of this
content you may like; other content you may not like.  In the absence of
either (practically) unlimited bandwidth or resource reservation and
per-packet billing, there is very little beyond perhaps common courtesy
and pruning that will prevent other content from potentially trashing
something that you do want to listen to.  (Of course this presumes that
pruning is even working correctly in a global sense, which doesn't seem to
be true of the current state of the MBONE.) 

Whether I stop or continue sending KCOU is then really nothing more than a
short-term issue.  As with the World Wide Web, I think it likely that the
MBONE's popularity will grow beyond its rather select group of
scientific/research institutional sites to include most of the Internet at
large; and if the MBONE is not prepared for the increase in usage that
this will certainly involve, then perhaps we should re-evaluate what
should be done to make it ready for widespread use.

>                                                 At a time when there's
> quality rendition of CDs anyone can buy at the corner store.  

As an aside, some of the content being multicast by college radio stations
cannot be found in corner record stores.  (But this isn't even really the
issue.)

>                                                               Oh

Yo, man, eat my shorts.

I should conclude this discussion with the reiteration that I (as always) 
am completely willing to discontinue multicasts -- no matter whether they
are seminars, or college radio stations -- that others feel are causing
lossage on the MBONE.  However, it would be nice to see some genuine
evidence that something about those streams is the _real_ problem, rather
than merely some network administrator at UUnet overcommitting their
multicast router(s). 

- Paul "Shag" Walmsley <[email-masked]>
  "Knowing is not enough." -- Hal Hartley, "Surviving Desire"

---

From: Bill Fenner <[email-masked]>
Date: Mon, 10 Feb 1997 22:15:49 PST

In message <[email-masked]> you write:
>mtrace says all the loss is

I saw that too, and logged onto dec3800-2-fddi-0.SanFrancisco.mci.net .
I ping'd mbone2.uu.net, and saw responses with very high RTT variance and
about 22% loss, which jibes pretty well with there just being very bad
congestion (I think).

  Bill

---

From: John Hawkinson <[email-masked]>
Date: Tue, 11 Feb 1997 02:34:10 -0500 (EST)

> about 22% loss, which jibes pretty well with there just being very bad

OK, can we get mci/uunet to troubleshoot said unicast congestion
and down this tunnel in the meantime [there are alternate
paths...]

--jhawk

---

From: "William Griffin" <[email-masked]>
Date: Tue, 11 Feb 1997 04:19:38 -0500

John, I just logged in to dec3800-2-fddi-0.SanFrancisco.mci.net and pinged mbone2.uu.net and am not seeing any slow response's:

root@dec2# ping mbone2.uu.net                                               [/]
PING mbone2.uu.net (137.39.246.98): 56 data bytes
64 bytes from 137.39.246.98: icmp_seq=0 ttl=249 time=7 ms
64 bytes from 137.39.246.98: icmp_seq=1 ttl=249 time=10 ms
64 bytes from 137.39.246.98: icmp_seq=2 ttl=249 time=7 ms
64 bytes from 137.39.246.98: icmp_seq=3 ttl=249 time=6 ms
64 bytes from 137.39.246.98: icmp_seq=4 ttl=249 time=7 ms
64 bytes from 137.39.246.98: icmp_seq=5 ttl=249 time=9 ms
64 bytes from 137.39.246.98: icmp_seq=6 ttl=249 time=6 ms
64 bytes from 137.39.246.98: icmp_seq=7 ttl=249 time=6 ms
64 bytes from 137.39.246.98: icmp_seq=8 ttl=249 time=12 ms
64 bytes from 137.39.246.98: icmp_seq=9 ttl=249 time=11 ms
64 bytes from 137.39.246.98: icmp_seq=10 ttl=249 time=8 ms
64 bytes from 137.39.246.98: icmp_seq=11 ttl=249 time=9 ms
64 bytes from 137.39.246.98: icmp_seq=12 ttl=249 time=28 ms
64 bytes from 137.39.246.98: icmp_seq=13 ttl=249 time=11 ms
64 bytes from 137.39.246.98: icmp_seq=14 ttl=249 time=11 ms
64 bytes from 137.39.246.98: icmp_seq=15 ttl=249 time=7 ms
64 bytes from 137.39.246.98: icmp_seq=16 ttl=249 time=6 ms
64 bytes from 137.39.246.98: icmp_seq=17 ttl=249 time=8 ms
64 bytes from 137.39.246.98: icmp_seq=18 ttl=249 time=7 ms
64 bytes from 137.39.246.98: icmp_seq=19 ttl=249 time=13 ms
64 bytes from 137.39.246.98: icmp_seq=20 ttl=249 time=13 ms
64 bytes from 137.39.246.98: icmp_seq=21 ttl=249 time=6 ms

----mbone2.uu.net PING Statistics----
22 packets transmitted, 22 packets received, 0% packet loss
round-trip (ms)  min/avg/max = 6/9/28 ms

And the tunnel is currently up.

204.70.158.61 -> 137.39.246.98 (MBONE2.UU.NET) [1/64/tunnel/leaf]

Does the problem still exist? Please let us know.

Thanks

BT  iMCI/MBONE

---

From: "Dorian R. Kim" <[email-masked]>
Date: Tue, 11 Feb 1997 04:57:52 -0500 (EST)

> not actually demonstrating a very high level of coolness.

This seems a bit backwards to me. Multicast infrastructure should be able to
accomodate both. If it doesn't than it needs to be fixed so that it can. I
don't see why the people who've tkaen the trouble to put content on mbone
should be chided for doing so.

This sort of arguments to keep our old toy(boy, I guess I must be in a
masochistic mood today), velveteen Mbone, going the way it is probably won't
ever help make it be more than just a toy.

Oh well, I guess the parties involved can just go ahead and set up streaming
unicast servers.

-dorian, who considers college radio stations a more "interesting" content
than NANOG..

---

From: Petri Helenius <[email-masked]>
Date: Tue, 11 Feb 1997 12:19:04 +0200 (EET)

> should be chided for doing so.

But remember that MBONE IS NOT READY.

:-)

 > ever help make it be more than just a toy.
And remember that many of the people don't want the MBone to be ready.
(and many, like you and I do)

 > Oh well, I guess the parties involved can just go ahead and set up streaming
That is the politically correct way to do. You buy enough T3's to push
traffic out with as many copies as you like. Simple and easy and IT'S READY.

 > -dorian, who considers college radio stations a more "interesting" content
As soon as you are into CD-quality audio, tune into 224.42.42.2.

Pete

---

From: "Dorian R. Kim" <[email-masked]>
Date: Tue, 11 Feb 1997 05:35:34 -0500 (EST)

> But remember that MBONE IS NOT READY.

And at this rate, probably will never be.

> (and many, like you and I do)

Actually, I couldn't care less about Mbone. What I want to see is a global,
ubiquitous, scalable and manageable ip multicast infrastructure.

-dorian

---

From: Dermot Tynan <[email-masked]>
Date: Tue, 11 Feb 1997 12:12:28 +0000 (GMT)

Van Jacobson wrote:
> at least 3 of them active [...].  Oh well, perhaps one day the
> parties involved will realize they're not actually demonstrating
> a very high level of coolness.

What's more interesting is what'll happen to them when the BMI and
other Performing Rights societies get ahold of them.  As I mentioned
before, these guys are just coming out of the starting gate now, and
they're looking around.  When you pay for a broadcasting license, it
doesn't give you the right to broadcast music or performances.  These
must be licensed separately from PRS (or their equivalent).  This
license covers a given region, and the cost is directly related to
the geographic area covered by the broadcast.  Once you start putting
music onto the Internet, then you need to renegotiate your PRS
agreement to cover the new catchment area.  I'm willing to bet that
the initial agreements for the Internet won't be cheap...
						- Der
-- 
Dermot Tynan						+353 91 754608
[email-masked]					 DTN: 822-4608

AltaVista Internet Software, Galway, Ireland

---

From: Erik Sherk <[email-masked]>
Date: Tue, 11 Feb 1997 10:59:47 -0500

> ubiquitous, scalable and manageable ip multicast infrastructure.

Hear, hear! A "Global Multicast Internet" faces four fundamental hurdles

	1) A workable egp. We can't run a worldwide network with RIP.

	2) Decent management tools. Right now it takes a guru to debug
mbone problems.

	3) An address allocation policy that makes sense. 

	4) A business model that ISPs can use. Billing by the size of the 
pipe doesn't cut it.

I think that we have learned all we can with the current mbone experiment
and people who want to see multicast succeed should be working to solve
these problems.

Erik

---

From: Bill Fenner <[email-masked]>
Date: Tue, 11 Feb 1997 07:57:43 PST

In message <[email-masked]> you write:
>John, I just logged in to dec3800-2-fddi-0.SanFrancisco.mci.net and pinged mbo
>  ne2.uu.net and am not seeing any slow response's:

That's because it was 1am =)

At 8am, I'm seeing

----137.39.246.98 PING Statistics----
135 packets transmitted, 125 packets received, 7% packet loss
round-trip (ms)  min/avg/max = 28/95/207 ms

Lots of (queueing delay?  overloaded router cpu?) and a little loss.
I'm sure it gets worse as more people wake up.

  Bill

---

From: Peter Berger <[email-masked]>
Date: Tue, 11 Feb 1997 12:42:09 -0500

Hi.

I'm the guy who decided to put WRCT on the MBone.  Interestingly,
I'm writing this message from the NANOG terminal room.  I think
I have some understanding of the issues involved.  More importantly,
I think we all understand what the issues aren't.

The greater issue isn't whether the content being multicast is 'cool' or
not.  The issue is "Why does the rest of the world think multicast
sucks?"  I think that the discussion we are having ("Dear sir: your
content is causing our structure to collapse.  Please
stop multicasting.") is a great example of why, when presented
with a choice between multicast and unicast technologies, content
providers almost always choose unicast.

I actually had to push to get WRCT to go MBone rather than using
some unicast technology.  "Multicasting is the right thing to do,"
I said.  "It's friendlier to the net, and uses less bandwidth to
get the same information to more people."

It seems to me that the real question is not, "Why is someone
using the infrastructure we've built?" but "Why is the infrastructure
collapsing under a not-even-moderate(*) load?"  We know the answers:
routers that, when you turn on native multicast, explode in huge
gouts of radioactive flame (hi, Cisco), less than optimal topologies
in certain areas, and probably deliberate throttling by certain
providers, among other reasons.  So perhaps our energies could be
concentrated on fixing these problems, rather than convincing people
to not use the infrastructure.

By the way, if the consensus is that it is inappropriate to use
the MBone for decadent, worldly activities such as entertainment and spreading
music which can't be found at even the largest "CD stores" (since most
of what 'RCT broad/multicasts is vinyl from independant labels), I would
be glad to reduce the TTL on our multicasts, or simply find some alternative
technology which uses unicast.  Presumably, Tony Rutkowski won't call me up
and tell me that our unicasts are overloading the routers in his backbone,
and would I please stop.

Peter
(*) A "not-even-moderate" load is defined by me as a load which,
if sent by unicast technology, would not even make the routers break
a sweat.

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 11 Feb 1997 11:39:56 -0800

>> to not use the infrastructure.

    I have heard very very few cases that a unicast router enabled for 
    multicast explodes due to load. Most of the multimedia streams are much
    less than 1 mbps that come through the infrastructure. Even many aggregate
    streams can be handled with current hardware.

Dino

---

From: Ross Finlayson <[email-masked]>
Date: Tue, 11 Feb 1997 14:06:20 -0800 (PST)

At 11:12 AM 2/11/97 -0800, [email-masked] wrote:
>|> "Dorian R. Kim" <[email-masked]> wrote:
>|> 
>|> Hear, hear! A "Global Multicast Internet" faces four fundamental hurdles:
>:
>|>         4) A business model that ISPs can use. Billing by the size of the 
>|> pipe doesn't cut it.
>
>"Doesn't cut it" *compared to what*?
>
>Not obvious.

Agreed.  Some of the messages in this thread sound like they come from a
university economics term paper, but that's not the way that ISPs think.

It's really a lot simpler: ISPs are basically "pipe mongers".  Multicast has
the potential to reduce the size (& thus cost) of the pipes that ISPs *buy*,
and could conceivably (by promoting interesting content) increase the number
(& perhaps size) of pipes that they sell.

        Ross.

---

From: Amancio Hasty <[email-masked]>
Date: Tue, 11 Feb 1997 16:08:57 -0800

>     streams can be handled with current hardware.

Hi Dino,

Dumb question : How many video streams of lest say 200kb/sec can we now
handle? tens , hundreds , thousands?

	Cheers,
	Amancio
P.S.: I trimmed the CC list , got no interest in getting a VAN-O-GRAM 8)

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 11 Feb 1997 17:31:37 -0800

>> handle? tens , hundreds , thousands?

    Depending on fanout, it could be hundreds. Most definitely tens.

Dino

---

From: Amancio Hasty <[email-masked]>
Date: Tue, 11 Feb 1997 17:48:47 -0800

>     Depending on fanout, it could be hundreds. Most definitely tens.

Not hard to imagine a new hardware project 8)

	Cheers and Good Luck,
	Amancio

---

From: Amancio Hasty <[email-masked]>
Date: Tue, 11 Feb 1997 18:26:52 -0800

I would check out the surrounding mbone topology before going on-line
with a trasmission of something like a radio station.

For instance,   pen-mbone-1.sprintlink.net is dropping packets and
is just a hop away from umkc.edu.

The radio station is "Local Kansas City Missouri Radio Station".

So I would say that at bare minimum the tunnel provider should 
monitor new tunnels to their facility and periodically monitor
mbone traffic.

	Cheers,
	Amancio

134.193.220.2   phoenix.umkc.edu 
     v     ^      ttl   67         0 pps        1/518  =  0%   0 pps
207.41.200.99   pen-mbone-1.sprintlink.net 
     v     ^      ttl   68         0 pps      152/517  = 29%   0 pps
206.54.224.150  rs1-chi.nap.net 
     v     ^      ttl   69         0 pps        4/365  =  1%   0 pps
204.70.104.45  
204.70.104.29   dec3800-1-fddi-0.WillowSprings.mci.net 
     v     ^      ttl   69         0 pps        0/361  =  0%   0 pps
204.70.152.77  
204.70.152.61   dec3800-2-fddi-0.Denver.mci.net

---

From: Amancio Hasty <[email-masked]>
Date: Tue, 11 Feb 1997 18:58:19 -0800

> >"Doesn't cut it" *compared to what*?

Well, some technologies charge by the number of concurrent connections,
so it is obvious to me the original statement. In the case of IP Multicast
which stands to affect many more routers than lets say a unicast solution
then it makes sense not to charge by the size of the pipe or we have
to redefine the meaning of "pipe size" for the case of IP Multicast.

> (& perhaps size) of pipes that they sell.

Yeap, and directly related to the above comes the simple question from 
a content provider: With a given Internet Technology how many people
can I reach? Thousands or millions?

It is starting to look like successful mbone deployment is going to take
place on large ISPs and then fan out from there. 

	Amancio

---

From: Erik Sherk <[email-masked]>
Date: Wed, 12 Feb 1997 11:01:31 -0500

> (& perhaps size) of pipes that they sell.

It also has the potential to increase the amount of traffic you have to carry
over your backbone without increasing revenue by making a customer buy a bigger
pipe.

Erik

---

From: Bill Fenner <[email-masked]>
Date: Wed, 12 Feb 1997 10:48:19 PST

>(& perhaps size) of pipes that they sell.

But what the ISP's mostly see is that it is likely to *decrease* the
number of pipes they sell.  How many T3's does pointcast currently
have, and how many would they need if they used multicast?  How about
audionet.com?  How much of a loss is that for the ISP?  Why would they
encourage such a move?

  Bill

---

From: Petri Helenius <[email-masked]>
Date: Wed, 12 Feb 1997 23:16:28 +0200 (EET)

> audionet.com?  How much of a loss is that for the ISP?  Why would they
I would say that best for the ISP's is to get to sell pipes but not
having to carry any traffic. That is the less the pipe is utilized,
the better. (some pricing schemes do not do this though)

So a busy site is 'bad business' for an ISP. Since they push out data
at the rate they pay for. As usual 'corporate site' is quite 'good
business' since their pipe is idle for most of the day and they
usually buy more bandwidth than they use. (just to keep the QoS high)

Pete

---

From: Peter Berger <[email-masked]>
Date: Wed, 12 Feb 1997 17:34:30 -0500

[CC list snipped.]

Ok.  I am definately guilty of exaggerating the "huge gouts of flame"
part.  It was hyperbole.  In the interests of not having the folks at
Cisco telnet to my routers and type in the Secret Detonation Code
(It's a JOKE!  I SWEAR it! :-), what I referring to was:

At NANOG, at least one Large Provider Who Shall Remain Nameless
indicated in conversation that they had tried using native multicast
on Cisco IOS (I believe version 11.2, though I wouldn't swear to it)
and encountered performance degradation in (unicast) packet forwarding
sufficient to convince them to immediately revert to tunneling.

That's all I meant.  And I am -certainly- not implying that there is
any other router vendor that I am aware of that does native multicast
better. 

Pete Berger
Coordinator, Regional Information Infrastructure
Pittsburgh Supercomputing Center
[email-masked]	http://www.psc.edu/~peterb

---

From: "Louis A. Mamakos" <[email-masked]>
Date: Wed, 12 Feb 1997 17:52:42 -0500

> audionet.com?  How much of a loss is that for the ISP?  Why would they

While I certainly can't claim to speak for other ISPs, and I'm not
sure I even speak for UUNET, but I don't believe that the decrease
in the number of pipes to multicast usage is of paramount concern.
(Of course our sales associates who have sales goals to meet each
month might have a different slant..) 

The concern that I have, and my product marketing people have, is
how to reconcile the resource use of multicast with the cost model
we use.  You see, we've got this Really Big Excel spreadsheet.  On it
is an array of cells which correspond to how much backbone capacity
(in kbits/second) each class of customer typically consumes.  Then
there's the cost of providing a unit of backbone capacity.  And 
depreciation expense of the hardware, etc.  You turn the big crank 
and then hopefully the number at the bottom pops out with a plus
sign, our finance people are happy and the stockholders are happy.

The question is how do you attribute the costs associated with
multicast traffic?  Here you've got a product where the multicast 
source on the end of a T1 pipe can consume much greater quantities of
backbone capacity than the equivilent unicast situation.  So, what
should it cost?  Who do you charge, and how much?

There's no argument that you can make more "efficient" use of backbone
capacity in the multicast cast vs. the unicast case - it's just that
there's a revenue stream you can attribute to offset the unicast
traffic expense, and no obvious corresponding revenue stream for the
multicast traffic.

So what remains for us to do in the ISP community is to invent a
viable business model for products/features which can support the
cost of implementing the infrastructure.  And we haven't even yet
opened the can of worms on what the operational cost of the multicast
infrastructure is.

This isn't a protocol or technology problem - it's a spreadsheet
and business problem.  Driving a spreadsheet around and understanding
the costs can be a quite enlightening experience which more engineers
and network designers should share in.  While some amount of hand-waving
with the explanation of participating in the "MBONE experiment" will be
tolorated, moving this technology into a real product offering requires
somewhat more methodical analysis.  This is what some of us are faced
with.  This is my perspective and reality.

-- 
Louis A. Mamakos,  Manager, Strategic Technologies  [email-masked],  uunet!louie
UUNET Technologies, Inc.                            Voice: +1 703 206 5823
3060 Williams Drive                                 Fax:   +1 703 208 5390
Fairfax, VA 22031

---

From: John Konstantopoulos <[email-masked]>
Date: Wed, 12 Feb 1997 18:23:26 -0500 (EST)

> should it cost?  Who do you charge, and how much?

Louis, you may want to read the following paper: 
"sharing the cost of multicast trees" by Herzog et al. It first appeared 
in SIGCOMM'95.  It addresses some of the issues using game theory.

regards,
-yannis
--
[email-masked]

---

From: Petri Helenius <[email-masked]>
Date: Thu, 13 Feb 1997 08:09:56 +0200 (EET)

> sufficient to convince them to immediately revert to tunneling.

This is definately possible. All current Cisco routers are seriously
flawed in design so that they depend on the main CPU on building and
distributing their forwarding tables (route-caches). This is
independent of the technology generation (SSE / VIP1 / VIP2). The flaw
in this is that if you are suddenly subjected to increased amount of
destinations, your forwarding performance goes down the drain. (or in
case of IP multicast, if you are subjected to increased amount of S,G
pairs) 
 > any other router vendor that I am aware of that does native multicast

Depending on your topology, it's level of meshing and the multicast
routing protocol you tried to use, you very well can run into
problems. However, it definetly deserves a mention that if you would
run the tunnels between the same boxes, you'll end up with a worse
performance problems but obviously if you run your IP mcast traffic
across the boxes having the problems, you'll less impacted.

Pete

---

From: Tony Li <[email-masked]>
Date: Thu, 13 Feb 1997 00:35:03 -0800 (PST)

All current Cisco routers are seriously

Just in case anyone mis-parsed that, please note that the bug is that it's
a cache.  Having the main CPU computing forwarding tables is an entirely
reasonable thing to do.  Routers which have full forwarding tables
(computed by the main CPU) are much more stable.

Folks adding (S,G) entries might consider this...  ;-)

Tony

---

From: Petri Helenius <[email-masked]>
Date: Thu, 13 Feb 1997 15:06:05 +0200 (EET)

> Folks adding (S,G) entries might consider this...  ;-)
Yes, I forgot to clarify it enough that routers building and
distributing their forwarding caches based on traffic, not their
routing tables. This is the same drawback Ipsilon has with their flow
setup (but it's even worse since the number of flows is far greater
than number of destinations or even number of S,G entries)

One can do interesting math what is your forwarding table building
and usage penalty in the two following cases, where you have say 20000
active prefixes (active == traffic within 5 minutes) and 45000 routes
(the current amount in the Internet).

The "average flow" consists of 10 packets sent with TCP slow start.

Case 1:
- You enter cache entries on reception of the first packet, generating
  a cache entry takes 10ms. (meanwhile all packets are queued up to
  the CPU) You throw 5% of your cache entries overboard every minute.
  Additionally you "invalidate" cache entries when routing changes.
  Since about half of your prefixes are in the cache, the chance of
  having to invalidate a forwarding entry on routing change is about 50%.

Case 2:
- You build your "cache" or forwarding entries based on routing
  information. The current BGP routing in the Internet seems to
  experience little less than a routing update per second. 

In case 1 your "cache" maintenance rate is 31800 updates / hour.
In case 2 your "cache" maintenance rate is 3600 updates / hour.

All this does not take into account the packets that are received
during the cache generation nor maintenance. Since around 10% CPU
spent for cache maintenance is considered "normal" in Cisco
environment, that means that 10% of the time packets going to
destinations not in the cache are pushed up to the CPU in addition to
initial packet to a specific destination.

Obviously the math above is oversimplification of the issues since
depending on the media of your choice, the media specific addresses
and forwarding destinations (like ATM VC's) can change without routing
changes being made and depending on the structure of forwarding
cache, added complexity and performance penalty based on these changes
varies a lot.

And it should not go unnoted that there is performance penalty for
carrying forwarding databases with routing information and the
breakeven point between cache based and other approaches varies from
implementation to another. Playing with the math above and
implementation field experience gives the correct figures for your case.

Pete

---

From: [email-masked]
Date: Thu, 13 Feb 1997 13:15:29 -0800 (PST)

Pardon the length of this reply.  Please just skip it
[rather than sending flames :-) ].  I'm just trying to clarify
for myself what the "real" obstacles are regarding global
multicast.

"Louis A. Mamakos" <[email-masked]> writes:

|> [Bill Fenner wrote:]
|> > But what the ISP's mostly see is that it is likely to *decrease* the
|> > number of pipes they sell.  How many T3's does pointcast currently
:
|> While I certainly can't claim to speak for other ISPs, and I'm not
|> sure I even speak for UUNET, but I don't believe that the decrease
|> in the number of pipes to multicast usage is of paramount concern.
:
|> The concern that I have, and my product marketing people have, is
|> how to reconcile the resource use of multicast with the cost model
|> we use.  You see, we've got this Really Big Excel spreadsheet.  On it

So far, the discussion has identified a couple of costs unique to 
multicast: first, you may have backbone packet traffic multiplication 
for each possible exit point to other networks.  [If the network has 
only one connection to an IX/NAP/etc., then there is no multiplication 
and there is no additional cost.]  Second, you have the possible
extra router cost of packet replication and forwarding beyond that
of a unicast packet.  But, for both these costs, should you
be comparing it to a single unicast stream, as some ISP posters
contend, or, should you compare it to N equivalent unicast streams?
The latter.  Therefore, I would agree with Bill Fenner, that the
"problem" from an ISP's point of view is not (necessarily) that of
"increased cost" due to multicast, but rather, "decreased revenue" 
due to the sender being able to get the same service as N unicast 
streams with a pipe 1/N as big, and therefore being able to pay less
for the smaller pipe.  [Unless multicast replication and forwarding
is horrendously expensive: that is, far beyond N*(unicast cost).]
[I think I agree with Bill, but, I may have misunderstood something,
so he may not agree with me.]  

In any case, I am a little puzzled by why this argument keeps 
coming up in this way.  I think that N-way multicast should 
be compared to the equivalent N unicast data streams, not 1 
unicast data stream.

|> backbone capacity than the equivilent unicast situation.  So, what
|> should it cost?  Who do you charge, and how much?
|> 
|> There's no argument that you can make more "efficient" use of backbone
|> capacity in the multicast cast vs. the unicast case - it's just that
|> there's a revenue stream you can attribute to offset the unicast
|> traffic expense, and no obvious corresponding revenue stream for the
|> multicast traffic.

Part of the equation is what people perceive as "fair".  In the
previous (mostly non-profit, "fair-use" restricted) Internet world, 
it seemed "fair" to attribute as much as possible of total costs 
to information receivers, since often the information sources 
were individuals, companies, universities, etc. who were providing 
a lot of information services gratis, pro bono publico.  In the new 
(for-profit) world, which includes advertising revenues for the 
information providers, that perception may change.  

|> This isn't a protocol or technology problem - it's a spreadsheet
|> and business problem.  ...
:

Toerless Eckert <[email-masked]> writes:

|> There are a lot mote obstacles to internet-wide multicast deployment than
|> sparse-mode pruning alone. You'd first have to get good symmetric
|> routing into most places to make sparse-mode routing work (at least
|> for PIM). With the abundance of peering providers and ad-hoc BGP
|> connections, i guess that you won't have much success. 

Isn't symmetric routing among backbone networks with multiple IX/NAP 
connections rather unusual these days?

                                                          Second you do
|> have to change router architectures to make them sustain multicast traffic
|> in the same way as they sustain unicast traffic. This is not
|> very likely either unless demand is there to pay for this. Until there's
|> enough of that demand, nobody will dare to spare a high percentage of his
|> routers capacity to multicast because it eats up this capacity faster than
|> it fills up the links thus wasting expensive link ressources. 

I'm confused about this is saying.  Is it saying that multicast forwarding
a packet to N interfaces is vastly more expensive than forwarding N 
unicast packets to those same interfaces?  Are there numbers supporting 
this?  I have no knowledge either way.

Dino Farinacci <[email-masked]> writes:

|> >> There are a lot mote obstacles to internet-wide multicast deployment than
|> >> sparse-mode pruning alone. You'd first have to get good symmetric
|> >> routing into most places to make sparse-mode routing work (at least
|> >> for PIM). With the abundance of peering providers and ad-hoc BGP
|> 
|>     Toerless, this is not true. We have sites running sparse-mode PIM over
|>     a partially deployed native multicast topology. The only problem is that
|>     they have to use slow-converging DVMRP unicast routing to make it work.
|>     But it does work.

There seems to be a disagreement on this point.  Further discussion 
would be enlightening to me.

|>     People need to be a little more forward thinking and look at the long-term
|>     benefits of multicast. Right now it's unicast maintenance in your face
|>     because of the Internet growth. But if people can figure out how to
|>     balance maintenance and development (I contend with this everyday) we can
|>     get to full multicast deployment.

It does seem that the growth in demand for unicast is making 
it difficult for a number of development activities to catch up.
The definition of scalability keeps changing.

Petri Helenius <[email-masked]> writes:

|> This is definately possible. All current Cisco routers are seriously

Is "Cisco" an important adjective here?  Is there any router
available that can sustain a very high bandwidth/forwarding rate 
and simultaneously a very high rate of new forwarding requirements?  
Including multicast?

|> flawed in design so that they depend on the main CPU on building and
|> distributing their forwarding tables (route-caches). This is
|> independent of the technology generation (SSE / VIP1 / VIP2). The flaw
|> in this is that if you are suddenly subjected to increased amount of
|> destinations, your forwarding performance goes down the drain. (or in
|> case of IP multicast, if you are subjected to increased amount of S,G
|> pairs) 
|>  > 
|>  > That's all I meant.  And I am -certainly- not implying that there is
|>  > any other router vendor that I am aware of that does native multicast
|>  > better. 
|> 
|> Depending on your topology, it's level of meshing and the multicast
|> routing protocol you tried to use, you very well can run into
|> problems. However, it definetly deserves a mention that if you would
|> run the tunnels between the same boxes, you'll end up with a worse
|> performance problems but obviously if you run your IP mcast traffic
|> across the boxes having the problems, you'll less impacted.

I'm not sure I understand this last paragraph.

Petri Helenius <[email-masked]> writes:

|> Tony Li writes:

:
|>  > Just in case anyone mis-parsed that, please note that the bug is that it's
|>  > a cache.  

Is this a "bug", or, a reasonable (unicast) optimization for routers 
with significant distributed (non-main-CPU) computational resources?

                Having the main CPU computing forwarding tables is an entirely
|>  > reasonable thing to do.  Routers which have full forwarding tables
|>  > (computed by the main CPU) are much more stable.
|>  > 
|>  > Folks adding (S,G) entries might consider this...  ;-)
|>  > 
|> Yes, I forgot to clarify it enough that routers building and
|> distributing their forwarding caches based on traffic, not their
|> routing tables. 

I'm not sure I understand this, either.

                    This is the same drawback Ipsilon has with their flow
|> setup (but it's even worse since the number of flows is far greater
|> than number of destinations or even number of S,G entries)

|> One can do interesting math what is your forwarding table building
|> and usage penalty in the two following cases, 
:
|> Case 1:
|> - You enter cache entries on reception of the first packet, generating
:
|> Case 2:
|> - You build your "cache" or forwarding entries based on routing
|>   information. The current BGP routing in the Internet seems to
|>   experience little less than a routing update per second. 

How do you do this?

|> In case 1 your "cache" maintenance rate is 31800 updates / hour.
|> In case 2 your "cache" maintenance rate is 3600 updates / hour.
|> 
: 
|> And it should not go unnoted that there is performance penalty for
|> carrying forwarding databases with routing information and the
|> breakeven point between cache based and other approaches varies from
|> implementation to another. Playing with the math above and
|> implementation field experience gives the correct figures for your case.

This looked like an interesting analysis, but I was unable to follow
it completely.  Also, I'm not sure what the implications for multicast
are.

  Hugh LaMaster, M/S 258-5,     Email:       Please send ASCII documents to:
  NASA Ames Research Center     Internet:    [email-masked]
  Moffett Field, CA 94035-1000  Or:          [email-masked] 
  Phone:  415/604-1056          Disclaimer:  Unofficial, personal *opinion*.

---

From: [email-masked] (Alan Batie)
Date: Thu, 13 Feb 1997 13:37:54 -0800 (PST)

> ... since often the information sources 

The trick is to not kill the goose to get the golden egg.  *Both* models
have to co-exist, or everyone loses.

-- 
Alan Batie                     ------        What goes up, must come down.
[email-masked]        \    /        Ask any system administrator.
+1 503-264-8844 (voice)         \  /         --unknown
D0 D2 39 0E 02 34 D6 B4          \/          5A 41 21 8F 23 5F 08 9D

---

From: Tony Li <[email-masked]>
Date: Thu, 13 Feb 1997 14:52:06 -0800 (PST)

There are implementations which clearly implement a more stable
architecture.  The fact that the particular implementation is somewhat
dated is mostly orthogonal.  That implementation does not support multicast.

It is very much a bug.  More specifically, it's MY bug.  ;-)

It _was_ a reasonable implementation prior to CIDR, when utilization of the
core routing tables was relatively sparse.

   How do you do this?

Well, that answer is worth approximately $2B.  ;-)

The implication for multicast is pretty simple: your forwarding table
should contain all of your (*, G) entries.  Further, if you have to compute
an (S, G) entry, it should be possible to do so with your forwarding
processor and must be done extremely quickly.

Architecturally, it would be much, much, much better if (S, G) entries
never existed at all.  This would improve both stability and scalability.

Tony

---

From: Petri Helenius <[email-masked]>
Date: Wed, 26 Feb 1997 19:05:36 +0200 (EET)

>     Depending on fanout, it could be hundreds. Most definitely tens.
I actually went into some effort trying this out on a 7206, having
NPE-150, multicast fastswitching enabled, fanout of three running all
connected interfaces (one in, three out) and my result is that at
around 12 streams I start to experience frame loss. Each stream is
25pps one kilobyte frames.

I wonder what's the fanout needed to achieve the 100 streams figure?

Pete