Subject: Re: Raising the bar on the MBone

From: [email-masked]
Date: Thu, 31 Jul 1997 17:17:12 -0700 (PDT)

Steve Casner wrote:

>> > IPMI is now planning a project called the "Internet Multicast Channel"
>> > (IMC) to transmit compelling content over the MBone for a fixed number
>> > of hours per day, several days per week.  By bringing together ISPs
>> > and content providers the IPMI hopes to overcome the "chicken and egg"
>> > problems that have so far have hampered development of commercial
>> > multicast capabilities in the Internet: content providers need
>> > assurance that the network will work; ISPs won't implement multicast
>> > as a service if there is no content and no demand. 

Louis Mamakos replied:
 
>> Demand for what?  I suspect there's unlimited demand for free Internet
>> connectivity, yet that's not compelling enough to build a network to
>> offer free service.  There must be demand for a credible multicast-IP
>> based service offering.

I think it would help to understand your reply if you stated what 
your assumptions are about: an internet pricing model, about
the IPMI project, about routing, pruning, technical issues ...
And, in particular, ways in which multicast differs from unicast
wrt the pricing model.

I haven't heard anyone mention "free service".  Is there an unstated
assumption here that offering multicast connectivity will, in and
of itself, cost a huge amount of money?  But, many ISP's and end-users 
already have routers which already support multicast, so the large costs 
you assume must be for data links.  On the customer's side, the
customer has to pay for the pipe, and it is a customer decision, but, 
clearly, the customer will save money over the equivalent unicast. 
As for the national-carrier-ISP, it seems a debatable point whether
or not multicast will save them money, cost them money, or be neutral, 
relative to the equivalent unicast.
 
>> ISP's won't implement a large-scale multicast-IP based service if there's 
>> no compelling business case to do so.  "Content" is compelling only
>> to end users.  It would be good if that available content drives a service
>> which can support the infrastructure deployed to implement it.

I regret to say that I am apparently not clever enough to understand this.
If end users "demand" multicast, why isn't that a compelling business case?
By "demand", I mean, that they will go to the trouble to switch to a
provider which offers multicast.  At what cost, you reply?  What if the
cost (for the same sized pipe, whether T1 or OC-192) is the same?  Again, 
there seem to be assumptions here which I'm not grasping...

>> How much bandwidth do you think that ISP's are willing to devote to the
>> (essentially free) MBONE experiment?  1 Mb/s?  10 Mb/s?  100 Mb/s?  

Well, it is an *experiment*, and also, I thought that the suggested
bandwidth was somewhat clear.  e.g. ~2 Mbits/sec.  

At some
>> point the free ride is going to run up against financial realities when somone
>> is faced with having to buy another DS3 to carry traffic which produces
>> no revenue.  Either the overall cost of getting Internet service is going
>> to increase to cover (subsidize) these multicast infrastructure costs, or
>> people that "use" multicast traffic will end up having to cover the costs.  
>> Are these traffic sources or receivers?  I have my ideas, but eventually 
>> someone's gonna have to belly up to the bar to pay the bill.

At the moment, the Internet is basically customer endpoint-pay, 
whether sender or receiver, although, Web-host sites have a 
variation on this, by charging for bytes sent, but, the Web site 
sells advertising revenue.  Such arrangements help information-
providers pay for "pro-bono" information.  Obviously, advertising
is on the increase, complicating things.  etc. etc.

Anyway, is there an assumption here that customer endpoint-pay 
is impermissible? Why?  If a business customer feels that by 
switching from a T1 to dual T1's or a T3, they will receive 
benefits greater than their costs, then, they will switch.
This is really independent of multicast.  

I'm not trying to be difficult.  Apparently there are assumptions
about the future of Internet pricing which not all of us share
or even are privy to.

-Hugh LaMaster

  Hugh LaMaster, M/S 258-5,     ASCII Email:  [email-masked]
  NASA Ames Research Center     Or:           [email-masked]
  Moffett Field, CA 94035-1000  No Junkmail:  USC 18 section 2701
  Phone:  415/604-1056          Disclaimer:   Unofficial, personal *opinion*.

---

From: Petri Helenius <[email-masked]>
Date: Mon, 4 Aug 1997 17:39:51 +0300 (EEST)

At 10:25 3.8.1997 +0200, Luca Bertagnolio wrote:
>At 22.05 02/08/97 +0100, Martin Hamilton wrote:
>>Hmm...  Everyone seems so hung up on audio and video :-)
>
>Agree.  I am thinking also of the possible benefits of turning push
>technlogies like PointCast to multicast, at least for distribution over the
>higly overpriced WAN links (well, overpriced here in Europe!), with some
>local caching for the dialup users that can't be forced to wait online
>until their requested info comes in via multicast.
>
Correction. WAN links are generally overpriced in central and southern
European countries. Fortunately the Nordic is much less worse in that sense. 
Unfortunately this does not help intercontinental links that much.

Pete

---

From: [email-masked]
Date: Mon, 4 Aug 1997 14:37:54 -0700 (PDT)

[email-masked] (Ran Atkinson) wrote:

> On Aug 4 10:28, Steve Deering wrote:
> 
> > TCP does not currently work over IP multicast, so multicast apps
> > use UDP.  UDP does not have any congestion avoidance properties.
> 
> % This is orthogonal to multicast -- the same issue arises with
> % unicast UDP apps, or with hacked or misused TCPs.

Agreed.  I would like to apply the same standards in the discussion
to multicast and unicast, whether present or future.  In the future,
the standards for robustness in the face of misuse will have to be
much higher than they are now, whether unicast or multicast.

> 
>         In a non-operational abstract sense, they could be viewed
> as orthogonal.

>         However, in an operational pragmatic sense, they are not
> orthogonal.  

Also agreed.  From an operational standpoint, not many people
(out of the total internet-using population) are using or abusing 
non-multicast UDP globally.  Yet.  [What do all those different
internet phone apps that are now for sale use?  UDP or TCP?]

But, since the IPMI is talking about a temporary, experimental 
service, I think we should apply "current operational"
standards to such experiments.  If we are trying to solve
long-term problems, then, let's apply the same standards
to unicast.

Steve Casner's original email was to the existing mbone 
community.  I don't see much abuse right now on the
mbone/multicast UDP side, either.   

But, we all recognize that as soon as somebody sees, 
or thinks they see, a ton of money to be made abusing 
multicast, they will, just as email is now being abused.  
Do we cut off unicast completely just because email is being
abused? Fortunately, with Mbone-like broadcasts, pruning 
will take care of the sources nobody wants to see.

>               There are very few unicast UDP applications in use,
> even though they require no special support from the infrastructure.

Hmm.  There are tons of (LAN and nearest-neighbor) UDP applications.  
I assume that you mean end-user, globally-routed, UDP.

> The most commonly used applications (e.g. http, ftp, telnet) use TCP
> and consequently have reasonable congestion avoidance properties.

>         A "hacked TCP" is broken, pure and simple.  However, the
> majority of the TCP implementations that are deployed _do_ implement
> slow-start and exhibit reasonable (if not ideal) congestion avoidance
> properties.  

Even so, "greedy" implementations may get more than their 
share of bandwidth, undesirable traffic and loss oscillations 
may occur over congested WAN links, etc.  And, what about
applications that like to open up multiple TCP streams each 
of which gets a piece of the pie now?  Is that considered 
"abuse"?

In short, a unicast solution will also be needed "eventually",
just like a multicast solution will be needed.  (And, the 
answer will probably turn out to be some form of--- "weighted
fair queueing").

> % What is required is mechanisms in routers (such as statistical
> % approximation of fair queueing, or RED-with-penalty-box) to prevent
> % non-congestion-sensitive flows from hogging links.
> 
        When those features are available in a stable router release,
> I'm sure that ISPs will deploy them with glee.  Until then,
> we are where we are.

I observe, while we are on the subject, that UDP audio/video streams
with a *very* long-lived address/port should be easily identified 
as "flows", making them much easier to identify and manage, 
queue-wise, than the random 20% of packets that are web-related 
"button clicks".

>         If one wants native IP multicast to be more widely deployed,
> one might reasonably spend effort working on making the above mechanisms
> more widely available to ISPs and other network operators.  Of course,
> folks who build routers are in a better than average position to
> influence things in this regard.

Here, however, I would have to disagree.  I think many ISPs
seem to be overlooking a major advantage to themselves of
supporting and encouraging the growth of multicast-enabled
information distribution, whether web, multimedia, software,
etc..   To the extent that the same packet can be delivered
to N users where N >> 1, WAN bandwidth has been conserved
relative to end-user bandwidth consumed.  From an ISP standpoint
that should be a Good Thing.  A user with a (T1, say)-sized 
pipe who is pulling lots of multicast in to their network will be
consuming less WAN bandwidth than a user who is consuming the
same bandwidth with unicast apps.  To the extent that multimedia
multicast takes off, the ISP will be able to re-sell the same
WAN bandwidth N times.

> % We are no longer in a world where we can trust that all IP sources
> % will be "well-behaved", not even TCP sources.
> 
>         Perhaps.
> 
>        However, for the present, TCP-based sources are behaving,
> while a number of deployed IP multicast applications lack any
> form of congestion avoidance and do offer excessive bandwidth load
> under congested conditions.

Again, we seem to be mixing the present and the future.

        The multicast games tend to be the worst offenders, perhaps
> because the vat/vic/wb users have largely been educated to rate-limit
> their offered load and the vat/vic/wb implementers have reasonable
> default settings.

Multicast games represents, IMHO, a much larger future challenge, 
for the simple reason that the state required in the routers is 
the same as an "Mbone broadcast", or even a small multiway 
videoconference, but, the amount of data flowing is typically small.
Modems at 28.8 are quite adequate for some games, and one could
easily have many thousands of small groups playing games and taking
up router state.  This will be a major hurdle for the IDMR working
group to solve.

The "bandwidth" issue is, IMHO, a red herring.  An ISP who is selling
a capacity-based link to an end-user would benefit if the same bandwidth
is consumed via multicast-based "Mbone"-type broadcasts received by 
N people.

-Hugh LaMaster

---

From: [email-masked]
Date: Mon, 4 Aug 1997 15:50:04 -0700 (PDT)

First of all, sorry about a previous unintended garbage email
which some of you may have received.  A literal slip of the 
fingers.  Oops.  Delete.

"John M. Zwiebel" <[email-masked]>

> Hugh said:    
> 
>     I really don't want to let this part of the discussion die.

Well, I guess I succeeded.

>     Consider that N receivers want to receive a particular datastream.
:
>     to carry more than one copy.
> 
> Everyone knows this.  Everyone accepts this.

Unfortunately, although (almost) everyone accepts it, 
abstractly, that is, at the gut-reaction level,
the response is quite different.

Many people can't seem to help themselves thinking
that a user sending multicast is getting something 
for nothing.  The user is actually providing an
opportunity for an internet provider to re-sell
the same packets N times.

> Some things that I think keep getting ignored:
> 
>   --the additional state that the routers may
>         have to carry depending on the protocol being used.

I agree that this is serious, although, not for any near-future
broadcasting or even videoconferencing.  I agree it is a serious
issue with respect to distributed games.  I again note that 
distributed games have not been the subject of discussion under
the IPMI internet-channel idea/proposal which originally precipitated
this discussion.

>   --if the ISP's use closest exit routing, one ISP that has a lot of 
>         sources can cause other ISPs to carry all of the multicast traffic.
>         If the only way to charge for multicast is that the source pays,
>         those ISP's with no sources get stuck with no income and a lot of
>         traffic.  Sources pay for it with advertising.  How do receivers
>         pay?

I think this is a red herring, although, I realize of course 
that not everyone agrees with me.  ;-) 

If I ftp a file from your site to my site, the data traffic 
(from you to me) flows over an exit route at the exchange 
between your provider and mine (simplest case), which
your provider gets to choose for its own benefit.  At least, 
that is what your provider usually is trying to do.

When native multicast is running, if you multicast/broadcast 
a multimedia "Mbone" session to me, the traffic result will be 
(approximately, modulo a bunch of multicast routing things), 
the same, although without "multicast BGP" in some form I won't
be able to control multiple exit point(s) as well.  (It might even
be in both of our interests to cooperate and attempt to optimize
the use of our respective WANs.  It could get interesting and
complicated.)

Anyway, since I, as information receiver, pull lots of traffic 
onto my site from various sources, and since I am paying for 
a (T1, T3, etc.) from my ISP, I am already paying for a T1's 
worth of traffic, whether I choose to use unicast or multicast.
The difference, from the traffic perspective, is that with multicast 
my ISP has the opportunity to re-sell the same traffic to other users.  
Do we, as a group, get a discount, since, we are effectively reducing
the ability of our combined sites to pull other unicast traffic over 
our T1s while not consuming the full bandwidth over all links?  ;-)

I will grant you that there is more to it than this.  In today's
"competitive" environment, other ISPs may adopt scorched-earth
policies of various kinds.  [Scorched earth meaning, that, it
isn't enough to optimize my own case; I may also be trying
to pessimize yours.]  At one time, closest exit routing
was seen as a scorched-earth tactic, and some people went to 
considerable lengths to try to force symmetric routing, but 
eventually, people learned to live with it.   To me, only 
those policies which affect unicast and multicast differently 
should be considered here.

>   --Current routing protocols are flat, no heirarchy.  Lot's of routes
> >         use lots of memory.  No BGP.

I'm painfully aware of that.  

>   --There aren't good ways of setting policy for who carries what.  No
>         BGP for multicast.

True.  There are, of course, a couple of proposals now being fleshed
out for this.

>   --There is little experience with multicasting over a large area.  The
>         imagine the dykstra that MOSPF would have to calculate?  
>            Every router would have to be a cray.

I don't think anyone is proposing using MOSPF as a global IDMR
protocol.  MOSPF could still be interesting to some users on their
internal networks, if it can be stitched together with (generic) 
"Multicast BGP".  [And today, of course, some of the latest routers 
are almost a Cray (1S) from the standpoint of collective CPU power 
and memory bandwidth.  But, that's not enough power to make MOSPF 
work globally.]

>   --There are a lot of mrouted machines running on unix boxes.  They can't
> >         handle all the routes in the internet.  How do you propose keeping
>         these systems up and running?  

I don't.  :-)  I propose letting the routers do the routing, 
and running native multicast everywhere.

> At first glance, one only sees the advantages of using multicast.  Its the
> details that hurt.
> 
> This is just an opinion, but if I hold my breath till I'm blue (ie, if 
> I constantly demand, which seems to be the operative word here) I'm not
> going to get to ride in the space shuttle.  So, users can demand multicast
> on the internet all they want, it isn't going to happen until the tools arrive
> .

Oops.

> They are coming.

The IPMI proposal did not state an intention to immediately bring 
about ubiquitous global multicast routing.  There is, as stated, 
a chicken-and-egg problem.  To the extent that large corporate users
with T1+ service and so on see multicast as a good thing and ask 
their ISPs for the service, it could benefit everyone later.  I don't
think anyone is trying to push global multicast routing "now" - 
but, I agree that the time is ripe for raising the bar, pushing 
the envelope, whatever.

-Hugh LaMaster

---

From: Petri Helenius <[email-masked]>
Date: Tue, 5 Aug 1997 12:15:27 +0300 (EEST)

At 17:06 4.8.1997 +0200, Heiko W. Rupp wrote:
>On Mon, Aug 04, 1997 at 05:40:05PM +0300, Petri Helenius wrote:
>
>> has NOT been is higher bandwidth experiments with any content with the
>
>What is 'higher bandwidth' in that context?
>
I'm talking about around 500 kbps for video and 64kbps for audio and upwards
to MPEG-1 standard which do 1.2M for video and 128-256kbps for audio.
Additionally, if you've ever seen an DVD to play on a decent monitor, you'll
know what you would get from MPEG-2.

Pete

>-- 
>NTG Netzwerk und Telematic GmbH   /          Heiko W.Rupp
>Geschaeftsbereich Xlink         \/           phone: +49 721 9652 0
>Vincenz-Priessnitz-Str. 3       /\ link      fax:   +49 721 9652 210
>D-76131 Karlsruhe, Germany     /_______      email: [email-masked]
>[ NTG Netzwerk und Telematic GmbH, Sitz Chemnitz                          ]
>[ Kreisgericht Chemnitz/Stadt HRB 4217, Geschaeftsfuehrer: Michael Rotert ]
>
>

---

From: Denis DeLaRoca <[email-masked]>
Date: Tue, 05 Aug 97 03:56 PDT

Today's New York Times features an article by John Markoff

    http://www.nytimes.com/library/cyber/week/080597video.html

on an announcement by MCI and Progressive Networks to deploy a
service to multicast video on the Internet.

Anybody have any technical details on the scale of this service
and how it complements/upstages the recently announced IPMI initiative
for scaling up the MBONE experiment?

-- Denis

---

From: Petri Helenius <[email-masked]>
Date: Wed, 6 Aug 1997 00:32:07 +0300 (EEST)

At 08:54 5.8.1997 +0100, Jon Crowcroft wrote:
>
>and the rate of increase of PERCENT of the mbone that is in native
>routers is quite healthy last time i looked.....
>
If you are referring to Piete's statistics, they seem to have stalled
somewhere around 50-50 situation but I'm not sure whether it's the ability
of the tool keeping up with the general mbone growth (it misses 20% of
routers around here) or actual stalling of the percentage change.

Pete

---

From: Petri Helenius <[email-masked]>
Date: Wed, 6 Aug 1997 00:32:01 +0300 (EEST)

At 15:23 4.8.1997, Ross Finlayson wrote:
>
>INSTEAD:
>3'/ We should try to embrace the "unwashed masses" by making our technology
>(& services) available to them, at least in a limited form, while we
>continue to move forward.  This gives them an incentive - and a clean
>migration path - to upgrade later.

This is easy to agree on.
>
>Looking specifically at dialup modem users, there are now mechanisms
>available -  e.g., see "http://www.lvn.com/multikit/tunnel.html" - that
>will allow them to access the MBone, albeit with some restrictions, pending
>direct multicast support in dialup routers.  What we need now, however, is
>more low-bandwidth MBone content.
>
I find it hard to figure out appealing content that could be delivered over
connections that are not constantly established nor don't have that much
bandwidth. I can imagine that many of the ideas here, like the morning news
or weather forecast (Pointcast?) could be delivered unattended even to all
the end users if their connection would be up 24 hours. Because of this, I
think this is something that should be fixed first and while we're fixing
it, we might as well put more bandwidth into place at the same time. All we
have to make sure is that the infrastructure is going to do native multicast
to the CPE. (and there are efforts underway to do just that)

Pete

---

From: Petri Helenius <[email-masked]>
Date: Tue, 5 Aug 1997 12:15:27 +0300 (EEST)

Pete

---

From: Denis DeLaRoca <[email-masked]>
Date: Tue, 05 Aug 97 03:56 PDT

-- Denis

---

From: [email-masked] (Tim Dorcey)
Date: Tue, 5 Aug 1997 11:04:29 -0700

I can't resist replying to the post that mentioned "hacks like CU-SeeMe
reflectors," though I can no longer find it.  In my experience, the biggest
problem with CU-SeeMe reflectors is precisely that:

1.  they were originally designed as IP multicast emulators and
2.  they were accessible to anyone on the Internet

The design restriction (1) meant that they were not allowed to peform any
service that would not translate to IP multicast.  Among other things, this
meant that they could not do any kind of flow control or bandwidth
management.  They simply forwarded whatever data a user requested.  So,
e.g., someone on a modem in Europe could connect to a reflector in the US,
open half a dozen windows, and generate a 500 kbps UDP stream.

This was not tolerable, so the multicast-compatibility requirement was
abandoned.  Reflectors began monitoring throughput to each destination and
only forwarding as much data as was expected to reach its destination.
This was a big improvement in terms of network citzenship and the user
experience (e.g., users weren't being trashed by their own congestion,
audio could be prioritized over video).

Now, what if CU-SeeMe used IP multicast instead of reflectors?  Simply
replacing a reflector with IP multicast would be a disaster.  I.e.,
multicast routing helps the "500 kbps to a modem" problem only in so much
as several nearby people could do the same thing without causing cumulative
damage.  To provide some measure of congestion avoidance, receivers could
monitor packet loss and only request streams that they could successfully
receive.  Assuming dynamic bandwidth availability, they could periodically
test their connection by turning desired streams on, then off when they saw
too much loss.  To provide enough granularity to support receivers with
widely varying capacity and different reception preferences, media streams
could be divided into multiple layers.  Approximating something like
CU-SeeMe's flexibility would probably require at least 6 video layers, say,
between 3 kbps and 80 kbps.  Allowing for other media types (e.g., audio,
text, white board, slides), each source could spread their output over,
say, 10 multicast addresses.  So, we could replace a reflector with a dozen
people on it, with 120 rapidly changing multicast groups (or 10, when
source-specific pruning is in place).  This doesn't strike me as a great
tradeoff to eliminate some duplicate packets on the backbones.  On the
other hand, it would probably be a boon to the CU-SeeMe community, since
they wouldn't need to compete for access to reflector bottlenecks.  But, I
don't see ISP's jumping to support this.

One can argue whether CU-SeeMe-like applications are a good use of Internet
resources, but I don't think it's accurate to suggest they would be better
supported by IP multicast, certainly not today, probably never.  Yes, the
people trying to do "Internet TV" would be better served by IP multicast,
but I can't see that it makes sense for interactive conferencing.  A key
feature of this paradigm is that what makes the source especially
interesting is that it is also a destination.  E.g., I am looking at you,
because you are looking at me.  This places bounds on how large an
"audience" is ever likely to be and limits the potential efficiency of IP
multicast.  I have heard the argument that we should first develope routing
technology to support arbitrarily large distributions, whence serving
smaller N will be an easy special case, but I don't buy it.

Tim

---

From: Petri Helenius <[email-masked]>
Date: Wed, 6 Aug 1997 19:03:43 +0300 (EEST)

At 16:08 6.8.1997 +0200, Toerless Eckert wrote:
>> However, what I meant was a little finer grained form of pruning, checking to
>> see that the particular video packets are grouped together by frames, and
>> forward complete frames, not just randomly throw away a packet as it comes in
>
>You mean on the base of "if an ATM cell-gateway can snoop into AAL5 frames,
>why can't an IP packet-gateway snoop into RTP frames" ?

Actually the EOM bit lives in the header, not in the payload. So this is
different. 
However I see us floating, wondering, freefalling, whatever towards
closed-loop-congestion-control schemes which work equally to the cuse-mee
reflectors nowadays. However, I'm not convinced that we'll end up with
useful results here unless we go a couple of steps back and define whether
we actually want to solve the congestion and oversubscription problems by:
1) the source throttling back (== everybody gets hurt)
2) the gateway experiencing congestion drops packets (== one receiver gets
hurt, possible network impact)
3) the gateway experiencing congestion drops units of information like video
frames (== possible network impact, impact on gateway cost)
4) the gateway informs an upstream gateway to slow down which eventually
ends up in one or more of the 1-3 happening (== much increased state in
gateways, impact on gateway cost)

>Boy am i annoyed. Why can't people use the right schemes from the
>start instead of investigating into glorious hacks afterwards ?
>
I'm more than willing to hear a solution for point-to-multipoint data
distibution in non-uniform environment. Would solve a lot of problems and
would be useful both in IETF and ATMF.

(if you haven't patented it yet, just tell us that it's coming soon :-)

Pete