Subject: Re: Raising the bar on the MBone

From: [email-masked] (Tim Dorcey)
Date: Tue, 5 Aug 1997 11:04:29 -0700

I can't resist replying to the post that mentioned "hacks like CU-SeeMe
reflectors," though I can no longer find it.  In my experience, the biggest
problem with CU-SeeMe reflectors is precisely that:

1.  they were originally designed as IP multicast emulators and
2.  they were accessible to anyone on the Internet

The design restriction (1) meant that they were not allowed to peform any
service that would not translate to IP multicast.  Among other things, this
meant that they could not do any kind of flow control or bandwidth
management.  They simply forwarded whatever data a user requested.  So,
e.g., someone on a modem in Europe could connect to a reflector in the US,
open half a dozen windows, and generate a 500 kbps UDP stream.

This was not tolerable, so the multicast-compatibility requirement was
abandoned.  Reflectors began monitoring throughput to each destination and
only forwarding as much data as was expected to reach its destination.
This was a big improvement in terms of network citzenship and the user
experience (e.g., users weren't being trashed by their own congestion,
audio could be prioritized over video).

Now, what if CU-SeeMe used IP multicast instead of reflectors?  Simply
replacing a reflector with IP multicast would be a disaster.  I.e.,
multicast routing helps the "500 kbps to a modem" problem only in so much
as several nearby people could do the same thing without causing cumulative
damage.  To provide some measure of congestion avoidance, receivers could
monitor packet loss and only request streams that they could successfully
receive.  Assuming dynamic bandwidth availability, they could periodically
test their connection by turning desired streams on, then off when they saw
too much loss.  To provide enough granularity to support receivers with
widely varying capacity and different reception preferences, media streams
could be divided into multiple layers.  Approximating something like
CU-SeeMe's flexibility would probably require at least 6 video layers, say,
between 3 kbps and 80 kbps.  Allowing for other media types (e.g., audio,
text, white board, slides), each source could spread their output over,
say, 10 multicast addresses.  So, we could replace a reflector with a dozen
people on it, with 120 rapidly changing multicast groups (or 10, when
source-specific pruning is in place).  This doesn't strike me as a great
tradeoff to eliminate some duplicate packets on the backbones.  On the
other hand, it would probably be a boon to the CU-SeeMe community, since
they wouldn't need to compete for access to reflector bottlenecks.  But, I
don't see ISP's jumping to support this.

One can argue whether CU-SeeMe-like applications are a good use of Internet
resources, but I don't think it's accurate to suggest they would be better
supported by IP multicast, certainly not today, probably never.  Yes, the
people trying to do "Internet TV" would be better served by IP multicast,
but I can't see that it makes sense for interactive conferencing.  A key
feature of this paradigm is that what makes the source especially
interesting is that it is also a destination.  E.g., I am looking at you,
because you are looking at me.  This places bounds on how large an
"audience" is ever likely to be and limits the potential efficiency of IP
multicast.  I have heard the argument that we should first develope routing
technology to support arbitrarily large distributions, whence serving
smaller N will be an easy special case, but I don't buy it.

Tim

---

From: Jon Knight <[email-masked]>
Date: Tue, 5 Aug 1997 19:40:12 +0100 (BST)

> 1.  they were originally designed as IP multicast emulators and

Er, I seem to remember running CUSeeMe reflectors many moons ago without
any IP multicast at all - well before we had MBONE access (and we were a
pretty early site to have it in the UK).

> open half a dozen windows, and generate a 500 kbps UDP stream.

But you can rate limit with IP multicast; mrouted does it all the time.
The fact that the CUSeeMe reflector didn't has nothing to do with IP
multicast.
 
> damage.

Luckily the popular mrouted multicast router does allow you to rein back
bandwidth over its tunnels.  So you could rate limit multicast streams
back on modem links.  Whether this would make the dial multicast
tunnel useful or not depends upon how much bandwidth your unicast app
wants.

> To provide some measure of congestion avoidance, receivers could
> too much loss.  

> To provide enough granularity to support receivers with
 
Or alternatively could the clients could just use a single stream on a
single multicast address that can survive some degree of packet loss.  But
for many applications there comes a point when your bitrate is just too
low and you just have to give up.  Often that decision is a Quality of
Service decision that only the end user can make.  I often get MBONE
streams from the other side of the planet throttled back to riduclously
low rates.  Audio is the first victim, then video and lastly the
whiteboard/multicast data distribution protocols as the total
link bandwidth available declines.  Sometimes I don't mind if my video is
only infrequently updated due to packet loss (such as when I'm idly
watching the shuttle in a window whilst doing other stuff).  I'm glad that
vic doesn't decide I can't use those links.

> supported by IP multicast, certainly not today, probably never.  

Most of the CUSeeMe applications I've seen have been very similar to the
MBONE applications I've seen.  Indeed I once used CUSeeMe, Maven and
monstermash to relay nv/vat encoded video/audio streams over a unicast
link from a MBONE connected subnet to a Mac in my bedroom to watch an IETF
meeting.  I think there's a big win getting a single multicast stream
coming into SuperJANET which flows only to the pruning multicast routers
that want it.  Even on campus multicast scales better than CUSeeMe as we
engineer the MBONE links carefully to ensure that only one copy of a
multicast packet traverses each subnet.  Even if there are only ten
CUSeeMes talking to a reflector on a single subnet, that subnet will
suffer ten times the load that a multicast aware app would generate.

> Yes, the
> multicast.

If you're limiting yourself to a one-to-one teleconference then indeed
multicast offers no advantages over multicast.  However I've frequently
taken part in sessions with multiple particpants, some situated in
different parts of the same organisation.  In this case, multicast is much
nicer to the infrastructure.  And I'd like my conferencing
tools to handle both styles of communication.  The MBONE tools usually
work in unicast mode as well so they're my tools of choice.

> I have heard the argument that we should first develope routing

I do - I want an infrastructure that has good scaling properties, not a
unicast hack that will max out as N exceeds double figures.

Tatty bye,

Jim'll

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Jon "Jim'll" Knight, Researcher, Sysop and General Dogsbody, Dept. Computer
Studies, Loughborough University of Technology, Leics., ENGLAND.  LE11 3TU.
* I've found I now dream in Perl.  More worryingly, I enjoy those dreams. *

---

From: "Brian O'Shea" <[email-masked]>
Date: Tue, 05 Aug 1997 19:52:53 -0400

-
-On Tue, 5 Aug 1997, Tim Dorcey wrote:
-> I can't resist replying to the post that mentioned "hacks like CU-SeeMe
-> reflectors," though I can no longer find it.  

Thanks Tim, I wanted to reply too....

-In my experience, the biggest
-> problem with CU-SeeMe reflectors is precisely that:
-> 
-> 1.  they were originally designed as IP multicast emulators and
-
-Er, I seem to remember running CUSeeMe reflectors many moons ago without
-any IP multicast at all - well before we had MBONE access (and we were a
-pretty early site to have it in the UK).

I don't want to put words into Tim's mouth, but what I beleive he is saying is
that the underlying protocol was written such that it would always work in a
multicast enviornment.  As a consequence the point to point performance was
sub-optimal in that it restricted what you could do.

-
-> The design restriction (1) meant that they were not allowed to peform any
-> service that would not translate to IP multicast.  Among other things, this
-> meant that they could not do any kind of flow control or bandwidth
-> management.  They simply forwarded whatever data a user requested.  So,
-> e.g., someone on a modem in Europe could connect to a reflector in the US,
-> open half a dozen windows, and generate a 500 kbps UDP stream.
-
-But you can rate limit with IP multicast; mrouted does it all the time.
-The fact that the CUSeeMe reflector didn't has nothing to do with IP
-multicast.

Well, in the later version, 4.0B3 from Cornell, and all of the White Pine
versions since then, there IS rate control.  I beleive Tim wrote it, nes pas?

- 
-> Now, what if CU-SeeMe used IP multicast instead of reflectors?  Simply
-> replacing a reflector with IP multicast would be a disaster.  I.e.,
-> multicast routing helps the "500 kbps to a modem" problem only in so much
-> as several nearby people could do the same thing without causing cumulative
-> damage.
-
-Luckily the popular mrouted multicast router does allow you to rein back
-bandwidth over its tunnels.  So you could rate limit multicast streams
-back on modem links.  Whether this would make the dial multicast
-tunnel useful or not depends upon how much bandwidth your unicast app
-wants.

But does it do it's rate limiting intelligently?  Does it prune video at the
expense of audio?  Does it keep track of video frames and send complete frames
whenever possible?  No.  The latest version of the original Cornell reflector,
White Pine MeetingPoint 3.0, does prune bandwidth intelligently.

	Stuff deleted.......

-
-> I have heard the argument that we should first develope routing
-> technology to support arbitrarily large distributions, whence serving
-> smaller N will be an easy special case, but I don't buy it.
-
-I do - I want an infrastructure that has good scaling properties, not a
-unicast hack that will max out as N exceeds double figures.

So, use a combination of White Pines multicast capable CU-SeeMe clients
connected to a multicast enabled conference controlled by a reflector.  The
users that have bandwidth connect in using multicast, and those that don't
connect to the server using unicast.  The server receives the multicast
traffic and forwards it to the rate controlled unicast endpoint.  If the
unicasted endpoint requests more data then it can handle by opening too many
video windows, the server will intelligently prune the data destined for it. 
Similarly, any of the multicast endpoints that wish to watch the unicasted
endpoint will receive the data via multicast from the server.

The best of both worlds!

-bos

+***********************************************************************+
+ Brian O'Shea ([email-masked])			White Pine Software	+
+ Engineering Manager 				542 Amherst St		+
+ MeetingPoint Conferencing Server (MPCS)	Nashua NH, 03063	+
+ Phone: 603-886-9050				Fax: 603-886-9051	+
+		All it takes is all you've got.				+
+***********************************************************************+

---

From: Bill Fenner <[email-masked]>
Date: Tue, 5 Aug 1997 17:42:18 PDT

"Brian O'Shea" <[email-masked]> wrote:
>But does [the UNIX kernel's multicast rate-limiter] do it's rate
>limiting intelligently?

It classifies packets based upon their UDP port number.  The address
allocator in sdr allocates ports giving audio high priority and video
low priority.  I've listened to lots of sessions with 64kbps audio
and 128kbps video over my 128kbps ISDN line with no audio degradation.

  Bill

---

From: Jon Knight <[email-masked]>
Date: Tue, 5 Aug 1997 19:40:12 +0100 (BST)

Tatty bye,

Jim'll

---

From: Jon Knight <[email-masked]>
Date: Tue, 5 Aug 1997 19:40:12 +0100 (BST)

Tatty bye,

Jim'll

---

From: "Brian O'Shea" <[email-masked]>
Date: Wed, 06 Aug 1997 09:46:11 -0400

-
-"Brian O'Shea" <[email-masked]> wrote:
->But does [the UNIX kernel's multicast rate-limiter] do it's rate
->limiting intelligently?
-
-It classifies packets based upon their UDP port number.  The address
-allocator in sdr allocates ports giving audio high priority and video
-low priority.  I've listened to lots of sessions with 64kbps audio
-and 128kbps video over my 128kbps ISDN line with no audio degradation.
-
-  Bill

Cool Bill!!  I'm assuming the range is as stated on Hennnings web page FAQ:

	from 	to	Application	priority
	0	16383	uncassified	lowest
	16384	32767	audio		highest
	32768	49151	whiteboard	medium
	49152	65535	video		low

However, what I meant was a little finer grained form of pruning, checking to
see that the particular video packets are grouped together by frames, and
forward complete frames, not just randomly throw away a packet as it comes in
based on rate limiting.  That's what the original bandwidth manager of the
reflector used to do, and the receiver, which required complete frames to
decode, would end up throwing away a large amount of data.  That meant that
the server was sending a lot of data that was never used, which is a real
waste of bandwidth.  So, we do some queing to try to not throw away packets
from a frame that we've already begun to transmit.

The difference is quite noticeable, 28.8 connections to servers watching 4
videos is actually possible!

This of course would be much easier for us to do if RTP had included a start
of frame bit along with the end of frame bit...  Sigh....

-bos

---

From: Karl Auerbach <[email-masked]>
Date: Thu, 7 Aug 1997 14:44:01 -0700 (PDT)

> of frame bit along with the end of frame bit...  Sigh....

RTP has a marker bit, not a "end of frame" or "start of frame" bit.

		--karl--