Subject: Re: FIX/NAP Connections

From: Peter Lothberg <[email-masked]>
Date: Sun, 11 Feb 96 6:40:06 MET

> SprintNAP:
> 
> We would like to place one of these machines at the SprintNAP to
> peer with other MBONE providers at the NAP.  It will be on a private
> FDDI behind our IDREN router.  The IDREN router is T3 connected to
> IDREN and has a FDDI port on the NAP Gigaswitch.  Our proposal is
> to use it as a NAP resource to tunnel between us and any others at
> the NAP that are interested (SprintLink, NetworkMCI, and the vBNS
> perhaps for example).  Please let me know if you would be interested
> in participating in this.

There is already a Mbone router at the Sprint-NAP, it's a Sparcstation
with a cisco FDDI interface connected to a dedicated FDDI interface on
a ICM-router. 

It is primary used as a 'gateway' between Europe and the US.

If there will be more Mbone routers in Pennsauken, it might be a good
idea of having a simple FDDI hub and a dedicated Multicast FDDI-ring?!

--Peter

---

From: Bill Fenner <[email-masked]>
Date: Mon, 12 Feb 1996 11:54:12 PST

I would like to push for native peering rather than tunnels wherever possible. 
 Native peering allows the Ethernet/FDDI/whatever-physical-medium to do your 
packet replication for you, so instead of 10 tunnels to 10 providers meaning 
10 copies of each packet, you can have a single interface on a shared medium 
and have only one copy of each packet.

Now, this has its own expenses; if your existing routers don't support 
multicast (or don't support it well) then you need to put another 
two-interface box at the NAP & give it an interface on your router like Peter 
was describing.  My vision of how things might work is something like

                                                 T3 to provider A
                    _____________          ________/___
                    | A mrouter | -------- | A router | ---- Gigaswitch port
                    -------------          ------------
                     /        \
T1 to provider B    /          \                     T3 to provider C
____/____     _____/____    ____\_____          _______/___
|B Router| -- |B Mrouter|   |C Mrouter| ------- | C router | --- Gigaswitch 
port
---------     -------\---   --/--------         ------------
                      \______/

(I hate drawing ASCII diagrams).  Basically, there is a single FDDI ring 
(well, it could even be an ethernet, given the quantity of multicast traffic 
we are currently seeing) where each mrouter has an interface and everyone 
peers for multicast traffic.  Then each of these multicast routers has a 
private interface with the provider's unicast router in order to carry tunnels 
into the provider; this could also be an ethernet, because this box should 
only have one tunnel to the provider's internal multicast topology.

There is no real reason for the seperate subnet, other than isolation of 
resources, and assuming that the gigaswitch handles multicast properly.

This kind of thing is happening at MAE-East; PSI, Digex, Sprint and Sura are 
peering natively.  I would like to see it happen at every provider 
interconnect.

  Bill

---

From: George Michaelson <[email-masked]>
Date: Tue, 13 Feb 1996 08:34:21 +1000

I think we can assume FIX-owners understand the issues well enough to
decide to do this. Down the tree a bit, Its not clear in-router support
is mature enough for a smaller player to enable this and survive the
cockups. Witness my own, and other peoples postings questioning if
cisco PIM support was viable for real deployment yet. If somebody
could clarify what can and will break for a 1 and 2 and n-way interconnect
and for use as a transit cloud into DVMRP instances...

-George

---

From: Dino Farinacci <[email-masked]>
Date: Mon, 12 Feb 1996 18:00:00 -0800

>> and for use as a transit cloud into DVMRP instances...

    Advertising what can and will break is pretty much like forecasting
    the weather. How about hearing about what does work. I count about 500 
    ciscos on the MBONE, surely there is some data there. 

    Any volunteers want to share their experiences?

Dino

---

From: Piete Brooks <[email-masked]>
Date: Tue, 13 Feb 1996 10:46:11 +0000

>     ciscos on the MBONE, surely there is some data there.

My scanner sees 585 or 34% of the Mbone -- that's more than mrouted 3.8 !

11.0PM     102  6.0%
11.0PMS     65  3.8%
11.1PMS     28  1.6%
11.1PM      12  0.7%
10.3pim    169  9.9%
10.2pim     45  2.6%
11.0Mpim    39  2.3%
10.3        78  4.6%
10.2        43  2.5%
11.0M        4  0.2%

>     Any volunteers want to share their experiences?

It's not really what George and Bill were thinking of, but here are mine.

I set up the MBone Pilot on JIPS (ac.uk) and I started drooling when routers started supporting multicast -- like NTP, they are the natural providers.
However, then came the realization that the CISCO code was based on very old DVMRP code, so had no pruning, so it was right out (pruning is a requirement on the JIPS MBone). This state of affairs lasted for what seemed a *LONG* time.
Then pruning appeared, just at the time the very old Sun4/110 which I had patched together from broken bits I had found in the scrap heap, was starting to generate memory faults fairly frequently.
With *GREAT* trepidation and lots of monitoring, I started turning on M/C.
It was back to the days of black magic !
Most things I tried failed, but I found alternative ways which worked, or just have to resign myself to not having that feature.
For example, I found that if I had a Cisco and an mrouted on the same LAN, then when the Cisco is set up with a lower metric than the mrouted, the latter shuts up, as one would hope. However, after a short interval, it doesn't see the Cisco any more, so starts forwarding packets again, so all are duplicated.
As such I have had to junk all mrouted, and *JUST* rely on the Cisco.
mtrace doesn't seem to work, which somewhat b*ggers my WWW mtrace interface :-(

However, the good news is that I haven't looked at it for months -- it just works!

Next step is to try linking two Ciscos -- directly, using DVMRP tunnels, or using a GRE tunnel. I'm not happy about using PIM "for real", but as a DVMRP router, it works just fine.

mbone.cl.cam.ac.uk runs 11.0(4), and uses DVMRP tunnels for all links.
It has a primary and a backup external feed, feeds the LAN (through some Catalyst boxes) and feeds three other departments as clients.
( to see the links pictorially, visit http://www.cl.cam.ac.uk/mbone/CAMMB.gif which is linked to from http://www.cl.cam.ac.uk/mbone/#JIPSinfo )

Router version info:
====================

Cisco Internetwork Operating System Software 
IOS (tm) GS Software (GS7-K-M), Version 11.0(4), RELEASE SOFTWARE (fc1)
Copyright (c) 1986-1995 by cisco Systems, Inc.
Compiled Mon 18-Dec-95 21:47 by alanyu
Image text-base: 0x00001000, data-base: 0x00738838

ROM: System Bootstrap, Version 11.0(3), SOFTWARE
ROM: GS Software (GS7-K-M), Version 11.0(3), RELEASE SOFTWARE (fc1)

heathrow.cl.cam.ac.uk uptime is 2 weeks, 1 day, 1 hour, 42 minutes
System restarted by reload at 08:43:01 GMT Mon Jan 29 1996
System image file is "slot0:gs7-k-mz.110-4", booted via slot0

cisco RP1 (68040) processor (revision B0) with 16384K bytes of memory.
Processor board ID 0012A6E9
G.703/E1 software, Version 1.0.
Bridging software.
X.25 software, Version 2.0, NET2, BFE and GOSIP compliant.
1 Switch Processor.
1 EIP controller (6 Ethernet).
1 AIP controller (1 ATM).
6 Ethernet/IEEE 802.3 interfaces.
1 ATM network interface.
128K bytes of non-volatile configuration memory.
4096K bytes of flash memory sized on embedded flash.
16384K bytes of Flash PCMCIA card at slot 0 (sector size 128K).
          
Configuration register is 0x2102

Relevant config info (I think ...):
===================================

ip multicast-routing
...
...
interface Tunnel0
 description noc.cam.ja.net [131.111.8.32 old mbone.cl]
 ip unnumbered Ethernet0/1
 ip pim sparse-mode
 ip multicast ttl-threshold 8
 tunnel source 131.111.8.32
 tunnel destination 193.63.107.100
 tunnel mode dvmrp
!
interface Tunnel1
 description Backup 7/24 lea.cs.ucl.ac.uk link
 ip unnumbered Ethernet0/3
 ip pim sparse-mode
 ip multicast ttl-threshold 24
 ip dvmrp metric-offset 7
 tunnel source 131.111.8.32
 tunnel destination 128.16.64.24
 tunnel mode dvmrp
!
interface Tunnel5
 description suna.hep.phy.cam
 ip unnumbered Ethernet0/1
 ip pim sparse-mode
 ip multicast ttl-threshold 4
 ip dvmrp metric-offset 2
 tunnel source 131.111.8.32
 tunnel destination 131.111.66.1
 tunnel mode dvmrp
!
interface Tunnel6
 description jay.pmms.cam (John) <[email-masked]>
 ip unnumbered Ethernet0/1
 ip pim sparse-mode
 ip multicast ttl-threshold 4
 ip dvmrp metric-offset 2
 tunnel source 131.111.8.32
 tunnel destination 131.111.24.22
 tunnel mode dvmrp
!
interface Tunnel7
 description leopard.statslab.cam
 ip unnumbered Ethernet0/1
 ip pim sparse-mode
 ip multicast ttl-threshold 4
 ip dvmrp metric-offset 2
 tunnel source 131.111.8.32
 tunnel destination 131.111.20.5
 tunnel mode dvmrp
!
interface Ethernet0/0
 description Interface to cl.cam.ac.uk (128.232.0) the departmental LAN
...
 ip pim sparse-mode
 ip multicast ttl-threshold 1
...

---

From: Petri Helenius <[email-masked]>
Date: Tue, 13 Feb 1996 14:10:09 +0200

> Next step is to try linking two Ciscos -- directly, using DVMRP tunnels, or using a GRE tunnel. I'm not happy about using PIM "for real", but as a DVMRP router, it works just fine.

A clarification... One cannot use DVMRP IP/IP tunnels to connect Cisco's
to each other but you can use Dense mode PIM over GRE/IP with or without
DVMRP unicast routing. (which pretty much does the same thing)

Pete

---

From: "David M. Meyer 503/346-1747" <[email-masked]>
Date: Tue, 13 Feb 1996 06:13:39 -0800

Well,

>> decide to do this. Down the tree a bit, 

I guess I qualify. I'm down tree (but not underwater)... 

>> Its not clear in-router support is mature enough for a smaller 
>> player to enable this and survive the cockups. 

This has not been my experience. What I've found is that 
if anything, "in-router" support has made deployment easier, 
more reliable, and more managable. BTW, I don't think it's 
legimate to mix what is essentially pilot error (e.g., injecting
your unicast routing table in the MBONE routing system) with the 
underlying technology; I suppose some student with gated and a BSDI
(linux, freeBSD, ..) box could (will) accomplish the same effect.
On the other hand, arguing that we don't yet have the administrative 
machinery that we'd like is a valid criticism.

Now, all of this is not to say that I haven't experienced problems 
(bugs), but that's usually what you trade for rapid feature evolution.
I can say that, at this point, things are very stable. In addition,
can say that my router vendor has deployed new code (and hence 
features) at a much greater rate than my workstation vendor. 

>> Witness my own, and other peoples postings questioning if

As Dino stated, this is a pretty difficult question. I don't know
what <n> you have in mind. Our transit network has n = 7 (counting 
only BGP peers). I run PIM-DM between these peers, which are FDDI
connected.  I've had little difficulty with the kind of problems you 
mention (i.e., reliability and correctness).  This is just an 
extraordinarily simple environment, and presents few (or no) problems
of the type you are asking about.

As an aside, I'll just note that I run pretty diverse regional 
multicast environment which includes SM, DM, DVMRP unicast routing, 
GRE tunneling and a few mrouters (where we graft to the global MBONE). 
This environemnt is also very reliable.

Dave

-- 

David M. Meyer                  Voice:    +1 541.346.1747
Senior Network Engineer         SkyPager: +1 800.580.7929
Office of University Computing  Cellular: +1 541.954.1103
Computing Center                FAX:      +1 541.346.4397
University of Oregon            Internet: [email-masked]
1225 Kincaid			URL:	  http://ns.uoregon.edu/~meyer
Eugene, OR 97403

---

From: George Michaelson <[email-masked]>
Date: Wed, 14 Feb 1996 08:54:43 +1000

So what I cull so far:

	(1) run 11.0(4) or better.

The decision to run 11 should be pretty moot by now. I know lots of
people are still on 10.2(4) but if you want things like MPP you'd need
to be above that anyway.

	(2) you can use DVMRP, PIM sparce, PIM DENSE all together
	    and its really not that hard.

I retain some confusion about when you choose which and why. The GRE 
tunnelling is cisco-cisco only for anybody who missed that bit, and is
how you will ship multicast along your own internal point-to-point fabric.

Presumably any time you need to inject an IP tunnel into a mrouted box its
DVMRP (and thats where the unicast root explosion problem is going to crop
up) and probably PIM dense on local fast (LAN?) media, PIM sparce on any
other instance.

	(3) having FIX-like setups where ISP's meet on truly multicast
	    capable media is much much simpler than hand-crafting 1:1
	    links. 

Australia has nothing like this. Many of us wish it did. Market
conditions do not seem to allow for any of the players deciding
to take the first step. Why? who knows. Maybe after 1997 it'll be different.

Optimal values for ttl, scoping rules, rate limiting aren't covered yet.
(does the cisco do scoping and rate limiting filtering?)

Inter-vendor router connectivity isn't mentioned. Wellfleet have DVMRP but
I don't know if its released code yet.

Quite why gated-multicast exists isn't discussed yet. I presume its intended
to be an external "smart" directing some other routing-active box into
doing good things.

Pietes example uses some values that I found surprising but he looks to
have a fair mix of superjanet/campus/other feeds in there.

I'm sorry if my wording was less than charitable to you Dino, looking
back over the last 6-9 months of commentary on PIM/Cisco there has been
more than a little criticism of things, and rightly or wrongly I've let
that act as FUD in my decision to deploy mrouted/Solaris in-house. 

Thanks for all input to date...

-George

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 13 Feb 1996 17:30:26 -0800

>> people are still on 10.2(4) but if you want things like MPP you'd need

    And by the way, we aren't putting all bugfixes into 10.2. Only critical
    fixes. I urge people on 10.2 to upgrade at least to 10.3 and most
    preferably 11.0. 11.0 gives everyone the very needed mtrace capability.

>> up) and probably PIM dense on local fast (LAN?) media, PIM sparce on any

    As I mentioned so many times. The explosion only occurs when you light
    the bomb. By default, your site's network number(s) are injected, that
    is typically a few classful routes. You get your whole site ready for 
    multicast and you conserve DVMRP routing table space due to the 
    auto-summarization.

>> (does the cisco do scoping and rate limiting filtering?)

    Yes. See ftp://ftp.cisco.com/ipmulticast.html for more details.

>> I don't know if its released code yet.

    We demostrated DVMRP interoperability with Bay routers during the RSVP
    Interop showcase last year. It was over a serial PPP link.

>> to be an external "smart" directing some other routing-active box into

    There exists a PIM and DVMRP capable gated. See Thomas Pusateri or Merit
    for details.

>> that act as FUD in my decision to deploy mrouted/Solaris in-house. 

    No problem, no skin off my back.

Dino

---

From: Petri Helenius <[email-masked]>
Date: Wed, 14 Feb 1996 08:55:31 +0200

> >> that act as FUD in my decision to deploy mrouted/Solaris in-house. 
Being involved in the discussion on this list that occurred a while back
regarding the up's and down's of both implementations and protocol
design of the two widely used multicast routing engines (Cisco IOS w PIM
and mrouted w DVMRP) and their interoperability and co-existence issues,
I would say that there is severe lack of momentum of change in many
places where IP multicasting has already been deployed. This also seems
a great field for FUD since the information how this holds together is
in quite few. Specially I see things in the area of "IP multicast is bad
for your network since it consumes bandwidth" at the same time the same
people are running conferencing applications over replicated unicast.
(and of course having no problem with the bandwidth)

And I do agree on that the implementors in many cases have better things
to do than fight their cause in a public forum. I would still say that
outright FUD should be shot down as soon as it appears as a cloud of
smoke.

Pete