Subject: mtrace and Cisco PIMs

From: Phil Pishioneri <[email-masked]>
Date: Mon, 22 Jun 1998 09:35:01 -0400

I've heard that there are problems when trying to mtrace through
a Cisco PIM cloud.  Does anyone know if this is true and what the
problem might be?

-Phil		     Cornell Information Technologies

---

From: Kevin Thompson <[email-masked]>
Date: Mon, 22 Jun 1998 14:27:46 -0400

Good question.

Someone at Cisco had once told me
that mtrace essentially is
not supported thru their routers, but I don't
recall exactly why. He also said there were no
plans at Cisco to do anything about it.

I've seen lots of mtraces break going
thru our PIM cloud eventhough mcast routing state
was correct, and I've seen incorrect loss #s being
reported in the mtrace results. Of course, this
makes debugging real mcast problems a bit more
challenging :)

Kevin Thompson
vBNS Engineering

---

From: [email-masked]
Date: Mon, 22 Jun 1998 21:09:47 +0200

> Someone at Cisco had once told me that mtrace essentially is
> not supported thru their routers, but I don't recall exactly
> why. He also said there were no plans at Cisco to do anything
> about it.

Do you know what he meant when he said "essentially is not
supported"?  For unsupported functionality I'd say it works quite
well...

> I've seen lots of mtraces break going thru our PIM cloud
> eventhough mcast routing state was correct [...]

Well, as far as I can tell it has "always" (?) worked fine for
us; here is a recent example:

Mtrace from 128.223.83.24 to 129.241.100.134 via group 224.2.186.225
Querying full reverse path... 
  0  vader.runit.sintef.no (129.241.100.134)
 -1  sf-gw.sintef.no (129.241.225.1)  PIM  thresh^ 0  
 -2  sintef-gw.sintef.no (129.241.249.1)  PIM  thresh^ 0  
 -3  trdS-gw.uninett.no (129.241.181.15)  PIM  thresh^ 0  
 -4  osloS-gw.uninett.no (128.39.46.1)  PIM  thresh^ 0  
 -5  no-gw2.nordu.net (128.39.0.141)  PIM-special  thresh^ 0  RP/Core
 -6  sauce-ether.uio.no (129.240.140.21)  DVMRP  thresh^ 6  
 -7  stockholm.mbone.ebone.net (192.36.148.206)  DVMRP  thresh^ 40  
 -8  schnell.ebone.net (192.36.137.1)  PIM-special  thresh^ 64  
 -9  dec3800-1-fddi-0.WestOrange.mci.net (204.70.64.29)  DVMRP  thresh^ 32  
-10  dec3800-2-fddi-1.WestOrange.mci.net (204.70.64.77)  DVMRP  thresh^ 1  
-11  cambridge1-mbone1.bbnplanet.net (199.94.207.2)  PIM-special  thresh^ 32  
-12  Loopback0.MB1.BOS1.Alter.Net (137.39.3.65)  PIM/BGP4+  thresh^ 32  
-13  Loopback0.MB1.DCA1.Alter.Net (137.39.3.91)  PIM/BGP4+  thresh^ 0  
-14  Loopback0.MB1.EWR1.Alter.Net (137.39.3.92)  PIM/BGP4+  thresh^ 0  
-15  MBGP-UO.Mbone.Alter.Net (137.39.26.98)  PIM  thresh^ 32  RP/Core
-16  cisco10-gw.uoregon.edu (128.223.3.10)  PIM  thresh^ 0  
-17  rock.uoregon.edu (128.223.83.24)
Round trip time 490 ms

> [...], and I've seen incorrect loss #s being reported in the
> mtrace results. Of course, this makes debugging real mcast
> problems a bit more challenging :)

Well, that's another matter.  I've seen negative packet counts
etc. (see below), and that probably isn't quite right. ;-)

- Havard

------------------------------

Waiting to accumulate statistics... Results after 9 seconds:

  Source        Response Dest    Packet Statistics For     Only For Traffic
128.223.83.24   129.241.100.134  All Multicast Traffic     From 128.223.83.24
     v       __/  rtt  299 ms    Lost/Sent = Pct  Rate       To 224.2.186.225
128.223.83.1   
128.223.3.10    cisco10-gw.uoregon.edu 
     v     ^      ttl    0    -1746/955  = --% 106 pps     0/237  =  0%  26 pps
128.223.3.7    
137.39.26.98    MBGP-UO.Mbone.Alter.Net RP/Core
     v     ^      ttl   32     1194/2095 = 57% 232 pps    18/237  =  8%  26 pps
137.39.26.97   
137.39.3.92     Loopback0.MB1.EWR1.Alter.Net 
     v     ^      ttl   33        0/869  =  0%  96 pps     0/219  =  0%  24 pps
137.39.3.91     Loopback0.MB1.DCA1.Alter.Net 
     v     ^      ttl   34      -10/924  =  0% 102 pps    -1/219  =  0%  24 pps
137.39.3.65     Loopback0.MB1.BOS1.Alter.Net 
     v     ^      ttl   35        0/1444 =  0% 160 pps    -1/220  =  0%  24 pps
199.94.207.2    cambridge1-mbone1.bbnplanet.net 
     v     ^      ttl   36      617/2090 = 30% 232 pps     0/221  =  0%  24 pps
204.70.64.61   
204.70.64.77    dec3800-2-fddi-1.WestOrange.mci.net 
     v     ^      ttl   37        2/1731 =  0% 192 pps     0/221  =  0%  24 pps
204.70.64.45   
204.70.64.29    dec3800-1-fddi-0.WestOrange.mci.net 
     v     ^      ttl   38       66/3788 =  2% 420 pps     4/221  =  2%  24 pps
192.36.137.1    schnell.ebone.net 
     v     ^      ttl   64       -4/3761 =  0% 417 pps     1/217  =  0%  24 pps
192.36.148.206  stockholm.mbone.ebone.net 
     v     ^      ttl   65        2/4124 =  0% 458 pps     0/216  =  0%  24 pps
129.240.140.21  sauce-ether.uio.no 
     v     ^      ttl   66        1/4759 =  0% 528 pps     0/216  =  0%  24 pps
128.39.0.85    
128.39.0.141    no-gw2.nordu.net RP/Core
     v     ^      ttl   67        0/509  =  0%  56 pps     0/216  =  0%  24 pps
128.39.0.142   
128.39.46.1     osloS-gw.uninett.no 
     v     ^      ttl   68       -1/250  =  0%  27 pps    -1/216  =  0%  24 pps
128.39.46.2    
129.241.181.15  trdS-gw.uninett.no 
     v     ^      ttl   69     -127/251  =-50%  27 pps     0/217  =  0%  24 pps
129.241.181.4  
129.241.249.1   sintef-gw.sintef.no 
     v     ^      ttl   70        1/251  =  0%  27 pps     0/217  =  0%  24 pps
129.241.249.2  
129.241.225.1   sf-gw.sintef.no 
     v      \__   ttl   71          220         24 pps       217         24 pps
129.241.100.134 129.241.100.134
  Receiver      Query Source

---

From: John Meylor <[email-masked]>
Date: Mon, 22 Jun 1998 12:39:39 -0700

Kevin Thompson wrote:
> not supported thru their routers, 

Mtrace is supported through and from cisco routers.

there were no
> plans at Cisco to do anything about it.

There is ongoing work to improve mtrace support as well
as other measures of multicast management in general. 
We consider this very important.

We're very interested in any problems people are having
with mtrace in their cisco topology and comments on what
could be improved.

- John

---

From: "John M. Zwiebel" <[email-masked]>
Date: Mon, 22 Jun 1998 12:44:18 -0700

There are some versions of IOS that have mtrace problems.  You can
look them up on www.cisco.com.  

The negative packet counts may result from duplicates on that network.

 ^ > Someone at Cisco had once told me that mtrace essentially is
 ^ > not supported thru their routers, but I don't recall exactly
 ^ > why. He also said there were no plans at Cisco to do anything
 ^ > about it.
 ^ 
 ^ Do you know what he meant when he said "essentially is not
 ^ supported"?  For unsupported functionality I'd say it works quite
 ^ well...
 ^ 
 ^ > I've seen lots of mtraces break going thru our PIM cloud
 ^ > eventhough mcast routing state was correct [...]
 ^ 
 ^ Well, as far as I can tell it has "always" (?) worked fine for
 ^ us; here is a recent example:
 ^ 
 ^ Mtrace from 128.223.83.24 to 129.241.100.134 via group 224.2.186.225
 ^ Querying full reverse path... 
 ^   0  vader.runit.sintef.no (129.241.100.134)
 ^  -1  sf-gw.sintef.no (129.241.225.1)  PIM  thresh^ 0  
 ^  -2  sintef-gw.sintef.no (129.241.249.1)  PIM  thresh^ 0  
 ^  -3  trdS-gw.uninett.no (129.241.181.15)  PIM  thresh^ 0  
 ^  -4  osloS-gw.uninett.no (128.39.46.1)  PIM  thresh^ 0  
 ^  -5  no-gw2.nordu.net (128.39.0.141)  PIM-special  thresh^ 0  RP/Core
 ^  -6  sauce-ether.uio.no (129.240.140.21)  DVMRP  thresh^ 6  
 ^  -7  stockholm.mbone.ebone.net (192.36.148.206)  DVMRP  thresh^ 40  
 ^  -8  schnell.ebone.net (192.36.137.1)  PIM-special  thresh^ 64  
 ^  -9  dec3800-1-fddi-0.WestOrange.mci.net (204.70.64.29)  DVMRP  thresh^ 32  
 ^ -10  dec3800-2-fddi-1.WestOrange.mci.net (204.70.64.77)  DVMRP  thresh^ 1  
 ^ -11  cambridge1-mbone1.bbnplanet.net (199.94.207.2)  PIM-special  thresh^ 32
      
 ^ -12  Loopback0.MB1.BOS1.Alter.Net (137.39.3.65)  PIM/BGP4+  thresh^ 32  
 ^ -13  Loopback0.MB1.DCA1.Alter.Net (137.39.3.91)  PIM/BGP4+  thresh^ 0  
 ^ -14  Loopback0.MB1.EWR1.Alter.Net (137.39.3.92)  PIM/BGP4+  thresh^ 0  
 ^ -15  MBGP-UO.Mbone.Alter.Net (137.39.26.98)  PIM  thresh^ 32  RP/Core
 ^ -16  cisco10-gw.uoregon.edu (128.223.3.10)  PIM  thresh^ 0  
 ^ -17  rock.uoregon.edu (128.223.83.24)
 ^ Round trip time 490 ms
 ^ 
 ^ > [...], and I've seen incorrect loss #s being reported in the
 ^ > mtrace results. Of course, this makes debugging real mcast
 ^ > problems a bit more challenging :)
 ^ 
 ^ Well, that's another matter.  I've seen negative packet counts
 ^ etc. (see below), and that probably isn't quite right. ;-)
 ^ 
 ^ 
 ^ - Havard
 ^ 
 ^ ------------------------------
 ^ 
 ^ Waiting to accumulate statistics... Results after 9 seconds:
 ^ 
 ^   Source        Response Dest    Packet Statistics For     Only For Traffic
 ^ 128.223.83.24   129.241.100.134  All Multicast Traffic     From 128.223.83.2
    4
 ^      v       __/  rtt  299 ms    Lost/Sent = Pct  Rate       To 224.2.186.22
    5
 ^ 128.223.83.1   
 ^ 128.223.3.10    cisco10-gw.uoregon.edu 
 ^      v     ^      ttl    0    -1746/955  = --% 106 pps     0/237  =  0%  26 
    pps
 ^ 128.223.3.7    
 ^ 137.39.26.98    MBGP-UO.Mbone.Alter.Net RP/Core
 ^      v     ^      ttl   32     1194/2095 = 57% 232 pps    18/237  =  8%  26 
    pps
 ^ 137.39.26.97   
 ^ 137.39.3.92     Loopback0.MB1.EWR1.Alter.Net 
 ^      v     ^      ttl   33        0/869  =  0%  96 pps     0/219  =  0%  24 
    pps
 ^ 137.39.3.91     Loopback0.MB1.DCA1.Alter.Net 
 ^      v     ^      ttl   34      -10/924  =  0% 102 pps    -1/219  =  0%  24 
    pps
 ^ 137.39.3.65     Loopback0.MB1.BOS1.Alter.Net 
 ^      v     ^      ttl   35        0/1444 =  0% 160 pps    -1/220  =  0%  24 
    pps
 ^ 199.94.207.2    cambridge1-mbone1.bbnplanet.net 
 ^      v     ^      ttl   36      617/2090 = 30% 232 pps     0/221  =  0%  24 
    pps
 ^ 204.70.64.61   
 ^ 204.70.64.77    dec3800-2-fddi-1.WestOrange.mci.net 
 ^      v     ^      ttl   37        2/1731 =  0% 192 pps     0/221  =  0%  24 
    pps
 ^ 204.70.64.45   
 ^ 204.70.64.29    dec3800-1-fddi-0.WestOrange.mci.net 
 ^      v     ^      ttl   38       66/3788 =  2% 420 pps     4/221  =  2%  24 
    pps
 ^ 192.36.137.1    schnell.ebone.net 
 ^      v     ^      ttl   64       -4/3761 =  0% 417 pps     1/217  =  0%  24 
    pps
 ^ 192.36.148.206  stockholm.mbone.ebone.net 
 ^      v     ^      ttl   65        2/4124 =  0% 458 pps     0/216  =  0%  24 
    pps
 ^ 129.240.140.21  sauce-ether.uio.no 
 ^      v     ^      ttl   66        1/4759 =  0% 528 pps     0/216  =  0%  24 
    pps
 ^ 128.39.0.85    
 ^ 128.39.0.141    no-gw2.nordu.net RP/Core
 ^      v     ^      ttl   67        0/509  =  0%  56 pps     0/216  =  0%  24 
    pps
 ^ 128.39.0.142   
 ^ 128.39.46.1     osloS-gw.uninett.no 
 ^      v     ^      ttl   68       -1/250  =  0%  27 pps    -1/216  =  0%  24 
    pps
 ^ 128.39.46.2    
 ^ 129.241.181.15  trdS-gw.uninett.no 
 ^      v     ^      ttl   69     -127/251  =-50%  27 pps     0/217  =  0%  24 
    pps
 ^ 129.241.181.4  
 ^ 129.241.249.1   sintef-gw.sintef.no 
 ^      v     ^      ttl   70        1/251  =  0%  27 pps     0/217  =  0%  24 
    pps
 ^ 129.241.249.2  
 ^ 129.241.225.1   sf-gw.sintef.no 
 ^      v      \__   ttl   71          220         24 pps       217         24 
    pps
 ^ 129.241.100.134 129.241.100.134
 ^   Receiver      Query Source
 ^ 

------------------------------------------------------------------------------
	John Zwiebel                       Phone: 408-526-5303
	Cisco Systems Inc.                 
	Multimedia Group

---

From: "Charles Spurgeon" <[email-masked]>
Date: Mon, 22 Jun 1998 16:23:53 -0500 (CDT)

>etc. (see below), and that probably isn't quite right. ;-)

If I recall correctly, the speaker at the "Troubleshooting IP
Multicast" session at the Denver Networker's conference held last week
mentioned that negative packet counts in the "from/to" column of mstat
output indicates duplicate packets.  

Mstat does two trace runs and then compares the differences to derive
the loss rates and pps displays.  Given that multicast is UDP,
duplicate packets are not unexpected.

He also mentioned that when you see negative time deltas in the "hop"
display that basically means that the routers being probed by mstat
are not synced to NTP, so you're seeing clock skew between the various
routers in the path.

-Charles

Charles E. Spurgeon
ACITS/Networking Services			
[email-masked]
512.475.9265