Subject: mrouted IPIP bombing

From: Ronan KERYELL <[email-masked]>
Date: Thu, 13 Nov 1997 12:04:49 +0100

Because of a current topological instability in the French Mbone, I
use another feed for my site.
A side effect is that the peer site is saturating our Internet access
with 512 kbit/s IPIP packets. By adding a rate_limit I can limit the
IPIP bombing to this limit... Not very cute...

I've investigated with some debug flags (from -d 3 up to -d pkt,prunes,route_detail,peers,cache,callout,interface,groups,traceroute,igmp,rsrr)
but it looks like no debug information is given for received IP in IP
packet. Only igmp packets are dumped... Is there a way to have
information on all these IPIP packets ?

Looks like there is no pruning from some groups. But which one ?

Here is a sample of /var/tmp/mrouted.cache :
Multicast Routing Cache Table (75 entries)
 Origin             Mcast-group         CTmr     Age      Ptmr Rx IVif Forwvifs
<(prunesrc:vif[idx]/tmr) prunebitmap
>Source             Lifetime SavPkt     Pkts     Bytes    RPFf
 131.136.234.106/32 224.2.172.238    0:05:23  0:00:09        - -1
>131.136.234.106
 199.94.220.184/32  224.2.0.1        0:03:46  0:00:31        - -1
>199.94.220.184
 164.58.253.9/32    224.2.2.2        0:02:15  0:00:58        - -1
>164.58.253.9
 130.240.112.8/32   224.2.172.238    0:03:02  0:01:22        - -1
>130.240.112.8
 199.94.220.184/32  224.2.20.165     0:05:48  0:01:23        - -1
>199.94.220.184
 200.245.220.33/32  224.0.1.1        0:03:13  0:01:37        - -1
>200.245.220.33
 143.106.51.1/32    224.2.240.109    0:04:56  0:01:47        - -1
>143.106.51.1
[...]
Strangely, there are many "-" as Ptmr saying that no prunes have been
sent upstream... :-(

By the way, why does mrinfo give a 3.255 version number for
mrouted3.9beta2? mrinfo gives wrong leaf information for
mrouted3.9beta2.

On my host (chailly.ensmp.fr) :
mrinfo 
127.0.0.1 (localhost) [version 3.255,prune,genid,mtrace]:
  192.54.172.200 -> 0.0.0.0 (local) [1/1/querier/leaf]
  192.54.172.201 -> 0.0.0.0 (local) [1/1/querier/leaf]
  192.54.172.202 -> 0.0.0.0 (local) [1/1/querier/leaf]
  192.54.172.203 -> 0.0.0.0 (local) [1/1/querier/leaf]
  192.54.172.204 -> 0.0.0.0 (local) [1/1/querier/leaf]
  192.54.172.200 -> 157.159.10.11 (babylone.int-evry.fr) [1/32/tunnel/leaf]
  192.54.172.200 -> 193.54.112.94 (seychelles.cma.fr) [3/15/tunnel/leaf]
  192.54.172.200 -> 192.54.165.171 (calloway) [6/7/tunnel/down/leaf]
  192.54.172.200 -> 129.104.30.50 (soleil.polytechnique.fr) [3/32/tunnel/down/leaf]

On the peer :
mrinfo seychelles.cma.fr
193.54.112.94 (seychelles.cma.fr) [version 3.255,prune,genid,mtrace]:
  193.54.112.94 -> 0.0.0.0 (local) [1/1/querier/leaf]
  193.54.112.94 -> 193.51.208.8 (sobone.inria.fr) [1/32/tunnel/leaf]
  193.54.112.94 -> 192.54.172.200 (chailly) [3/15/tunnel/leaf]

mrinfo sobone.inria.fr
193.51.208.8 (sobone.inria.fr) [version 3.255,prune,genid,mtrace]:
  193.51.208.8 -> 0.0.0.0 (local) [1/1/querier/leaf]
  138.96.152.17 -> 138.96.152.64 (pacman.inria.fr) [1/1/querier/leaf]
  193.51.208.8 -> 193.51.208.130 (plvxv6.inria.fr) [1/14/tunnel/down/leaf]
  193.51.208.8 -> 192.35.82.97 (MBONE.CIT.CORNELL.EDU) [12/32/tunnel/leaf]
  193.51.208.8 -> 192.93.1.34 (oreste.inria.fr) [1/32/tunnel/down/leaf]
  193.51.208.8 -> 192.144.75.1 (192.144.75.1) [4/32/tunnel/down/leaf]
  193.51.208.8 -> 193.55.113.141 (eurecom.eurecom.fr) [1/32/tunnel/leaf]
  193.51.208.8 -> 193.54.112.94 (seychelles.cma.fr) [1/32/tunnel/leaf]
  193.51.208.8 -> 134.59.1.17 (graffiti.unice.fr) [1/32/tunnel/leaf]

Neither sobone.inria.fr, seychelles.cma.fr or chailly.ensmp.fr are
leaves... 

I am a bit puzzled...

Is it a side effect of the current frequent route changes that happen
now ?

-- 
    Ronan KERYELL
    Centre de Recherche en Informatique
    de l'École Nationale Supérieure des Mines de Paris (CRI-ENSMP)
    35, Rue Saint-Honoré
    77305 FONTAINEBLEAU CEDEX
    FRANCE                               Tel:    (+33|0) 1.64.69.48.44
                                         Fax:    (+33|0) 1.64.69.47.09
                                         E-mail: [email-masked]
                                         http://www.cri.ensmp.fr/~keryell

---

From: Bill Fenner <[email-masked]>
Date: Thu, 13 Nov 1997 08:50:28 PST

>information on all these IPIP packets ?

No; you have to use "tcpdump" or other network snooper.

The -1 means there is no route for the traffic being received (so mrouted
has no idea where to send a prune).

>mrouted3.9beta2? mrinfo gives wrong leaf information for

mrinfo hasn't been updated yet to know that 3.255 is the version number
that DVMRPv3-spec-compliant routers report.  The leaf reporting is a bug
which will be fixed in the next mrouted beta.

>Is it a side effect of the current frequent route changes that happen

I wouldn't think so.  Are both sides of the problem tunnel marked as up?
When you kill -USR1 `cat /etc/mrouted.pid`, are there routes in
/usr/tmp/mrouted.dump?

  Bill

---

From: Ronan KERYELL <[email-masked]>
Date: Fri, 14 Nov 1997 16:47:07 +0100

>>>>> On Thu, 13 Nov 1997 08:50:28 PST, Bill Fenner <[email-masked]> said:

    Bill> No; you have to use "tcpdump" or other network snooper.

I've picked the latest tcpdump version and yes, it's great, I can see
inside IPIP with something like 
/usr/local/sbin/tcpdump -v -s 2000 host seychelles.cma.fr and ip proto ipip

15:29:45.787737 seychelles.cma.fr > chailly: falcon.pilsnet.sunet.se.1095 > 224.2.189.198.19122: udp 176 (ttl 110, id 30956) (ttl 53, id 29907)
15:29:46.503308 seychelles.cma.fr > chailly: falcon.pilsnet.sunet.se.1095 > 224.2.189.198.19122: udp 176 (ttl 110, id 30971) (ttl 53, id 29915)
15:29:46.589490 seychelles.cma.fr > chailly: falcon.pilsnet.sunet.se.1095 > 224.2.189.198.19122: udp 176 (ttl 110, id 30977) (ttl 53, id 29916)
15:29:46.593461 seychelles.cma.fr > chailly: falcon.pilsnet.sunet.se.1095 > 224.2.189.198.19122: udp 176 (ttl 110, id 30976) (ttl 53, id 29917)
15:29:46.722960 seychelles.cma.fr > chailly: falcon.pilsnet.sunet.se.1095 > 224.2.189.198.19122: udp 176 (ttl 110, id 30978) (ttl 53, id 29918)
15:29:47.190383 seychelles.cma.fr > chailly: falcon.pilsnet.sunet.se.1095 > 224.2.189.198.19122: udp 176 (ttl 110, id 30991) (ttl 53, id 29919)
15:29:47.424533 seychelles.cma.fr > chailly: mbone.Sun.COM.17755 > 224.2.170.73.17755: udp 60 (ttl 114, id 22530) (ttl 53, id 29920)
15:29:47.534432 seychelles.cma.fr > chailly: falcon.pilsnet.sunet.se.1095 > 224.2.189.198.19122: udp 176 (ttl 110, id 30997) (ttl 53, id 29921)
15:29:47.540523 seychelles.cma.fr > chailly: falcon.pilsnet.sunet.se.1095 > 224.2.189.198.19122: udp 176 (ttl 110, id 30998) (ttl 53, id 29922)
15:29:47.595550 seychelles.cma.fr > chailly: falcon.pilsnet.sunet.se.1095 > 224.2.189.198.19122: udp 176 (ttl 110, id 31002) (ttl 53, id 29923)

    Bill> The -1 means there is no route for the traffic being
    Bill> received (so mrouted has no idea where to send a prune).

Sad.
Of course I cannot mtrace directly them since there is no route... :-(

    >> Is it a side effect of the current frequent route changes that
    >> happen now ?

    Bill> I wouldn't think so.  Are both sides of the problem tunnel
    Bill> marked as up?  When you kill -USR1 `cat /etc/mrouted.pid`,
    Bill> are there routes in /usr/tmp/mrouted.dump?

Yes, both side are marked up :
chailly-keryell > mrinfo seychelles.cma.fr
193.54.112.94 (seychelles.cma.fr) [version 3.255,prune,genid,mtrace]:
  193.54.112.94 -> 0.0.0.0 (local) [1/1/querier/leaf]
  193.54.112.94 -> 193.51.208.8 (sobone.inria.fr) [1/32/tunnel/leaf]
  193.54.112.94 -> 192.54.172.200 (chailly) [1/15/tunnel/leaf]
chailly-keryell > mrinfo chailly
192.54.172.200 (chailly) [version 3.255,prune,genid,mtrace]:
  192.54.172.200 -> 0.0.0.0 (local) [1/1/querier/leaf]
  192.54.172.201 -> 0.0.0.0 (local) [1/1/querier/leaf]
  192.54.172.202 -> 0.0.0.0 (local) [1/1/querier/leaf]
  192.54.172.203 -> 0.0.0.0 (local) [1/1/querier/leaf]
  192.54.172.204 -> 0.0.0.0 (local) [1/1/querier/leaf]
  192.54.172.200 -> 193.54.112.94 (seychelles.cma.fr) [1/15/tunnel/leaf]

What happens if a route has a distance greater than 32 ? Could we
imagine a situation with a far machine sending packet up to me ? If so
could my mrouted send a prune message ?  What happens if some tunnels
are misconfigured and have an assymetrical metric ? May this lead to
assymetrical (non-)pruning ?

Typically here are the machines :

on seychelles.cma.fr a grep 224.2.189.198 shows in mrouted debug
output :

Seychelles# grep 224.2.189.198 /tmp/mrouted
15:27:50.936 add cache entry (192.36.125.68 224.2.189.198) gm:4, parent-vif:1
15:27:52.598 add cache entry (192.36.125.164 224.2.189.198) gm:4, parent-vif:1
15:28:03.695 add cache entry (192.9.9.71 224.2.189.198) gm:4, parent-vif:1
15:28:04.173 add cache entry (130.240.2.156 224.2.189.198) gm:0, parent-vif:-1
15:28:17.722 del_table_entry deleting (192/8 224.2.189.198)
15:28:17.723 cleaning cache for group 224.2.189.198
15:28:18.434 add cache entry (192.36.125.68 224.2.189.198) gm:0, parent-vif:-1
15:28:19.248 add cache entry (192.36.125.164 224.2.189.198) gm:0, parent-vif:-1
15:28:19.839 add cache entry (130.240.64.16 224.2.189.198) gm:0, parent-vif:-1
15:28:39.941 add cache entry (129.99.50.54 224.2.189.198) gm:4, parent-vif:1
15:28:40.122 192.54.172.200 on vif 2 prunes (129.99.50.0 224.2.189.198)/7590
15:28:40.124 prune (129.99.50/24 224.2.189.198), stop sending on vif 2, gm:0
15:28:40.127 sent prune for (129.99.50/24 224.2.189.198)/5282 on vif 1 to 193.51.208.8
15:28:42.190 rexmit_prune deleting (129.99.50.54 224.2.189.198) (next is 6 sec)
15:29:02.129 add cache entry (192.36.125.68 224.2.189.198) gm:4, parent-vif:1
15:29:32.337 del_table_entry deleting (129.99.50/24 224.2.189.198)
15:29:32.338 cleaning cache for group 224.2.189.198
15:29:42.331 add cache entry (192.36.125.164 224.2.189.198) gm:4, parent-vif:1
15:30:48.413 add cache entry (129.99.50.54 224.2.189.198) gm:4, parent-vif:1
15:30:48.500 192.54.172.200 on vif 2 prunes (129.99.50.0 224.2.189.198)/5359
15:30:48.502 prune (129.99.50/24 224.2.189.198), stop sending on vif 2, gm:0
15:30:48.505 sent prune for (129.99.50/24 224.2.189.198)/5359 on vif 1 to 193.51.208.8
15:30:49.500 rexmit_prune deleting (129.99.50.54 224.2.189.198) (next is 6 sec)
15:31:45.207 (192.9.9/24 224.2.189.198) timed out, checking for traffic
15:31:45.209 refresh lifetim of cache entry (trfc flow) (192.9.9/24, 224.2.189.198)
15:32:16.145 add cache entry (192.36.125.76 224.2.189.198) gm:4, parent-vif:1
15:32:17.344 del_table_entry deleting (192/8 224.2.189.198)
15:32:17.345 cleaning cache for group 224.2.189.198
15:32:17.428 add cache entry (192.36.125.68 224.2.189.198) gm:0, parent-vif:-1
15:32:22.587 add cache entry (192.36.125.164 224.2.189.198) gm:0, parent-vif:-1
15:32:23.885 add cache entry (192.36.125.76 224.2.189.198) gm:0,
parent-vif:-1

Seychelles# grep 224.2.189.198 /var/tmp/mrouted*
/var/tmp/mrouted.cache: 130.240.2.156/32   224.2.189.198    0:02:11  0:00:43        - -1
/var/tmp/mrouted.cache: 130.240.64.16/32   224.2.189.198    0:02:48  0:02:03        - -1
/var/tmp/mrouted.cache: 129.99.50/24       224.2.189.198    0:02:19  0:01:07  1:09:56  1  1P  2p
/var/tmp/mrouted.cache: 192.9.9/24         224.2.189.198    0:04:36  0:08:08        -  -  1   2 
/var/tmp/mrouted.cache: 192/8              224.2.189.198    0:04:06  0:03:09        -  -  1   2 

What this 192/8 martian is ?

on chailly.ensmp.fr a grep 224.2.189.198 shows in :

chailly-keryell > grep 224.2.189.198 /tmp/mrouted
15:27:51.741 add cache entry (192.36.125.68 224.2.189.198) gm:0, parent-vif:-1
15:27:52.731 add cache entry (192.36.125.164 224.2.189.198) gm:0, parent-vif:-1
15:28:03.865 add cache entry (192.9.9.71 224.2.189.198) gm:0, parent-vif:-1
15:28:40.109 add cache entry (129.99.50.54 224.2.189.198) gm:0, parent-vif:5
15:28:40.110 sent prune for (129.99.50/24 224.2.189.198)/7590 on vif 5 to 193.54.112.94
15:28:42.389 rexmit_prune deleting (129.99.50.54 224.2.189.198) (next is 6 sec)
15:29:35.962 del_table_entry deleting (129.99.50/24 224.2.189.198)
15:29:35.962 cleaning cache for group 224.2.189.198
15:30:48.595 add cache entry (129.99.50.54 224.2.189.198) gm:0, parent-vif:5
15:30:48.596 sent prune for (129.99.50/24 224.2.189.198)/5359 on vif 5 to 193.54.112.94
15:30:49.680 rexmit_prune deleting (129.99.50.54 224.2.189.198) (next is 6 sec)
15:31:46.939 add cache entry (192.36.125.164 224.2.189.198) gm:0, parent-vif:-1
15:32:16.286 add cache entry (192.36.125.76 224.2.189.198) gm:0, parent-vif:-1
15:32:54.598 add cache entry (192.9.9.71 224.2.189.198) gm:0,
parent-vif:-1

chailly-keryell > grep 224.2.189.198 /var/tmp/mrouted*
/var/tmp/mrouted.cache: 192.9.9.71/32      224.2.189.198    0:04:12  0:01:37        - -1
/var/tmp/mrouted.cache: 192.36.125.76/32   224.2.189.198    0:00:49  0:02:15        - -1
/var/tmp/mrouted.cache: 192.36.125.164/32  224.2.189.198    0:00:38  0:02:45        - -1
/var/tmp/mrouted.cache: 192.36.125.68/32   224.2.189.198    0:00:31  0:06:40        - -1

Note that I've added a deny 192/8 and deny 10/8 in the mrouted.conf of
chailly.ensmp.fr to remove the martians above. But no improvement.

Hum... Where are all these routes getting lost ? :-(

chailly-keryell > mtrace -g seychelles.cma.fr falcon.pilsnet.sunet.se chailly.ensmp.fr
Mtrace from 192.36.125.68 to 192.54.172.200 via group 224.2.0.1
Querying full reverse path... * switching to hop-by-hop:
  0  chailly (192.54.172.200)
 -1  seychelles.cma.fr (193.54.112.94)  DVMRP  thresh^ 0  Wrong interface
 -2  sobone.inria.fr (193.51.208.8)  DVMRP  thresh^ 32  
 -3  MBONE.CIT.CORNELL.EDU (192.35.82.97)  DVMRP  thresh^ 32  
 -4  mbone.appliedtheory.com (169.130.122.18)  DVMRP  thresh^ 32  
 -5  pen-mbone-1.sprintlink.net (207.41.200.99)  DVMRP  thresh^ 16  
 -6  * * * nrl-mbone.dren.net (132.250.230.10) [mrouted 3.255] didn't respond
 -7  
Resuming...
 -6  nrl-mbone.dren.net (132.250.230.10)  DVMRP  thresh^ 64  
 -7  arl-mbone.dren.net (138.18.4.224)  DVMRP  thresh^ 32  
 -8  nccosc-mbone.dren.net (138.18.5.224)  DVMRP  thresh^ 32  
 -9  * * * FIXW-MBONE.NSN.NASA.GOV (192.203.230.243) [11.2] didn't respond
-10  * * * 
-11  
Resuming...
 -9  FIXW-MBONE.NSN.NASA.GOV (192.203.230.243)  Unknown protocol code 5  thresh^ 64  
-10  Loopback0.MB1.SFO1.Alter.Net (137.39.3.93)  Unknown protocol code 5  thresh^ 64  
-11  dec3800-2-fddi-0.SanFrancisco.mci.net (204.70.158.61)  DVMRP  thresh^ 64  
-12  * * * paloalto-mbone1.bbnplanet.net (192.42.110.249) [11.2] didn't respond
-13  * * * 
-14  * * * 
-15  
Resuming...
-12  paloalto-mbone1.bbnplanet.net (192.42.110.249)  Unknown protocol code 5  thresh^ 32  
-13  cambridge1-mbone1.bbnplanet.net (199.94.207.2)  Unknown protocol code 5  thresh^ 32  
-14  ? (192.36.137.1)  DVMRP  thresh^ 32  
-15  stockholm.mbone.ebone.net (192.36.148.206)  DVMRP  thresh^ 64  
-16  lower-gw.sunet.se (192.36.148.197)  PIM  thresh^ 32  
-17  falcon.pilsnet.sunet.se (192.36.125.68)
Round trip time 394 ms

Waiting to accumulate statistics...
* * Results after 15 seconds:

  Source        Response Dest    Packet Statistics For     Only For Traffic
192.36.125.68   192.54.172.200   All Multicast Traffic     From 192.36.125.68
     v       __/  rtt  863 ms    Lost/Sent = Pct  Rate       To 224.2.0.1
192.36.125.1   
192.36.148.197  lower-gw.sunet.se 
     v     ^      ttl   32       -1/499  =  0%  33 pps     0/0    = --%   0 pps
192.36.148.206  stockholm.mbone.ebone.net 
     v     ^      ttl   64       25/1516 =  2% 101 pps     0/0    = --%   0 pps
192.36.137.1    ?              
     v     ^      ttl   65        2/2098 =  0% 139 pps     0/0    = --%   0 pps
199.94.207.2    cambridge1-mbone1.bbnplanet.net 
     v     ^      ttl   66        9/2285 =  0% 152 pps     0/0    = --%   0 pps
131.119.0.197  
192.42.110.249  paloalto-mbone1.bbnplanet.net 
     v     ^      ttl   67      475/2692 = 18% 179 pps     0/0    = --%   0 pps
204.70.158.61   dec3800-2-fddi-0.SanFrancisco.mci.net 
     v     ^      ttl   68      163/2880 =  6%   0 pps     0/0    = --%   0 pps
208.205.11.6   
137.39.3.93     Loopback0.MB1.SFO1.Alter.Net 
     v     ^      ttl   69      183/2713 =  7% 180 pps     0/0    = --%   0 pps
198.32.136.99  
192.203.230.243 FIXW-MBONE.NSN.NASA.GOV 
     v     ^      ttl   70       -2/3246 =  0% 216 pps     0/0    = --%   0 pps
138.18.5.224    nccosc-mbone.dren.net 
     v     ^      ttl   71       73/3609 =  2% 240 pps     0/0    = --%   0 pps
138.18.4.224    arl-mbone.dren.net 
     v     ^      ttl   72        8/3061 =  0% 204 pps     0/0    = --%   0 pps
132.250.230.10  nrl-mbone.dren.net 
     v     ^      ttl   73      -90/2497 = -3% 166 pps     0/0    = --%   0 pps
207.41.200.99   pen-mbone-1.sprintlink.net 
     v     ^      ttl   74      699/3113 = 22% 207 pps     0/0    = --%   0 pps
169.130.122.18  mbone.appliedtheory.com 
     v     ^      ttl   75        5/1875 =  0% 125 pps     0/0    = --%   0 pps
192.35.82.97    MBONE.CIT.CORNELL.EDU 
     v     ^      ttl   76      461/1328 = 35%   0 pps     0/0    = --%   0 pps
193.51.208.8    sobone.inria.fr 
     v     ^      ttl   77      254/431  = 59%  28 pps     0/0    = --%   0 pps
193.54.112.94   seychelles.cma.fr Wrong interface
     v      \__   ttl   78   
192.54.172.200  192.54.172.200
  Receiver      Query Source

I am not very happy to see that France is splitted halfway, with each
MBone part communicating through the US ! :-(
Nice trip from south Paris to Sweden through San Francisco...

I am still puzzled...

---

From: Bill Fenner <[email-masked]>
Date: Fri, 14 Nov 1997 09:44:54 PST

>Of course I cannot mtrace directly them since there is no route... :-(

If you "kill -USR1 `cat /etc/mrouted.pid`", mrouted will dump its routing
table into /usr/tmp/mrouted.dump, and you can see exactly what routes you
do have.

>What happens if a route has a distance greater than 32 ?

It disappears at the router that has the distance of 32.  It would never
get to you.  Since you wouldn't have a route (and you wouldn't have sent
a poison reverse for it), your neighbors should never send the packets
to you.

Assymetric metrics don't enter into it, since router A only sends traffic
to router B if router B has sent a poison-reverse to router A.  A should
only be sending if B asked it to.

>What this 192/8 martian is ?

It's a symptom of a Bay Networks bug, present in revisions of they Bay
software prior to 12.0.

>chailly.ensmp.fr to remove the martians above. But no improvement.

I hope you said "deny 192/8 exact", or you will deny all routes in
192.x.x.x.

> -3  MBONE.CIT.CORNELL.EDU (192.35.82.97)  DVMRP  thresh^ 32

I wonder how long Cornell will be causing trouble for the MBone.
Cornell has all these "backup" tunnels which travel over insanely
bad paths and have lower metrics than the primary path.

>MBone part communicating through the US ! :-(

Looks like that's because oreste.inria.fr is down, so you use the
"backup" tunnel to Cornell.

  Bill