Subject: Incorrect source address on multicast packets on multihomed machines?

From: Phil Irey <[email-masked]>
Date: Mon, 22 Jun 1998 13:50:51 -0400 (EDT)

Hi,

I'm having a problem with a number of multihomed Sun Solaris 2.6
machines.  I have an application which selects outgoing interface on
which multicast packets are transmitted.  When an interface other than
the ethernet interface is selected, say for example the ATM interface,
the  multicast packet is sent out the ATM interface but the source IP
address in the packet is that of the ethernet interface and not ATM
interface.  This also happens on gigabit ethernet and FDDI interfaces
as well.  The source IP address is always that of the ethernet
interface despite the fact that the packet is forwarded out an
interface other than the ethernet. 

Is this correct behavior?  It looks like a form of spoofing. When we
run the same application on SGI machines, they insert the "proper"
address (i.e. IP address of the ATM interface on packets sent on the
ATM interface) in the source address field of the packet. 

The reason this is of concern to us is that we have a Cisco router
forwarding traffic between our ATM network (configured as a LANE
network) and other networks.  When the Cisco sees multicast traffic
with a source IP indicative of a subnetwork other than that on which
the packet arrived, it rejects the  packet logging "NOT RPF
interface". 

If the behavior described above is correct, how can we get the Cisco
to stop rejecting the packets?

thanks,

phil

---

From: "John M. Zwiebel" <[email-masked]>
Date: Mon, 22 Jun 1998 11:17:20 -0700

Hello Phil:

To modify the cisco, you need to change the routing so the packet is coming
in on the RPF interface.  The easiest way to do this will be to use a
static mroute.  However, you need to use caution with this since the router
will now reject mpackets coming in on the "correct" interface (ie the
interface that currently shows the RPF to the source with "sh ip rpf 
<source>".)

You can see that this command provides more than enough options that you'll
be able to very effectively shoot yourself in the foot if you aren't careful
with it.

    [no] ip mroute <source> <mask> [<protocol><as-number>] [route-map <map>]
                   <rpf-address> | <interface> [<distance>]
        Configures a multicast static route (called a "static mroute"). When 
        a source range is specified, the mroute applies only to those sources.
        When <protocol><as-number> is specified, the mroute applies to those 
        sources that have been learned by the corresponding routing process. 
        If route-map <map> is specified, further classification can be 
        accomplished by the match clauses from <map>. If the mroute is
        selected, the <rpf-address> address dictates the incoming interface 
        for the source that matches the mroute. If the <rpf-address> is a 
        PIM neighbor, PIM Joins, Grafts, and Prunes will be sent to it. 
        The <rpf-address> can be a host address of a directly connected router
        or a route. When it is a route, a recursive lookup is done from the 
        unicast routing table to find a directly connected neighbor. If 
        <rpf-address> is not specified, <interface> is used as the incoming 
        interface. <distance> is used to decide if a unicast route, a DVMRP 
        route, or a static mroute should be used for the RPF lookup. The 
        lower distances have better preference. If the static mroute has the 
        same distance as the other two RPF sources, the static mroute will 
        take precedence. There are only two exceptions to this rule, 
        directly connected routes and the default unicast route. Default 
        <distance> is 0. Static mroutes are local to the router and are not 
        redistributed by any dynamic routing protocol. [11.0]

WRT the source address that is used with the ATM interface, would you think
it was correct to place a different address in the source if it was a
unicast packet?

z

 ^ Hi,
 ^ 
 ^ I'm having a problem with a number of multihomed Sun Solaris 2.6
 ^ machines.  I have an application which selects outgoing interface on
 ^ which multicast packets are transmitted.  When an interface other than
 ^ the ethernet interface is selected, say for example the ATM interface,
 ^ the  multicast packet is sent out the ATM interface but the source IP
 ^ address in the packet is that of the ethernet interface and not ATM
 ^ interface.  This also happens on gigabit ethernet and FDDI interfaces
 ^ as well.  The source IP address is always that of the ethernet
 ^ interface despite the fact that the packet is forwarded out an
 ^ interface other than the ethernet. 
 ^ 
 ^ Is this correct behavior?  It looks like a form of spoofing. When we
 ^ run the same application on SGI machines, they insert the "proper"
 ^ address (i.e. IP address of the ATM interface on packets sent on the
 ^ ATM interface) in the source address field of the packet. 
 ^ 
 ^ The reason this is of concern to us is that we have a Cisco router
 ^ forwarding traffic between our ATM network (configured as a LANE
 ^ network) and other networks.  When the Cisco sees multicast traffic
 ^ with a source IP indicative of a subnetwork other than that on which
 ^ the packet arrived, it rejects the  packet logging "NOT RPF
 ^ interface". 
 ^ 
 ^ If the behavior described above is correct, how can we get the Cisco
 ^ to stop rejecting the packets?
 ^ 
 ^ thanks,
 ^ 
 ^ phil

------------------------------------------------------------------------------
	John Zwiebel                       Phone: 408-526-5303
	Cisco Systems Inc.                 
	Multimedia Group

---

From: Phil Irey <[email-masked]>
Date: Mon, 22 Jun 1998 16:20:02 -0400 (EDT)

John,

Thanks for the response.  We will take a look at the options of the ip
mroute command if we can't get a fix for the Sun system (assuming that
they are broken in some way).

> it was correct to place a different address in the source if it was a

As with unicast packets, we would expect the source IP address of a packet
sent by a host to be that of the interface on which the packet was
sent.  (I'm only talking about the source host and not intermetiate
systems.)  What we are seeing on the Suns is that the source IP
address is always that of the ethernet even though we specify that the
multicast packet be sent via ATM in our application.  The packet is
sent from the source host over the ATM interface but with the IP
address of the ethernet interface of the source host.  The FDDI
interface behaves the same way.

Is this acceptable behavior or is the Sun choosing the wrong source IP
address?  We are speculating that the Sun is doing something wrong.
Since we only have access to SGI boxes and Sun boxes it is hard to
tell which boxes are operating correctly or if both sets of boxes are
operating correctly. Is there some way to change the behavior of the
Sun? 

We talked to Cisco tech support who helped us narrow down the problem
to the source IP address in question.  They indicated that multicast
packets with a source IP addresses of a different network are rejected
by the routing loop detection mechinsms in the router.   If this is
true, I would expect that something has to change on the Sun for us to 
get this to work properly.  Would you agree with this?

thanks,

phil

---

From: Steve Deering <[email-masked]>
Date: Thu, 25 Jun 1998 13:04:03 -0700

At 1:20 PM -0700 6/22/98, Phil Irey wrote:
>            ...What we are seeing on the Suns is that the source IP

Phil,

>From your description, it does sound like the Sun is misbehaving.
The source address of a multicast packet must belong to the interface
where it originated.  Almost all of the existing multicast routing
protocols (in particular, DVMRP, MOSPF, PIM-DM, and PIM-SM) depend on
that for their correct operation.

But one question: is the Sun that is originating the multicast packet also
running a mutlicast routing demon?  If so, it is possible that the packet
is being initially sent out the Ethernet interface, and then also being
*forwarded* out the ATM interface, in which case it would be correct for
the copy on the ATM interface to have the Ethernet's IP address as its
source address.  That would still leave the question as to why the
application's request for origination on the ATM interface was not being
honored...

Steve

---

From: Phil Irey <[email-masked]>
Date: Thu, 25 Jun 1998 16:58:50 -0400 (EDT)

> running a mutlicast routing demon?  If so, it is possible that the
packet

We have shut down all instances of mrouted for this test.

 > application's request for origination on the ATM interface was not being

Another data point on this problem is that if you reboot the machine
and run our test application, the first packet has the correct source
IP address (e.g. gigabit ethernet interface address if the packet was
sent over that interface) all packets after the first have the incorrect
source IP address (e.g. ethernet address if the packet was sent over
the gigabit ethernet interface).  If you don't run the test
application for an sufficiently long period of time, the first packet
will also be correct followed by incorrect packets.  It seems as
though something needs to time out.  If you successive tests without a
sufficiently long period of time between them, the first packet is
incorrect as well as all subsequent packets.

In case anyone is interested, I have attached some sample test runs
below..

thanks,

phil
-----------------------------------------------------------------------------
In the scenario below:

128.38.42.0 is the ethernet
10.1.1.0 is the FDDI
10.6.1.0 is the Gigabit Ethernet

The sending machine is titanic.  The receiving machine is tide.

titanic_giga is the gigabit ethernet interface on titanic.
titanic_fddi is the FDDI interface on titanic.
tide_giga is the gigabit ethernet interface on tide.
tide_fddi is the FDDI interface on tide.

### 1) output from the sending application on titanic transmitting on
the gigabit ethernet interface:

                                                      specifies the 
                                                    outgoing interface
                                                            \/
119 (titanic) /home/sunoco/pirey/cast  -> multitest -s -i titanic_giga
Sending packets to 225.0.15.245:17490 with ttl 16
898539666: 64 bytes sent to 225.0.15.245:17490/1800 - seq # 0
898539667: 64 bytes sent to 225.0.15.245:17490/1800 - seq # 1
898539668: 64 bytes sent to 225.0.15.245:17490/1800 - seq # 2
898539669: 64 bytes sent to 225.0.15.245:17490/1800 - seq # 3
898539670: 64 bytes sent to 225.0.15.245:17490/1800 - seq # 4
898539671: 64 bytes sent to 225.0.15.245:17490/1800 - seq # 5
898539672: 64 bytes sent to 225.0.15.245:17490/1800 - seq # 6
898539673: 64 bytes sent to 225.0.15.245:17490/1800 - seq # 7
^C<<<***Exiting***>>>

At the same time a snoop at the receiver on the gigabit ethernet
interface reports:

51 (tide) /home/sunoco/pirey  -> snoop -V -d vge0 host 225.0.15.245
Using device /dev/vge (promiscuous mode)
________________________________
     titanic -> 225.0.15.245 ETHER Type=0800 (IP), size = 106 bytes
     titanic -> 225.0.15.245 IP  D=225.0.15.245 S=128.38.42.115 LEN=92, ID=10463
     titanic -> 225.0.15.245 UDP D=17490 S=60597 LEN=72
________________________________
     titanic -> 225.0.15.245 ETHER Type=0800 (IP), size = 106 bytes
     titanic -> 225.0.15.245 IP  D=225.0.15.245 S=128.38.42.115 LEN=92, ID=10464
     titanic -> 225.0.15.245 UDP D=17490 S=60597 LEN=72
________________________________
     titanic -> 225.0.15.245 ETHER Type=0800 (IP), size = 106 bytes
     titanic -> 225.0.15.245 IP  D=225.0.15.245 S=128.38.42.115 LEN=92, ID=10465
     titanic -> 225.0.15.245 UDP D=17490 S=60597 LEN=72
________________________________
     titanic -> 225.0.15.245 ETHER Type=0800 (IP), size = 106 bytes
     titanic -> 225.0.15.245 IP  D=225.0.15.245 S=128.38.42.115 LEN=92, ID=10466
     titanic -> 225.0.15.245 UDP D=17490 S=60597 LEN=72
________________________________
     titanic -> 225.0.15.245 ETHER Type=0800 (IP), size = 106 bytes
     titanic -> 225.0.15.245 IP  D=225.0.15.245 S=128.38.42.115 LEN=92, ID=10467
     titanic -> 225.0.15.245 UDP D=17490 S=60597 LEN=72
________________________________
     titanic -> 225.0.15.245 ETHER Type=0800 (IP), size = 106 bytes
     titanic -> 225.0.15.245 IP  D=225.0.15.245 S=128.38.42.115 LEN=92, ID=10468
     titanic -> 225.0.15.245 UDP D=17490 S=60597 LEN=72
________________________________
     titanic -> 225.0.15.245 ETHER Type=0800 (IP), size = 106 bytes
     titanic -> 225.0.15.245 IP  D=225.0.15.245 S=128.38.42.115 LEN=92, ID=10469
     titanic -> 225.0.15.245 UDP D=17490 S=60597 LEN=72
________________________________
     titanic -> 225.0.15.245 ETHER Type=0800 (IP), size = 106 bytes
     titanic -> 225.0.15.245 IP  D=225.0.15.245 S=128.38.42.115 LEN=92, ID=10470
     titanic -> 225.0.15.245 UDP D=17490 S=60597 LEN=72

You can see that the packets arrived on the gigabit ethernet
interface (e.g. vge0) and that the source address was from
128.38.42.115 which is an ethernet address not a gigabit ethernet
address. 

### 2) To prove that the same packets are not showing up on the
ethernet interface the same test was run again.

The sender was titanic as before sending over the gigabit ethernet
interface:

120 (titanic) /home/sunoco/pirey/cast  -> multitest -s -i titanic_giga
Sending packets to 225.0.15.245:17490 with ttl 16
898539727: 64 bytes sent to 225.0.15.245:17490/1801 - seq # 0
898539728: 64 bytes sent to 225.0.15.245:17490/1801 - seq # 1
898539729: 64 bytes sent to 225.0.15.245:17490/1801 - seq # 2
898539730: 64 bytes sent to 225.0.15.245:17490/1801 - seq # 3
898539731: 64 bytes sent to 225.0.15.245:17490/1801 - seq # 4
898539732: 64 bytes sent to 225.0.15.245:17490/1801 - seq # 5
898539733: 64 bytes sent to 225.0.15.245:17490/1801 - seq # 6
^C<<<***Exiting***>>>

The receiver on tide was set to snoop on the ethernet iterface:

52 (tide) /home/sunoco/pirey  -> snoop -V -d hme0 host 225.0.15.245
Using device /dev/hme (promiscuous mode)
^C53 (tide) /home/sunoco/pirey  -> netstat -r -n

As can be seen, no packets arrived on the ethernet interface.

### 3) Netstat and ifconfig

on titanic:

121 (titanic) /home/sunoco/pirey/cast  -> netstat -r

Routing Table:
  Destination           Gateway           Flags  Ref   Use   Interface
-------------------- -------------------- ----- ----- ------ ---------
10.1.1.0             titanic_fddi          U        2      2  pf0
B35_ethernet         titanic               U        3  12290  hme0
10.6.1.0             titanic_giga          U        2     72  vge0
BASE-ADDRESS.MCAST.NET titanic               U        3      0  hme0
default              routerdude            UG       0   6202  
localhost            localhost             UH       0 455333  lo0
122 (titanic) /home/sunoco/pirey/cast  -> ifconfig -a
lo0: flags=849<UP,LOOPBACK,RUNNING,MULTICAST> mtu 8232
        inet 127.0.0.1 netmask ff000000 
hme0: flags=863<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST> mtu 1500
        inet 128.38.42.115 netmask ffffff00 broadcast 128.38.42.255
pf0: flags=863<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST> mtu 4352
        inet 10.1.1.115 netmask ffffff00 broadcast 10.1.1.255
vge0: flags=863<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST> mtu 1500
        inet 10.6.1.1 netmask ffffff00 broadcast 10.6.1.255
123 (titanic) /home/sunoco/pirey/cast  -> 

on tide:

55 (tide) /home/sunoco/pirey  -> netstat -n -r

Routing Table:
  Destination           Gateway           Flags  Ref   Use   Interface
-------------------- -------------------- ----- ----- ------ ---------
10.1.1.0             10.1.1.86             U        2     18  nf0
128.38.42.0          128.38.42.86          U        3   6965  hme0
10.6.1.0             10.6.1.86             U        2    182  vge0
10.4.1.0             10.4.1.86             U        2      0  ba0
224.0.0.0            128.38.42.86          U        3      0  hme0
default              128.38.42.1           UG       0    348  
127.0.0.1            127.0.0.1             UH       0 113467  lo0
56 (tide) /home/sunoco/pirey  -> ifconfig -a
lo0: flags=849<UP,LOOPBACK,RUNNING,MULTICAST> mtu 8232
        inet 127.0.0.1 netmask ff000000 
hme0: flags=863<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST> mtu 1500
        inet 128.38.42.86 netmask ffffff00 broadcast 128.38.42.255
        ether 8:0:20:83:21:5a 
nf0: flags=863<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST> mtu 4352
        inet 10.1.1.86 netmask ffffff00 broadcast 10.1.1.255
        ether 8:0:20:83:21:5a 
vge0: flags=863<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST> mtu 1500
        inet 10.6.1.86 netmask ffffff00 broadcast 10.6.1.255
        ether 8:0:20:83:21:5a 
ba0: flags=863<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST> mtu 9180
        inet 10.4.1.86 netmask ffffff00 broadcast 10.4.1.255
        ether 8:0:20:84:ce:b1 

#############################################################################

---

From: Toerless Eckert <[email-masked]>
Date: Sat, 27 Jun 1998 09:32:55 +0200 (MET DST)

I don't know your test application (i havn't folled that thread so much),

1. are you sure that your application does correctly bind the local
   originator address AND interface (with IP_MULTICAST_IF) ? If in doubt,
   try my test program:

   ftp://ftp.informatik.uni-erlangen.de/local/inf4/eckert/push/push.sos5

   Should be similarily to use than yours:

   % push.sos5 -s 224.0.15.245 -T 16 -L titanic_giga -k 1 -p 1800

   Will send 1 kilobyte/sec rate of TTL 16 multicast packets originating
   from first ip-address found for titanic_giga to 224.0.15.245 on port 1800

   You can check on the receiver 

   % push.sos5 -r 224.0.15.245

2. I've seen all kind of broken multicast support for fddi interfaces
   on Sun. Maybe there's a problem with your driver. What kind of interface is
   this ?

Toerless