Bug 369591

Summary: multicast change in behavior
Product: Red Hat Enterprise Linux 4 Reporter: Aleksandar Kostadinov <akostadi>
Component: kernelAssignee: Neil Horman <nhorman>
Status: CLOSED NOTABUG QA Contact: Martin Jenner <mjenner>
Severity: low Docs Contact:
Priority: low    
Version: 4.5CC: bban, davem, ldimaggi, tgraf
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2008-05-21 17:47:13 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
jar containing sender and receiver
none
multicast receiver that can bind to specific interface
none
modified mcsend to set IP_MULTICAST_IF none

Description Aleksandar Kostadinov 2007-11-07 13:58:26 UTC
Description of problem:
Consider the following situation on a single host:
1. 224.0.0.0/240.0.0.0 route is through eth0
2. mcast receiver bound to localhost
3. mcast sender configured to send through lo

On RHEL 4U4 the receiver sees the send messages.
On RHEL 4U5 and RHEL5 it doesn't.

I can't tell what exactly the reason is. My guess is that packets are sent
throuth eth0 instead of lo.

How reproducible:
Always

Steps to Reproduce:
1.setup route as described above
2.set java 1.5 for use
3.use attached jgroups.jar (see #4)
4.
java -cp jgroups.jar org.jgroups.tests.McastSenderTest -mcast_addr 224.10.10.10
-bind_addr localhost
java -cp jgroups.jar org.jgroups.tests.McastReceiverTest -mcast_addr
224.10.10.10 -port 5555 -bind_addr localhost
(use -h or http://www.jgroups.org/javagroupsnew/docs/manual/html/ch02.html
section 2.8 for more information)
5.type something on the sender console

receiver will not see anything on 4U5 but will on 4U4

Comment 1 Aleksandar Kostadinov 2007-11-07 13:58:26 UTC
Created attachment 250061 [details]
jar containing sender and receiver

Comment 2 Aleksandar Kostadinov 2008-01-03 12:44:45 UTC
Let me express #2 from the description correctly: "mcast receiver joins the
multicast group on interface lo"

Comment 3 Neil Horman 2008-05-12 19:58:45 UTC
 I don't suppose you have a test case written in C so that I can inspect it do
you?  If not I'll write one to verify this behavior.  I'd just like to be able
to verify that the application is doing the right things to properly receive
multicast frames

Comment 4 Aleksandar Kostadinov 2008-05-13 06:51:34 UTC
Created attachment 305204 [details]
multicast receiver that can bind to specific interface

I don't have a test but mcsend.c from #231899 and the attached mcreceive.c
should aid you.

Comment 5 Neil Horman 2008-05-13 17:37:16 UTC
I just tried this on ibm-maple.rhts.bos.redhat.com:
===================================================================
====receiver=======================================================
[root@ibm-maple 2.6.9]# uname -a
Linux ibm-maple.rhts.bos.redhat.com 2.6.9-67.ELsmp #1 SMP Wed Nov 7 13:58:04 EST
2007 i686 i686 i386 GNU/Linux
[root@ibm-maple 2.6.9]# cd
[root@ibm-maple ~]# mount
curly.devel.redhat.com:/vol/engineering/devarchive/redhat /mnt/redhat/
[root@ibm-maple ~]# ls
anaconda-ks.cfg  a.out  c  c_send_receive.tar.gz  install.log 
install.log.syslog  ks-post.log  mcreceive.c  yum-2.2.2-1.rhts.EL4.noarch.rpm
[root@ibm-maple ~]# cd c
[root@ibm-maple c]# cd ..
[root@ibm-maple ~]# mv a.out mreceive
[root@ibm-maple ~]# ./mreceive 224.0.0.1 10000 127.0.0.1
Received 3 bytes from 10.16.64.145: hi

====sender===========================================================
[root@ibm-maple c]# ./msend 224.0.0.1 10000
Begin typing (return to send, ctrl-C to quit):
hi

======================================================================

This was with the -67 kernel, which IIRC is RHEL4 U6, using the receiver you
attached above and the sender you specified from bz 231899. I did try the same
thing on RHEL5 with the -92.el5 kernel and I found the behavior you describe. 
If however, I add a route for 224.0.0.1 to RHEL5 going through lo rather than
eth0, I can receive frames there too.

So this brings to mind the question, what is the correct behavior
?  My initial reaction would be that the RHEL5 behavior is correct, given that
we have specified a route that explicitly sends  frames to eth0 (or
inadvertently to eth0 via the default route), I would not expect  a receiver
bound to loopback to get those frames.  That being said, I'm not 100% sure of
that.  I need to find the relevant RFC's on the subject to see if they are clear
on this subject.  Do you have any documentation making the case for one behavior
or the other?

Comment 6 Aleksandar Kostadinov 2008-05-14 07:51:12 UTC
I verified I'm describing correct behavior on 4u6 and 5u1. You just need to use
some other multicast address - not 224.0.0.1 because host has joined it on all
interfaces by default.
See #231899 for why that allows the receiver to get messages even though it is
bound to lo and not eth0. You rejected that as a bug but I think it is...

So about your question what is the correct behavior - I don't know. 
Though the change is between 4u4 and 4u5 so could be a regression for some users
that expect to receive locally generated mcast messages by local procs no matter
where bound. One guy was confused recently
onhttps://www.redhat.com/archives/fedora-list/2008-May/msg00834.html if that's
any useful for your decision.

Comment 7 David Miller 2008-05-14 11:36:11 UTC
Note that David Stevens at IBM would be a good person to ask about this
issue.

He usually knows the answer faster than I can type the question, when
it comes to multicast semantic issues.


Comment 8 Neil Horman 2008-05-14 12:09:12 UTC
Thanks dave, I'll ping him for guidance on semantics here.

Aleksandar, regarding the other bz, I maintain that it is not a bug, given that
you were binding to INADDR_ANY.  I had missed your comment previously regarding
ipv6 behavior.  I agree that we should have consistent behavior between the
protocols there, although I think the consistency needs to settle on not making
that check.  That provides us the desired consistency, and allows for a more
flexible api.  I'll investigate that upstream.

Comment 9 Aleksandar Kostadinov 2008-05-14 14:37:57 UTC
Intuitively I'd expect to receive only messages for groups I've joined in, on
the interface/port pair I'm bound to. Hence my confusion...
We really need a well defined consistent behavior (whatever it is) to avoid
customer confusion. So thanks for looking into that.

Comment 10 Neil Horman 2008-05-14 14:54:28 UTC
I agree, I included a question regarding binding to INADDR_ANY and the reception
of multicast messages in my email as well.  Hopefully David will be able to
provide references on what standard behavior should be.

I retried my RHEL4 tests above, and the behavior is in fact different.  For any
group other than 224.0.0.1, I need to specify the address of the interface that
matches the route that any sent multicast traffic is sent on to recieve it on
the local host.  That at least seems sane to me.  Like you said, you shouldn't
recieve traffic from multicast groups if you haven't joined them (and thats
causing me to rethink my stance on bz 231899).  But we'll see what DaveS. has to
say.  Let see if we can nail down what standard behavior should be before we
start trying to code it in place.

Comment 11 Neil Horman 2008-05-14 17:43:04 UTC
Here are my questions regarding mc behavior, and Mr. Stevens responses:


==========================================================================
 The scenario is this:  Host A has the following route added to its
route
> table:
> Destination     Gateway         Genmask         Flags Metric Ref    Use
Iface
> 224.0.0.0       *               240.0.0.0       U     0      0        0
eth0

        This part is kind of a hack. Multicasting applications generally
should not make use of the unicast routing table at all. It does only
as a last resort, if the user hasn't explicitly specified the interface
for sending on (which it should do, since it matters). Apps do that by
using the IP_MULTICAST_IF socket option (or any other ways that force
a particular interface, such as SO_BINDTODEVICE). On UNIX systems, they
use SO_DONTROUTE, which makes it explicit. If it's something you're
developing or a customer app, I highly recommend that you make sure it
always knows exactly which interface it's sending to and doesn't rely
on the unicast routing table at all.

>
>    There are two processes (A and B) on this host.  A sends multicast
> messages to a multicast address that matches on that route.
>
>    Process B listens for these messgaes.  B binds its sockets to the
same
> multicast address, and the same port as the sender process A.  However,
when
> process B issues the IP_ADD_MEMBERSHIP sockopt, it joins the appropriate
> multicast group address, but subscribes with a local address
corresponding to
> the local loopback interface (127.0.0.1).  As a result, process B never
sees
> frames sent by process A
>
> So My questions to you are:
>
> 1) Is the above correct behavior?

        Yes, if I understand what you're saying correctly. Group
membership
is per-interface. Joining a group on "lo" has nothing at all to do with
groups (with the same number or not) on "eth0".


>
> 2) What if process B changed its ADD_MEMBERSHIP call to set the local ip
> subscription address to the ip address of eth0?  Should process B see
frames
> sent from process A then?

        Yes. In this case, it depends on the setting of mc_loop. The
default is
to receive, unless the app has done an IP_MULTICAST_LOOP with value 0 to
turn
it off.

> 3) Would setting IP_MULTICAST_LOOP on the sender affect weather process
B sees
> these frames, or is mc loop behavior restricted to the setting socket.

        It's on by default, so it isn't necessary as long as you don't use
this
to turn it off. It's done in the output path, so the check is based on the
sender's setting. I haven't tested it, but I don't believe anyone else's
setting
matters-- the sender decides whether local listeners hear it or not.

>
>
> 4) Finally, what if process B bound itself to INADDR_ANY rather than to
the
> specific multicast group.  Should it see process A's sent frames then?

        Not if the group membership is on lo and the sends are on eth0.
The
reason it isn't seeing the packets is not the binding, but the group
membership. To hear packets you're sending out an interface, you must
join that group on that interface *and* the sender must allow loopback by
not clearing IP_MULTICAST_LOOP. Joining the group on a different interface
really is joining a different group, as far as multicasting is concerned.
===========================================================================


The large takeaway I have from this is that multicasting applications need to be
explicit in the interfaces that they send frames from.  I tested his suggestion
of using IP_MULTICAST_IF on our test case, modifying mcsend.c to accept a 3rd
argument which was the source address of the packet.  Sure enough, adding those
bits allows us to work properly on RHEL5 and RHEL4.  

The conclusions I have from this conversation are:

1) The latest behavior in RHEL4 and RHEL5 is correct.  If you don't match
interfaces on the join request for a multicast sender and receiver on the same
host, you will not (nor should you expect to) receive frames that are sent from
local applicaions.  

2) The bug we discussed previously (bz 231899), I was wrong about.  It sounds
like, despite binding to INADDR_ANY, if we don't join a given group , then we
shouldn't see frames from that group.  Seems like I need to duplicate the ipv6
check in the ipv4 code.

Do you concur?

Comment 12 Aleksandar Kostadinov 2008-05-15 07:40:43 UTC
Could you attach the modified mcsend.c so I can run some tests?

Comment 13 Neil Horman 2008-05-15 11:09:21 UTC
Created attachment 305466 [details]
modified mcsend to set IP_MULTICAST_IF

Sure, there you go.  The only addition to this version is that it accepts a 3rd
argument that works exactly like mcreceive.c.  It specifies, in addition to a
multicast group and port to send to, a local ip address of the interface that
the packet should be sourced from.  This of course needs to be matches to the
local interface that the receiver is listening on, or frames won't be received.

Comment 14 Aleksandar Kostadinov 2008-05-20 11:38:40 UTC
That makes sense to me. But I'd like to ask JBoss jgroups team if they have
anything to say.

Comment 15 Neil Horman 2008-05-20 13:24:45 UTC
Sure, my proposal stands at closing this bug as NOTABUG, and reopening 231899 as
an actual which I will fix to not receive mcast frames if you don't join any
groups, even if bound to INADDR_ANY.  Let me know what they say.

Comment 16 Aleksandar Kostadinov 2008-05-21 14:26:16 UTC
They don't see an immediate issue with that so I think you can go ahead. Of
course they have to maintain compatibility with other platforms so could decide
to reopen the issue for compatibility reasons.

Thanks!

Comment 17 Neil Horman 2008-05-21 17:47:13 UTC
Ok, but Just so that everyone is aware, we don't maintain bug for bug
compatibility.  We need to be able to fix behavioral problems, even if they
adversely affect application behavior.