Bug 1288412 - MAC address of VLAN over TEAM interface keeps on changing frequently.
MAC address of VLAN over TEAM interface keeps on changing frequently.
Status: CLOSED NOTABUG
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: NetworkManager (Show other bugs)
7.2
x86_64 Linux
medium Severity high
: rc
: 7.3
Assigned To: Rashid Khan
Desktop QE
:
Depends On:
Blocks: 1304407
  Show dependency treegraph
 
Reported: 2015-12-04 01:49 EST by Shubhrata Priyadarshinee
Modified: 2016-02-03 11:02 EST (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-12-07 09:01:55 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
sosreport (5.42 MB, application/x-xz)
2015-12-04 01:49 EST, Shubhrata Priyadarshinee
no flags Details
NM debug logs (950.37 KB, text/plain)
2016-01-20 07:45 EST, Shubhrata Priyadarshinee
no flags Details

  None (edit)
Description Shubhrata Priyadarshinee 2015-12-04 01:49:56 EST
Created attachment 1102153 [details]
sosreport

Description of problem:
Interfaces are teamed and then vlan is added, the MAC address of vlan keeps on changing and new ip is obtained for vlan interface repeatedly.


Version-Release number of selected component (if applicable):
NetworkManager-1.0.6-27
kernel:- 3.10.0-327

How reproducible:
Always

Steps to Reproduce:
1.Setup server with 2 interfaces, both connected to swich ports having vlan tagging enabled.

2. install RHEL-7.2 GA. 

3. Create team interface using above two interfaces. Note that Team interface should not get any IP address.
#nmcli con add type team ifname loms
#nmcli con add type team-slave con-name lom1 ifname p2p1 master loms
#nmcli con add type team-slave con-name lom2 ifname p2p2 master loms
#nmcli con up lom1
#nmcli con up lom2
#nmcli con up team-loms

4. Create vlan interface over the team interface created in step 3.
#nmcli con add type vlan ifname vlan1140 dev loms id 1140

5.Once the vlan connection came up, it starts leasing new ip addresses every few seconds. 


Actual results:
MAC address and IP address of vlan interface keeps on changing frequently. 

Expected results:
MAC address of vlan interface should remain constant.

Additional info:
- Issue observed in Fedora 23 also.
- Issue specific to NetworkManager.
Comment 1 Sujith 2015-12-04 02:53:54 EST
Observations:
* Since the switch-ports are vlan-tagged, p1p2 and p2p2 interfaces won't get IP address initially. After teaming them up, the team-master also wont get the IP address. VLAN interface created over the team-master does get the IP address.

* NetworkManager changes the state of these 3 interfaces (p1p2, p2p2 and team-master) to failed/disconnected based on the failure to get DHCP response. Later, NM start auto-activating them (state = up) again to repeat the whole cycle again.
This seems to be causing the vlan interface also to change the state from up->down continuously.

I am not sure whether this is the reason why VLAN interface is changing its MAC address frequently. Can you tell us the reason behind this behavior?
Comment 2 Sujith 2015-12-04 03:51:11 EST
(In reply to Sujith from comment #1)
> Observations:
> * Since the switch-ports are vlan-tagged, p1p2 and p2p2 interfaces won't get

Correction in Comment#1:Point#1:
It should've been p2p1 and p2p2.


Adding to the above observations:
* After marking p2p1, p2p2, team-master, team-slaves and vlan interfaces with 'NM_CONTROLLED=no', the issue is not observed i.e vlan mac & ip remains constant.
Comment 3 Sujith 2015-12-04 08:28:24 EST
* Similar behavior (as the issue) is seen when a vlan interface is created over a bond-master having these two interfaces (p2p1 and p2p2) as bond-slaves. However the mac address remains constant and hence the vlan interface gets back the original IP address. 
Log shows that all these interfaces are going up->down and the same cycle repeats.

Do we need a separate bug-report to track this behavior?

* In both cases (1.vlan on team-master; 2.vlan on bond-master) ping to an external machine is not disrupted.
Comment 4 Beniamino Galvani 2015-12-04 16:38:41 EST
Hi,

when a team connection is added with command "nmcli con add type team
ifname loms", it gets created with ipv4.method=auto (DHCP) and
ipv6.method=auto (RA and possibly DHCPv6). This means that
NetworkManager will actively try to get addresses for the team
interface and if DHCP/RA fail the connection will be torn down and
retried later, as is happening in the log from sosreport:

 <warn>  (loms): DHCPv4 request timed out.
 <warn>  (loms): Activation: failed for connection 'team-loms'

and this seems to be the cause of the cycles you are seeing. As
explained in [1], if the purpose of the team connection is to provide
only L2 connectivity for the VLAN, IP addressing should be disabled on
it, i.e.:

 # nmcli con add type team ifname loms
 # nmcli con mod team-loms ipv4.method disable ipv6.method ignore

Please try if this solves the issue.

[1] https://access.redhat.com/solutions/1608803
Comment 5 Shubhrata Priyadarshinee 2015-12-07 01:25:10 EST
Hi,
After Adding this above command, issue is not observed. 

Is this a expected behavior from NetworkManager or is a bug?

Document you have attached is support for Red Hat Enterprise Linux 7.1 or earlier. Need to add support for 7.2 also?
Comment 6 Beniamino Galvani 2015-12-07 09:01:55 EST
(In reply to Shubhrata Priyadarshinee from comment #5)
> Hi,
> After Adding this above command, issue is not observed. 
> 
> Is this a expected behavior from NetworkManager or is a bug?

It is the expected behavior, if no IP configuration is required on the team connection NM must be configured accordingly.

> Document you have attached is support for Red Hat Enterprise Linux 7.1 or
> earlier. Need to add support for 7.2 also?

Yes, I'll ask to update the documentation.

As this seems a configuration issue I'm closing the bug, please reopen if necessary.
Comment 7 Sujith 2015-12-21 05:38:25 EST
Hi Beniamino,
(In reply to Sujith from comment #1)
> I am not sure whether this is the reason why VLAN interface is changing its
> MAC address frequently. Can you tell us the reason behind this behavior?

Why does the MAC address of VLAN change whenever the VLAN interface goes into the cycle of interface state up->down->up.?

Regards,
Sujith
Comment 8 Beniamino Galvani 2015-12-21 08:43:37 EST
(In reply to Sujith from comment #7)
> Hi Beniamino,
> (In reply to Sujith from comment #1)
> > I am not sure whether this is the reason why VLAN interface is changing its
> > MAC address frequently. Can you tell us the reason behind this behavior?
> 
> Why does the MAC address of VLAN change whenever the VLAN interface goes
> into the cycle of interface state up->down->up.?

Hi Sujith,

the correct behavior you should see after configuring connections as
described in comment 4 is that the vlan always comes up with the same
MAC, which matches the one of the underlying device (bond/team).

The MAC of a bond or team in some cases in initially generated
randomly by kernel and then updated when new devices are enslaved.  In
the past a bug existed (see bug 1264322) which caused vlans to come
up with the initial random MAC address assigned to bond/team.  In order
to fix this, NM now tries to keep the MAC of a vlan aligned with the
one of the underlying device.  This is the reason of the behavior you
are seeing.
Comment 9 Sujith 2015-12-21 23:30:20 EST
(In reply to Beniamino Galvani from comment #8)
> the correct behavior you should see after configuring connections as
> described in comment 4 is that the vlan always comes up with the same
> MAC, which matches the one of the underlying device (bond/team).
> 
> The MAC of a bond or team in some cases in initially generated
> randomly by kernel and then updated when new devices are enslaved.  In
> the past a bug existed (see bug 1264322) which caused vlans to come
> up with the initial random MAC address assigned to bond/team.  In order
> to fix this,   This is the reason of the behavior you
> are seeing.

Hi Beniamino,
I do not have access to BZ#1264322. Can you grant access to me?
Disabling the IP works properly.
I am just trying to understand what was going wrong leading to this issue.

In the case of VLAN over bond-master, the MAC address of VLAN was not changing. It remained constant throughout. Just the interface was shifting between up & down state.

> NM now tries to keep the MAC of a vlan aligned with the one of the underlying 
> device.

I don't see this happening here. In the case of VLAN over team-master, the MAC was not inter-changing between the two, it was changing randomly. 
Hence DHCP-server was out of free-IPs after a time period.

Thanks and Regards,
Sujith
Comment 10 Beniamino Galvani 2015-12-22 03:58:59 EST
(In reply to Sujith from comment #9)
> Hi Beniamino,
> I do not have access to BZ#1264322. Can you grant access to me?

Sorry, I didn't notice that the bug was not publicly accessible.
Unfortunately most of comments are also private so I guess I can't
make it public.

> Disabling the IP works properly.
> I am just trying to understand what was going wrong leading to this issue.
>
> In the case of VLAN over bond-master, the MAC address of VLAN was not
> changing. It remained constant throughout. Just the interface was shifting
> between up & down state.

Ok, this is expected.

> > NM now tries to keep the MAC of a vlan aligned with the one of the underlying
> > device.
>
> I don't see this happening here. In the case of VLAN over team-master, the
> MAC was not inter-changing between the two, it was changing randomly.
> Hence DHCP-server was out of free-IPs after a time period.

Did you see both VLAN and team-master changing the MAC? Or only the
VLAN was getting a random one while the one on team-master was stable?

I can't reproduce this, do you happen to have NM logs with debug
enabled? (To enable debug do 'nmcli general logging level debug' or,
to make the change persistent, add "level=DEBUG" to the "[logging]"
section of /etc/NetworkManager/NetworkManager.conf and restart NM).

Thanks.
Comment 11 Sujith 2015-12-22 04:24:35 EST
(In reply to Beniamino Galvani from comment #10)

Hi Beniamino,

> Did you see both VLAN and team-master changing the MAC? Or only the
> VLAN was getting a random one while the one on team-master was stable?

It has been a while and I don't remember noting that point on team-master behavior.

> I can't reproduce this, do you happen to have NM logs with debug
> enabled? (To enable debug do 'nmcli general logging level debug' or,
> to make the change persistent, add "level=DEBUG" to the "[logging]"
> section of /etc/NetworkManager/NetworkManager.conf and restart NM).

Hmm..We had the setup removed after the bug was closed.
Give me a few days and I'll try setting it up again to collect logs.

Regards,
Sujith
Comment 12 Shubhrata Priyadarshinee 2016-01-20 07:45 EST
Created attachment 1116637 [details]
NM debug logs

(In reply to Beniamino Galvani from comment #10)
> I can't reproduce this, do you happen to have NM logs with debug
> enabled? (To enable debug do 'nmcli general logging level debug' or,
> to make the change persistent, add "level=DEBUG" to the "[logging]"
> section of /etc/NetworkManager/NetworkManager.conf and restart NM).

Hi Beniamino,
I attached the debug messages, where you can find how mac and inet4 address of VLAN over team master interface keeps on changing between the intervals.


Thanks
Shubhrata

Note You need to log in before you can comment on or make changes to this bug.