Bug 1393853

Summary: [NMCI] add team fails after clean install, NM service restart helps
Product: Red Hat Enterprise Linux 7 Reporter: Vladimir Benes <vbenes>
Component: NetworkManagerAssignee: Beniamino Galvani <bgalvani>
Status: CLOSED ERRATA QA Contact: Desktop QE <desktop-qa-list>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.3CC: asoni, atragler, bblaskov, bgalvani, fgiudici, jhunt, lrintel, mpoole, rkhan, sukulkar, systemd-maint-list, thaller
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: NetworkManager-1.8.0-0.2.git20170215.1d40c5f4.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-01 09:19:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
[PATCH] team: ignore SIGPIPE when spawning teamd none

Description Vladimir Benes 2016-11-10 13:03:30 UTC 
Description of problem:
basic team master adding via nmcli con add type team leads to device disappear as it looks like teamd dies. Not exactly sure the steps to reproduce but I can get broken env from our tests quite easily but it looks it's caused by systemctl restart journald.


Version-Release number of selected component (if applicable):
teamd-1.25-4.el7.x86_64
NetworkManager-1.4.0-12.el7.x86_64
systemd-219-30.el7.x86_64

How reproducible:
always after install

Steps to Reproduce:
1.install the machine
2.restart NM service
3.restart journald service

Actual results:
NM cannot create team as teamd dies

Expected results:
no issue with team creation

Additional info:
restarting NM fixes the issue
Comment 1 Beniamino Galvani 2016-11-10 13:21:28 UTC 
teamd terminates with an error because stderr in detached after the
restart of systemd-journald. NM logs:

  [1478776054.0408] device[0x7f44e8864c40] (team2): running: /usr/bin/teamd -o -n -U -D -N -t team2 -gg
  [1478776054.0695] device (team2): Activation: (team) started teamd [pid 31824]...
  [1478776054.0696] device[0x7f44e8864c40] (team2): activation-stage: complete activate_stage1_device_prepare,2 (id 2653)
  [1478776054.0697] kill child process 'teamd' (31743): terminated normally with status 0 (31712 usec elapsed)
  [1478776054.0698] device[0x7f44e8864c40] (team2): teamd not on D-Bus (ignored)
  [1478776054.0701] device[0x7f44e8864c40] (team2): teamd died with status 13
  [1478776054.0701] device (team2): teamd process quit unexpectedly; failing activation

strace of teamd:

  [...]
  mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f55cba69000
  read(3, "################################"..., 4096) = 1130
  read(3, "", 4096)           = 0
  close(3)                    = 0
  munmap(0x7f55cba69000, 4096) = 0
  mkdir("/var/run/teamd/", 0755) = -1 EEXIST (File exists)
  open("/dev/urandom", O_RDONLY) = 3
  read(3, "\275}M\271", 4)    = 4
  close(3)                    = 0
  write(2, "Using team device \"team2\".", 28) = -1 EPIPE (Broken pipe)
  --- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=23889, si_uid=0} ---

systemd-219 shipped with RHEL 7	should include this commit [1] which
temporarily stores fds to pid1 while restarting journald and so this
situation should be handled well:

  [1] https://github.com/systemd/systemd/commit/13790add4bf648fed8163

But apparently it is not working. I'm reassigning this to systemd
for analysis.
Comment 2 Beniamino Galvani 2016-11-10 14:21:10 UTC 
Created attachment 1219409 [details]
[PATCH] team: ignore SIGPIPE when spawning teamd

This is a workaround for NM to let the child ignore SIGPIPE and survive
the lack of stdout/stderr if journald was restarted. Anyway, it's
still a sub-optimal solution because any output of the child will be
lost.

Ideally this should be solved in systemd by properly restoring the
file descriptors after the restart of journald.
Comment 3 Thomas Haller 2016-11-10 14:48:05 UTC 
(In reply to Beniamino Galvani from comment #2)
> Created attachment 1219409 [details]
> [PATCH] team: ignore SIGPIPE when spawning teamd

lgtm
Comment 4 Thomas Haller 2017-03-02 08:59:38 UTC 
patch from comment 2 got merged to master as https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=a6d34f9ae348603eebc1e53067c164ab352293d3
Comment 10 errata-xmlrpc 2017-08-01 09:19:37 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2299