Bug 1750417

Summary: [RHEL7.8] invalid key/value pair in file /usr/lib/udev/rules.d/59-fc-wwpn-id.rules
Product: Red Hat Enterprise Linux 7 Reporter: Frank Liang <xiliang>
Component: sg3_utilsAssignee: Tomáš Bžatek <tbzatek>
Status: CLOSED ERRATA QA Contact: zguo <zguo>
Severity: high Docs Contact:
Priority: high    
Version: 7.8CC: akarvi, ben.argyle, bubrown, bugzilla, dhill, dlehman, ebarrera, fadamo, huali, jcastillo, jmagrini, joerg.kastning, jpittman, jtougne, lachlan.simpson, leiwang, linl, matt.wyen, mezhang, nrm, nyewale, peter.vreman, rhayden, rmadhuso, sbonazzo, schhabdi, shgao, tbzatek, vkuznets, wshi, yacao, ymao, ysudarev, zguo
Target Milestone: rcKeywords: ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: sg3_utils-1.37-19.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1788875 1788876 (view as bug list) Environment:
Last Closed: 2020-03-31 19:58:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1758223, 1788875, 1788876    

Description Frank Liang 2019-09-09 14:51:36 UTC
In RHEL7.8 compose build, find the following error messages from  systemd-udevd service and journal log.

[root@ip-10-116-2-141 ec2-user]# systemctl status systemd-udevd -l
● systemd-udevd.service - udev Kernel Device Manager
   Loaded: loaded (/usr/lib/systemd/system/systemd-udevd.service; static; vendor preset: disabled)
   Active: active (running) since Mon 2019-09-09 14:39:35 UTC; 27s ago
     Docs: man:systemd-udevd.service(8)
           man:udev(7)
 Main PID: 514 (systemd-udevd)
   Status: "Processing with 12 children at max"
    Tasks: 1
   CGroup: /system.slice/systemd-udevd.service
           └─514 /usr/lib/systemd/systemd-udevd

Sep 09 14:39:35 ip-10-116-2-141.us-west-2.compute.internal systemd[1]: Starting udev Kernel Device Manager...
Sep 09 14:39:35 ip-10-116-2-141.us-west-2.compute.internal systemd-udevd[514]: starting version 219
Sep 09 14:39:35 ip-10-116-2-141.us-west-2.compute.internal systemd-udevd[514]: Network interface NamePolicy= disabled on kernel command line, ignoring.
Sep 09 14:39:35 ip-10-116-2-141.us-west-2.compute.internal systemd-udevd[514]: invalid key/value pair in file /usr/lib/udev/rules.d/59-fc-wwpn-id.rules on line 10, starting at character 26 (';')
Sep 09 14:39:35 ip-10-116-2-141.us-west-2.compute.internal systemd-udevd[514]: invalid key/value pair in file /usr/lib/udev/rules.d/59-fc-wwpn-id.rules on line 11, starting at character 29 (';')
Sep 09 14:39:35 ip-10-116-2-141.us-west-2.compute.internal systemd-udevd[514]: invalid key/value pair in file /usr/lib/udev/rules.d/59-fc-wwpn-id.rules on line 12, starting at character 25 (';')
Sep 09 14:39:35 ip-10-116-2-141.us-west-2.compute.internal systemd[1]: Started udev Kernel Device Manager.

[root@ip-10-116-2-141 ec2-user]# rpm -qa|grep sg3
sg3_utils-libs-1.44-4.el7.x86_64
sg3_utils-1.44-4.el7.x86_64

After replaced ';' with ',', there is no such errors.
# diff -u 59-fc-wwpn-id.rules.orig /usr/lib/udev/rules.d/59-fc-wwpn-id.rules
--- 59-fc-wwpn-id.rules.orig 2019-09-09 22:25:25.050859949 +0800
+++ /usr/lib/udev/rules.d/59-fc-wwpn-id.rules 2019-09-09 22:25:39.136891783 +0800
@@ -7,9 +7,9 @@
 
-ENV{FC_TARGET_WWPN}!="$*"; GOTO="fc_wwpn_end"
-ENV{FC_INITIATOR_WWPN}!="$*"; GOTO="fc_wwpn_end"
-ENV{FC_TARGET_LUN}!="$*"; GOTO="fc_wwpn_end"
+ENV{FC_TARGET_WWPN}!="$*", GOTO="fc_wwpn_end"
+ENV{FC_INITIATOR_WWPN}!="$*", GOTO="fc_wwpn_end"
+ENV{FC_TARGET_LUN}!="$*", GOTO="fc_wwpn_end"

[root@ip-10-116-2-125 ec2-user]# systemctl status systemd-udevd -l
● systemd-udevd.service - udev Kernel Device Manager
   Loaded: loaded (/usr/lib/systemd/system/systemd-udevd.service; static; vendor preset: disabled)
   Active: active (running) since Mon 2019-09-09 14:33:13 UTC; 13min ago
     Docs: man:systemd-udevd.service(8)
           man:udev(7)
 Main PID: 513 (systemd-udevd)
   Status: "Processing with 12 children at max"
    Tasks: 1
   CGroup: /system.slice/systemd-udevd.service
           └─513 /usr/lib/systemd/systemd-udevd

Sep 09 14:33:13 ip-10-116-2-125.us-west-2.compute.internal systemd[1]: Starting udev Kernel Device Manager...
Sep 09 14:33:13 ip-10-116-2-125.us-west-2.compute.internal systemd-udevd[513]: starting version 219
Sep 09 14:33:13 ip-10-116-2-125.us-west-2.compute.internal systemd-udevd[513]: Network interface NamePolicy= disabled on kernel command line, ignoring.
Sep 09 14:33:13 ip-10-116-2-125.us-west-2.compute.internal systemd[1]: Started udev Kernel Device Manager.

Version-Release number of selected component (if applicable):

RHEL Version:
RHEL7.8(3.10.0-1084.el7.x86_64)

How reproducible:
100%

Steps to Reproduce:
1. Start a RHEL7.8 t3a.large instance
2. Check systemd-udevd status or journal log.
# systemctl status systemd-udevd -l

Actual results:
invalid key/value pair in file
Expected results:
invalid key/value pair in file

Additional info:
- N/A

Comment 2 rhayden 2019-09-16 17:36:45 UTC
FYI - This issue is also in the RHEL 7.7 errata for sg3_utils-1.37-18.el7_7.1.x86_64.   Appears that the UDEV rule was introduced in the 1.37-18.el7_7.1 Errata.   Looking at previous version, sg3_utils-1.37-18.el7.x86_64, I don't see the UDEV rule.

Comment 3 Yuri Arabadji 2019-09-18 14:19:53 UTC
Seeing the same after upgrade to 7.7.

Comment 4 Tomáš Bžatek 2019-09-24 11:59:08 UTC
FYI, fix for this issue appeared upstream just on Sep 11. The following packages are affected:
  sg3_utils-1.37-17.el7_6.1
  sg3_utils-1.37-18.el7_7.1
  sg3_utils-1.44-4.el7

Comment 6 zguo 2019-09-26 02:35:18 UTC
Reproducer:

$ rpm -q sg3_utils sg3_utils-libs
sg3_utils-1.44-4.el7.x86_64
sg3_utils-libs-1.44-4.el7.x86_64

or
$ rpm -q sg3_utils sg3_utils-libs
sg3_utils-1.37-17.el7_6.1.x86_64
sg3_utils-libs-1.37-17.el7_6.1.x86_64

or
$ rpm -q sg3_utils sg3_utils-libs
sg3_utils-1.37-18.el7_7.1.x86_64
sg3_utils-libs-1.37-18.el7_7.1.x86_64


[root@rdma-qe-02 ~]$ systemctl status systemd-udevd -l
● systemd-udevd.service - udev Kernel Device Manager
   Loaded: loaded (/usr/lib/systemd/system/systemd-udevd.service; static; vendor preset: disabled)
   Active: active (running) since Wed 2019-09-25 22:26:20 EDT; 1min 16s ago
     Docs: man:systemd-udevd.service(8)
           man:udev(7)
 Main PID: 5861 (systemd-udevd)
   Status: "Processing with 24 children at max"
    Tasks: 1
   Memory: 2.9M
   CGroup: /system.slice/systemd-udevd.service
           └─5861 /usr/lib/systemd/systemd-udevd

Sep 25 22:26:19 rdma-qe-02.lab.bos.redhat.com systemd[1]: Starting udev Kernel Device Manager...
Sep 25 22:26:20 rdma-qe-02.lab.bos.redhat.com systemd-udevd[5861]: invalid key/value pair in file /usr/lib/udev/rules.d/59-fc-wwpn-id.rules on li>
Sep 25 22:26:20 rdma-qe-02.lab.bos.redhat.com systemd-udevd[5861]: invalid key/value pair in file /usr/lib/udev/rules.d/59-fc-wwpn-id.rules on li>
Sep 25 22:26:20 rdma-qe-02.lab.bos.redhat.com systemd-udevd[5861]: invalid key/value pair in file /usr/lib/udev/rules.d/59-fc-wwpn-id.rules on li>
Sep 25 22:26:20 rdma-qe-02.lab.bos.redhat.com systemd[1]: Started udev Kernel Device Manager.

Verification:

$ rpm -q sg3_utils sg3_utils-libs
sg3_utils-1.44-5.el7.x86_64
sg3_utils-libs-1.44-5.el7.x86_64

[root@rdma-qe-02 ~]$ systemctl restart systemd-udevd -l
[root@rdma-qe-02 ~]$ systemctl status systemd-udevd -l
● systemd-udevd.service - udev Kernel Device Manager
   Loaded: loaded (/usr/lib/systemd/system/systemd-udevd.service; static; vendor preset: disabled)
   Active: active (running) since Wed 2019-09-25 22:27:46 EDT; 3s ago
     Docs: man:systemd-udevd.service(8)
           man:udev(7)
 Main PID: 6418 (systemd-udevd)
   Status: "Processing with 24 children at max"
    Tasks: 1
   Memory: 2.7M
   CGroup: /system.slice/systemd-udevd.service
           └─6418 /usr/lib/systemd/systemd-udevd

Sep 25 22:27:45 rdma-qe-02.lab.bos.redhat.com systemd[1]: Starting udev Kernel Device Manager...
Sep 25 22:27:46 rdma-qe-02.lab.bos.redhat.com systemd[1]: Started udev Kernel Device Manager.

Comment 7 David Hill 2019-10-03 20:32:32 UTC
This issue also affects RHOSP ramdisks and overcloud images.

Comment 8 John Pittman 2019-10-04 15:04:01 UTC
Tomas, any chance you can add https://github.com/hreinecke/sg3_utils/pull/46 (if accepted) when you post the fix?

Comment 9 Nitin Yewale 2019-10-07 09:04:20 UTC
Hi,

One of our customers wishes to backport the fix in RHEL 7.7 as well.

Thank you,
Nitin Yewale

Comment 13 Ben 2019-10-24 07:00:00 UTC
Adding another request for this to be backported to 7.7, please.  We just updated and had to make the change manually.  We've got a _lot_ more servers to update and I don't want to have to fix them all too (-:

We're now on 

sg3_utils-1.37-18.el7_7.1.x86_64
sg3_utils-libs-1.37-18.el7_7.1.x86_64

Comment 16 jtougne 2019-12-05 14:49:16 UTC
Hi,
Also impacting our customer, which uses RHEL7.7:

10:25:07.578
4h 19m 36s ago
@timestamp2019-12-05T10:25:07.578Zmessageinvalid key/value pair in file /usr/lib/udev/rules.d/59-fc-wwpn-id.rules on line 10, starting at character 26 (';')hostzges-tb02-i54x-02severityerrfacilitydaemonsyslog-tagsystemd-udevd:sourcesystemd-udevd@timestamp_received2019-12-05T10:25:28.197Z


rpm -q sg3_utils sg3_utils-libs
sg3_utils-1.37-18.el7_7.1.x86_64
sg3_utils-libs-1.37-18.el7_7.1.x86_64


[root@ostp6-cpx4-t4 log]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.7 (Maipo)
[root@ostp6-cpx4-t4 log]# date
Thu Dec  5 15:45:51 CET 2019

Cheers,

Comment 17 jtougne 2019-12-05 14:49:50 UTC
Hi,
Also impacting our customer, which uses RHEL7.7:

10:25:07.578
4h 19m 36s ago
@timestamp2019-12-05T10:25:07.578Zmessageinvalid key/value pair in file /usr/lib/udev/rules.d/59-fc-wwpn-id.rules on line 10, starting at character 26 (';')hostzges-tb02-i54x-02severityerrfacilitydaemonsyslog-tagsystemd-udevd:sourcesystemd-udevd@timestamp_received2019-12-05T10:25:28.197Z


rpm -q sg3_utils sg3_utils-libs
sg3_utils-1.37-18.el7_7.1.x86_64
sg3_utils-libs-1.37-18.el7_7.1.x86_64


[root@ostp6-cpx4-t4 log]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.7 (Maipo)
[root@ostp6-cpx4-t4 log]# date
Thu Dec  5 15:45:51 CET 2019

Cheers,

Comment 19 Tomáš Bžatek 2019-12-10 15:58:38 UTC
*** Bug 1763604 has been marked as a duplicate of this bug. ***

Comment 33 Tomáš Bžatek 2020-01-13 10:55:58 UTC
*** Bug 1760847 has been marked as a duplicate of this bug. ***

Comment 39 Matt Wyen 2020-02-03 19:36:25 UTC
Hello,

This is also affecting a server of mine

invalid key/value pair in file /usr/lib/udev/rules.d/59-fc-wwpn-id.rules on line 12, starting at character 25 (';')


rpm -q sg3_utils sg3_utils-libs
sg3_utils-1.37-18.el7_7.1.x86_64
sg3_utils-libs-1.37-18.el7_7.1.x86_64

[root@null ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.7 (Maipo)
[root@null ~]# date
Mon Feb  3 19:35:59 GMT 2020

Comment 41 errata-xmlrpc 2020-03-31 19:58:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:1093