RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2175665 - nmstatectl service unit starts before NetworkManager dbus is ready
Summary: nmstatectl service unit starts before NetworkManager dbus is ready
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: nmstate
Version: 9.2
Hardware: Unspecified
OS: Linux
unspecified
high
Target Milestone: rc
: ---
Assignee: Gris Ge
QA Contact: Mingyu Shi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-03-06 09:32 UTC by Quique Llorente
Modified: 2023-11-07 09:02 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 2177188 (view as bug list)
Environment:
Last Closed: 2023-11-07 08:24:02 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github coreos fedora-coreos-config pull 2269#issuecomment-1455727702 0 None open networking-tools: Add nmstate 2023-03-06 09:32:17 UTC
Github nmstate nmstate pull 2265 0 None Merged nm: Retry when NetworkManager starting up 2023-03-13 06:08:48 UTC
Red Hat Issue Tracker NMT-361 0 None None None 2023-03-06 09:34:07 UTC
Red Hat Issue Tracker RHELPLAN-150714 0 None None None 2023-03-06 09:34:15 UTC
Red Hat Product Errata RHBA-2023:6323 0 None None None 2023-11-07 08:24:23 UTC

Description Quique Llorente 2023-03-06 09:32:17 UTC 
Description of problem:
When using the nmstate.service the state configured at /etc/nmstate is not applied with the following error 

Mar  6 07:50:05.269335 systemd[1]: Starting nmstate.service - Apply nmstate on-disk state...
Mar  6 07:50:05.277998 systemd[1]: Finished systemd-homed-activate.service - Home Area Activation.  
Mar  6 07:50:05.278000 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-homed-activate comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' 
Mar  6 07:50:05.280000 audit: BPF prog-id=67 op=LOAD                            
Mar  6 07:50:05.281000 audit: BPF prog-id=68 op=LOAD                            
Mar  6 07:50:05.281000 audit: BPF prog-id=69 op=LOAD                            
Mar  6 07:50:05.290390 systemd[1]: Starting systemd-hostnamed.service - Hostname Service...
Mar  6 07:50:05.294788 systemd[1]: Started chronyd.service - NTP client/server. 
Mar  6 07:50:05.298000 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=chronyd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Mar  6 07:50:05.308385 nmstatectl[1263]: [2023-03-06T07:50:05Z WARN  nmstate::nm::nm_dbus::error] Unknown DBUS error MethodError("org.freedesktop.DBus.Error.UnknownMethod", Some("Object does not exist at path “/org/freedesktop/NetworkManager”"), Msg { type: Error, sender: ":1.3", reply-serial: 2, body: Signature: [
Mar  6 07:50:05.308385 nmstatectl[1263]:             s (115),                    
Mar  6 07:50:05.308385 nmstatectl[1263]:     ] })                                
Mar  6 07:50:05.310309 nmstatectl[1263]: [2023-03-06T07:50:05Z ERROR nmstatectl::service] Failed to apply state file /etc/nmstate/br-ex.yml: NmstateError: Bug: DbusConnectionError: org.freedesktop.DBus.Error.UnknownMethod: Object does not exist at path “/org/freedesktop/NetworkManager”
Mar  6 07:50:05.310867 systemd[1]: nmstate.service: Deactivated successfully.    

Version-Release number of selected component (if applicable): 2.2.5-1.fc37


How reproducible: Always


Steps to Reproduce:
1. Add a configuration to /etc/nmstate/*.yml
2. Check configuration after startup
3.

Actual results:
State is not applied with the following error
Mar  6 07:50:05.269335 systemd[1]: Starting nmstate.service - Apply nmstate on-disk state...
Mar  6 07:50:05.277998 systemd[1]: Finished systemd-homed-activate.service - Home Area Activation.  
Mar  6 07:50:05.278000 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-homed-activate comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' 
Mar  6 07:50:05.280000 audit: BPF prog-id=67 op=LOAD                            
Mar  6 07:50:05.281000 audit: BPF prog-id=68 op=LOAD                            
Mar  6 07:50:05.281000 audit: BPF prog-id=69 op=LOAD                            
Mar  6 07:50:05.290390 systemd[1]: Starting systemd-hostnamed.service - Hostname Service...
Mar  6 07:50:05.294788 systemd[1]: Started chronyd.service - NTP client/server. 
Mar  6 07:50:05.298000 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=chronyd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Mar  6 07:50:05.308385 nmstatectl[1263]: [2023-03-06T07:50:05Z WARN  nmstate::nm::nm_dbus::error] Unknown DBUS error MethodError("org.freedesktop.DBus.Error.UnknownMethod", Some("Object does not exist at path “/org/freedesktop/NetworkManager”"), Msg { type: Error, sender: ":1.3", reply-serial: 2, body: Signature: [
Mar  6 07:50:05.308385 nmstatectl[1263]:             s (115),                    
Mar  6 07:50:05.308385 nmstatectl[1263]:     ] })                                
Mar  6 07:50:05.310309 nmstatectl[1263]: [2023-03-06T07:50:05Z ERROR nmstatectl::service] Failed to apply state file /etc/nmstate/br-ex.yml: NmstateError: Bug: DbusConnectionError: org.freedesktop.DBus.Error.UnknownMethod: Object does not exist at path “/org/freedesktop/NetworkManager”
Mar  6 07:50:05.310867 systemd[1]: nmstate.service: Deactivated successfully.   


Expected results:
State should be applied

Additional info:
There is already a fix https://github.com/nmstate/nmstate/pull/2265
Comment 1 Gris Ge 2023-03-06 09:35:04 UTC 
Patch posted to upstream https://github.com/nmstate/nmstate/pull/2265

To test the bug:
 * Store a desire state YAML file as /etc/nmstate/foo.yml
 * `sudo systemctl enable nmstate`
 * `reboot`
 * check whether desire state been applied. or `journalctl -u nmstate`
Comment 4 Mingyu Shi 2023-04-24 08:50:37 UTC 
Verified with:
nmstate-2.2.9-1.el9.x86_64
NetworkManager-1.43.6-1.el9.x86_64
DISTRO=RHEL-9.3.0-20230423.30

Put veth0.yml in /etc/nmstate:
dns-resolver: {}
route-rules: {}
routes:
  running:
  - destination: 2620:52:0:49c2::/64
    next-hop-interface: veth0
    next-hop-address: '::'
    metric: 100
    table-id: 254
  - destination: ::/0
    next-hop-interface: veth0
    next-hop-address: fe80::ee3e:f702:1d90:80a1
    metric: 100
    table-id: 254
  - destination: 0.0.0.0/0
    next-hop-interface: veth0
    next-hop-address: 10.73.195.254
    metric: 100
    table-id: 254
interfaces:
- name: veth0
  type: veth
  state: up
  veth:
    peer: veth0_p
  mac-address: 14:18:77:6D:51:08
  mtu: 8500
  min-mtu: 60
  max-mtu: 9000
  wait-ip: any
  ipv4:
    enabled: true
    dhcp: true
    address:
    - ip: 192.168.199.1
      prefix-length: 23
    auto-dns: true
    auto-gateway: true
    auto-routes: true
    auto-route-table-id: 0
  ipv6:
    enabled: true
    dhcp: true
    autoconf: true
    address:
    - ip: 2620:52:0:49c2:1618:77ff:fe6d:5107
      prefix-length: 64
    - ip: fe80::1618:77ff:fe6d:5107
      prefix-length: 64
    auto-dns: true
    auto-gateway: true
    auto-routes: true
    auto-route-table-id: 0
    addr-gen-mode: eui64
  accept-all-mac-addresses: false
  lldp:
    enabled: false

`systemctl enable nmstate` and reboot system, get:
[16:48:10@dell-per730-20 ~]0# nmcli con
NAME      UUID                                  TYPE      DEVICE  
veth0     c8130521-2694-42ab-b946-af1e9251b5f1  veth      veth0   
eno1      79a38ec3-db52-49cd-b172-85fc7f59cff9  ethernet  eno1    
lo        fda50bb7-a817-4024-9ad0-1db02fad5451  loopback  lo      
veth0_p   197ce7b3-4a2e-4cf2-a201-4ca4af6d621b  veth      veth0_p 
eno2      5ca4cb28-18bd-4493-9390-906024a2a123  ethernet  --      
eno3      1b5661b6-b42e-4f56-b816-20e7f4349c5e  ethernet  --      
eno4      d1f9d3af-87f1-468d-af04-c9347e61d367  ethernet  --      
enp6s0f0  c486a668-2d7d-4c17-8ad3-342d82c84483  ethernet  --      
enp6s0f1  82b43fd1-01a5-4e11-9f37-55ccdfc6da6c  ethernet  --      
[16:49:22@dell-per730-20 ~]0# nms show veth0
dns-resolver: {}
route-rules: {}
routes: {}
interfaces:
- name: veth0
  type: veth
  state: up
  mac-address: 14:18:77:6D:51:08
  mtu: 8500
  min-mtu: 68
  max-mtu: 65535
  wait-ip: any
  ipv4:
    enabled: true
    dhcp: true
    dhcp-client-id: ll
    auto-dns: true
    auto-gateway: true
    auto-routes: true
    auto-route-table-id: 0
  ipv6:
    enabled: true
    dhcp: true
    autoconf: true
    dhcp-duid: ll
    address:
    - ip: fe80::1618:77ff:fe6d:5108
      prefix-length: 64
    auto-dns: true
    auto-gateway: true
    auto-routes: true
    auto-route-table-id: 0
    addr-gen-mode: eui64
  accept-all-mac-addresses: false
  lldp:
    enabled: false
  ethtool:
    feature:
      tx-sctp-segmentation: true
      tx-generic-segmentation: true
      tx-gre-segmentation: true
      tx-ipxip4-segmentation: true
      rx-checksum: true
      tx-udp-segmentation: true
      rx-gro-list: false
      tx-scatter-gather-fraglist: true
      rx-udp-gro-forwarding: false
      tx-tcp-ecn-segmentation: true
      tx-udp_tnl-csum-segmentation: true
      tx-vlan-stag-hw-insert: true
      tx-checksum-sctp: true
      tx-gre-csum-segmentation: true
      tx-tcp6-segmentation: true
      tx-tcp-mangleid-segmentation: true
      rx-gro: false
      highdma: true
      tx-tcp-segmentation: true
      tx-vlan-hw-insert: true
      tx-ipxip6-segmentation: true
      rx-vlan-hw-parse: true
      tx-udp_tnl-segmentation: true
      tx-nocache-copy: false
      tx-gso-list: true
      rx-vlan-stag-hw-parse: true
      tx-checksum-ip-generic: true
  veth:
    peer: veth0_p
Comment 6 errata-xmlrpc 2023-11-07 08:24:02 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (nmstate bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:6323
Note You need to log in before you can comment on or make changes to this bug.