Bug 2123742

Summary:	Failed to enable unit: File networking-ovn-metadata-agent: Invalid cross-device link
Product:	[Community] RDO	Reporter:	Takashi Kajinami <tkajinam>
Component:	openstack-neutron	Assignee:	Takashi Kajinami <tkajinam>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Ofer Blaut <oblaut>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	trunk	CC:	chrisw, jcapitao, srevivo
Target Milestone:	---
Target Release:	trunk
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-12-08 04:53:31 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Takashi Kajinami 2022-09-02 13:28:33 UTC

Description of problem:

We see the puppet-openstack-integration-7-scenario00[35]-tempest-centos-9-stream job in upstream Puppet OpenStack project are consistently failing.
Puppet is not able to start the service and debug log shows
 # systemctl enable -- neutron-ovn-metadata-agent
is failing.

https://zuul.opendev.org/t/openstack/build/d94c3bb65df54adfb39312bbdc87df51/log/logs/puppet.txt#38824

I've managed to get output of the command and it is complaining about the link across different file systems.

$ sudo systemctl enable neutron-ovn-metadata-agent
Failed to enable unit: File networking-ovn-metadata-agent: Invalid cross-device link

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Takashi Kajinami 2022-09-02 13:30:08 UTC

This might be related to the recent bump of systemd package (systemd-250-7 -> systemd-250-11) but
I've not yet found the change which is causing the error.

Also, so far the issue is specific to the neutron-ovn-metadata-agent.service but I've not yet identified any specific setting we have for this service.

Comment 2 Takashi Kajinami 2022-09-02 13:36:58 UTC

Though the command shows error, the service is enabled as expected.

example:
```
* neutron-ovn-metadata-agent.service - OpenStack Neutron OVN Metadata Agent
     Loaded: loaded (/usr/lib/systemd/system/neutron-ovn-metadata-agent.service; enabled; vendor preset: disabled)
     Active: active (running) since Fri 2022-09-02 12:54:23 UTC; 9min ago
   Main PID: 125828 (neutron-ovn-met)
      Tasks: 3 (limit: 50480)
     Memory: 127.5M
        CPU: 3.783s
     CGroup: /system.slice/neutron-ovn-metadata-agent.service
             |-125828 "neutron-ovn-metadata-agent (/usr/bin/python3 /usr/bin/neutron-ovn-metadata-agent --config-file /etc/neutron/neutron_ovn_metadata_agent.ini --config-dir /etc/neutron/conf.d/neutron-ovn-metadata-agent --log-file /var/log/neutron/neutron-ovn-metadata-agent.log)"
             |-125852 "neutron-ovn-metadata-agent (/usr/bin/python3 /usr/bin/neutron-ovn-metadata-agent --config-file /etc/neutron/neutron_ovn_metadata_agent.ini --config-dir /etc/neutron/conf.d/neutron-ovn-metadata-agent --log-file /var/log/neutron/neutron-ovn-metadata-agent.log)"
             `-125853 "neutron-ovn-metadata-agent (/usr/bin/python3 /usr/bin/neutron-ovn-metadata-agent --config-file /etc/neutron/neutron_ovn_metadata_agent.ini --config-dir /etc/neutron/conf.d/neutron-ovn-metadata-agent --log-file /var/log/neutron/neutron-ovn-metadata-agent.log)"

Sep 02 12:54:23 centos-9-stream-rax-ord-0030951360 systemd[1]: Started OpenStack Neutron OVN Metadata Agent.
Sep 02 12:54:26 centos-9-stream-rax-ord-0030951360 sudo[125854]:  neutron : PWD=/ ; USER=root ; COMMAND=/usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf privsep-helper --config-file /etc/neutron/neutron_ovn_metadata_agent.ini --config-dir /etc/neutron/conf.d/neutron-ovn-metadata-agent --privsep_context neutron.privileged.namespace_cmd --privsep_sock_path /tmp/tmpptqao2d5/privsep.sock

``

Comment 3 Takashi Kajinami 2022-09-03 06:26:43 UTC

I've created bz 2123927 to get insights from systemd.

Comment 4 Takashi Kajinami 2022-09-05 12:29:40 UTC

So this turned out to be the issue caused by missing ".service" in Alias

[vagrant@localhost ~]$ sudo cat /usr/lib/systemd/system/neutron-ovn-metadata-agent.service 
[Unit]
Description=OpenStack Neutron OVN Metadata Agent
After=syslog.target network.target openvswitch.service
Requires=openvswitch.service

[Service]
Type=simple
User=neutron
PermissionsStartOnly=true
ExecStart=/usr/bin/neutron-ovn-metadata-agent --config-file /etc/neutron/neutron_ovn_metadata_agent.ini --config-dir /etc/neutron/conf.d/neutron-ovn-metadata-agent --log-file /var/log/neutron/neutron-ovn-metadata-agent.log
PrivateTmp=false
KillMode=process
Restart=on-failure

[Install]
WantedBy=multi-user.target
# (TODO) - Backwards compatibility in systemd service
Alias=networking-ovn-metadata-agent
[vagrant@localhost ~]$

If I fix the line by adding .service then it starts working.

Comment 5 Joel Capitao 2022-09-05 14:51:56 UTC

Thank you Takashi for the resolution, I merged it in master and managed to run a puppet promotion.

Comment 6 Takashi Kajinami 2022-12-08 04:53:31 UTC

The fix was merged in neutron-distgit and the zed release contains the fix so I think we can close this bug now.