Description of problem: We see the puppet-openstack-integration-7-scenario00[35]-tempest-centos-9-stream job in upstream Puppet OpenStack project are consistently failing. Puppet is not able to start the service and debug log shows # systemctl enable -- neutron-ovn-metadata-agent is failing. https://zuul.opendev.org/t/openstack/build/d94c3bb65df54adfb39312bbdc87df51/log/logs/puppet.txt#38824 I've managed to get output of the command and it is complaining about the link across different file systems. $ sudo systemctl enable neutron-ovn-metadata-agent Failed to enable unit: File networking-ovn-metadata-agent: Invalid cross-device link Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
This might be related to the recent bump of systemd package (systemd-250-7 -> systemd-250-11) but I've not yet found the change which is causing the error. Also, so far the issue is specific to the neutron-ovn-metadata-agent.service but I've not yet identified any specific setting we have for this service.
Though the command shows error, the service is enabled as expected. example: ``` * neutron-ovn-metadata-agent.service - OpenStack Neutron OVN Metadata Agent Loaded: loaded (/usr/lib/systemd/system/neutron-ovn-metadata-agent.service; enabled; vendor preset: disabled) Active: active (running) since Fri 2022-09-02 12:54:23 UTC; 9min ago Main PID: 125828 (neutron-ovn-met) Tasks: 3 (limit: 50480) Memory: 127.5M CPU: 3.783s CGroup: /system.slice/neutron-ovn-metadata-agent.service |-125828 "neutron-ovn-metadata-agent (/usr/bin/python3 /usr/bin/neutron-ovn-metadata-agent --config-file /etc/neutron/neutron_ovn_metadata_agent.ini --config-dir /etc/neutron/conf.d/neutron-ovn-metadata-agent --log-file /var/log/neutron/neutron-ovn-metadata-agent.log)" |-125852 "neutron-ovn-metadata-agent (/usr/bin/python3 /usr/bin/neutron-ovn-metadata-agent --config-file /etc/neutron/neutron_ovn_metadata_agent.ini --config-dir /etc/neutron/conf.d/neutron-ovn-metadata-agent --log-file /var/log/neutron/neutron-ovn-metadata-agent.log)" `-125853 "neutron-ovn-metadata-agent (/usr/bin/python3 /usr/bin/neutron-ovn-metadata-agent --config-file /etc/neutron/neutron_ovn_metadata_agent.ini --config-dir /etc/neutron/conf.d/neutron-ovn-metadata-agent --log-file /var/log/neutron/neutron-ovn-metadata-agent.log)" Sep 02 12:54:23 centos-9-stream-rax-ord-0030951360 systemd[1]: Started OpenStack Neutron OVN Metadata Agent. Sep 02 12:54:26 centos-9-stream-rax-ord-0030951360 sudo[125854]: neutron : PWD=/ ; USER=root ; COMMAND=/usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf privsep-helper --config-file /etc/neutron/neutron_ovn_metadata_agent.ini --config-dir /etc/neutron/conf.d/neutron-ovn-metadata-agent --privsep_context neutron.privileged.namespace_cmd --privsep_sock_path /tmp/tmpptqao2d5/privsep.sock ``
I've created bz 2123927 to get insights from systemd.
So this turned out to be the issue caused by missing ".service" in Alias [vagrant@localhost ~]$ sudo cat /usr/lib/systemd/system/neutron-ovn-metadata-agent.service [Unit] Description=OpenStack Neutron OVN Metadata Agent After=syslog.target network.target openvswitch.service Requires=openvswitch.service [Service] Type=simple User=neutron PermissionsStartOnly=true ExecStart=/usr/bin/neutron-ovn-metadata-agent --config-file /etc/neutron/neutron_ovn_metadata_agent.ini --config-dir /etc/neutron/conf.d/neutron-ovn-metadata-agent --log-file /var/log/neutron/neutron-ovn-metadata-agent.log PrivateTmp=false KillMode=process Restart=on-failure [Install] WantedBy=multi-user.target # (TODO) - Backwards compatibility in systemd service Alias=networking-ovn-metadata-agent [vagrant@localhost ~]$ If I fix the line by adding .service then it starts working.
Thank you Takashi for the resolution, I merged it in master and managed to run a puppet promotion.
The fix was merged in neutron-distgit and the zed release contains the fix so I think we can close this bug now.