Bug 2032031

Summary: Pacemaker lists 'service' agents ending in '@' but is unable to provide metadata for them
Product: Red Hat Enterprise Linux 9 Reporter: Ken Gaillot <kgaillot>
Component: pacemakerAssignee: Ken Gaillot <kgaillot>
Status: CLOSED ERRATA QA Contact: cluster-qe <cluster-qe>
Severity: low Docs Contact:
Priority: medium    
Version: 9.0CC: clumens, cluster-maint, cluster-qe, kgaillot, msmazova, tojeline
Target Milestone: rcKeywords: Triaged
Target Release: 9.0Flags: pm-rhel: mirror+
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: pacemaker-2.1.2-2.el9 Doc Type: Bug Fix
Doc Text:
This affects a very small subset of users, and is mainly for pcs, so a release note is not needed. Cause: Pacemaker did not recognize systemd service names ending in "@" when resolving a "service:" resource specification. Consequence: Attempting to show meta-data for such a resource would show an error. Fix: Pacemaker now recognizes systemd service names ending in "@". Result: Meta-data queries for such resources succeed.
Story Points: ---
Clone Of: 2003151 Environment:
Last Closed: 2022-05-17 12:20:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2003151    
Bug Blocks:    

Description Ken Gaillot 2021-12-13 21:57:29 UTC
+++ This bug was initially created as a clone of Bug #2003151 +++

Description of problem:
When listing resource agents using 'crm_resource', agents of 'service' standard are listed which contain '@' in their name. Yet, crm_resource is unable to provide metadata for them.


Version-Release number of selected component (if applicable):
pacemaker-2.1.0-8.el8


How reproducible:
always, easily


Steps to Reproduce:
1. list 'service' agents, notice there are few with '@'
# crm_resource --list-agents service | grep @
autovt@
chrony-dnssrv@
chrony-dnssrv@.timer
container-getty@
getty@
lvm2-pvscan@
rdma-load-modules@
serial-getty@
sshd-keygen@
sshd@
systemd-backlight@
systemd-coredump@
systemd-fsck@
systemd-hibernate-resume@
teamd@
user-runtime-dir@
user@

2. try to fetch metadata of such an agent
# crm_resource --show-metadata service:sshd@
crm_resource: Metadata query for service:sshd@ failed: No such device or address
Error performing operation: No such object


Actual results:
Unable to fetch metadata.


Expected results:
Either be able to provide metadata for such agents, or don't list them.


Additional info:
The 'service' standard is an alias to lsb, systemd and other init systems. Agents with '@' in their name come most likely from systemd. It seems, however, that pacemaker fails in figuring out the actual service class providing the specified agent:

# crm_resource --show-metadata service:sshd@ -VVVV
(crm_log_args)  notice: Invoked: crm_resource --show-metadata service:sshd@ -VVVV
(expand_resource_class)         info: Assuming resource class lsb for agent sshd@ for sshd@
(lrmd_api_get_metadata_params)  error: Failed to retrieve meta-data for service:(null):sshd@
(lrmd_api_disconnect)   info: Disconnecting IPC local executor connection
crm_resource: Metadata query for service:sshd@ failed: No such device or address
Error performing operation: No such object
(crm_xml_cleanup)       info: Cleaning up memory from libxml2
(crm_exit)      info: Exiting crm_resource | with status 105

Note: expand_resource_class assumes lsb:sshd@ instead of systemd:sshd@ even though there seem to be no lsb initscripts installed

# ls -la /etc/init.d/
total 32
drwxr-xr-x.  2 root root  4096 May 24 14:07 .
drwxr-xr-x. 10 root root  4096 Feb 15  2021 ..
-rw-r--r--.  1 root root 18434 Feb 15  2021 functions
-rw-r--r--.  1 root root  1161 Mar 11  2021 README

--- Additional comment from Ken Gaillot on 2021-09-10 19:56:42 UTC ---

You're right, this is a bug in service handling. I think the issue is ultimately in systemd_unit_by_name() not handling names with "@" specially.

--- Additional comment from Ken Gaillot on 2021-09-20 21:38:11 UTC ---

It's worth noting that any of these forms already work for a systemd unit that uses "@":

    crm_resource --show-metadata service:$UNIT
    crm_resource --show-metadata service:$UNIT@$SUBUNIT
    crm_resource --show-metadata systemd:$UNIT@

so the problem is limited to the one case

    crm_resource --show-metadata service:$UNIT@

--- Additional comment from Chris Lumens on 2021-11-30 18:55:08 UTC ---

(In reply to Ken Gaillot from comment #2)
> It's worth noting that any of these forms already work for a systemd unit
> that uses "@":
> 
>     crm_resource --show-metadata service:$UNIT
>     crm_resource --show-metadata service:$UNIT@$SUBUNIT
>     crm_resource --show-metadata systemd:$UNIT@
> 
> so the problem is limited to the one case
> 
>     crm_resource --show-metadata service:$UNIT@

It's a little trickier than that.  I think in this case, the top three commands work only because there is both an sshd.service and an sshd@.service file.  If you try something that doesn't have both (like user@service), the result is:

[root@cluster01 system]# crm_resource --show-metadata service:user
crm_resource: Metadata query for service:user failed: No such device or address
Error performing operation: No such object
[root@cluster01 system]# crm_resource --show-metadata service:user@
crm_resource: Metadata query for service:user@ failed: No such device or address
Error performing operation: No such object

--- Additional comment from Chris Lumens on 2021-11-30 19:17:30 UTC ---

If you add enough -Vs, you'll see what is going on:

# crm_resource --show-metadata service:sshd@ -VVVVVVV
...
(pcmk_dbus_find_error)  trace: DBus reply indicated error 'org.freedesktop.DBus.Error.InvalidArgs' (Unit name sshd@.service is missing the instance name.)
(pcmk_dbus_find_error)  trace: DBus reply indicated error 'org.freedesktop.DBus.Error.InvalidArgs' (Unit name sshd@.service is missing the instance name.)
(expand_resource_class)         info: Assuming resource class lsb for agent sshd@ for sshd@
(services__get_lsb_metadata)    trace: Looking into /etc/rc.d/init.d/sshd@

And sure enough, if you try cramming some text after the @ sign (the instance name):
# crm_resource --show-metadata service:sshd@root
<?xml version="1.0"?>
...

It works.

--- Additional comment from Ken Gaillot on 2021-12-01 00:10:52 UTC ---

Fixed in upstream master branch by commit 09ef95a

Comment 5 Markéta Smazová 2022-01-25 17:41:25 UTC
before fix
-------------

See bz2003151#c10


after fix
----------
>   [root@virt-029 ~]# rpm -q pacemaker
>   pacemaker-2.1.2-2.el9.x86_64


List "service" agents with "@" in name:

>   [root@virt-029 ~]# crm_resource --list-agents service | grep @
>   autovt@
>   container-getty@
>   getty@
>   modprobe@
>   qarshd@
>   serial-getty@
>   sshd-keygen@
>   sshd@
>   systemd-backlight@
>   systemd-coredump@
>   systemd-fsck@
>   systemd-hibernate-resume@
>   systemd-journald-varlink@.socket
>   systemd-journald@
>   systemd-journald@.socket
>   teamd@
>   user-runtime-dir@
>   user@

Try to show metadata:

>   [root@virt-029 ~]# crm_resource --show-metadata service:container-getty@
>   <?xml version="1.0"?>
>   <!DOCTYPE resource-agent SYSTEM "ra-api-1.dtd">
>   <resource-agent name="container-getty@" version="0.1">
>     <version>1.1</version>
>     <longdesc lang="en">
>       Container Getty on /dev/pts/x
>     </longdesc>
>     <shortdesc lang="en">systemd unit file for container-getty@</shortdesc>
>     <parameters/>
>     <actions>
>       <action name="start"     timeout="100" />
>       <action name="stop"      timeout="100" />
>       <action name="status"    timeout="100" />
>       <action name="monitor"   timeout="100" interval="60"/>
>       <action name="meta-data" timeout="5"   />
>     </actions>
>     <special tag="systemd"/>
>   </resource-agent>

>   [root@virt-029 ~]# crm_resource --show-metadata service:sshd@
>   <?xml version="1.0"?>
>   <!DOCTYPE resource-agent SYSTEM "ra-api-1.dtd">
>   <resource-agent name="sshd@" version="0.1">
>     <version>1.1</version>
>     <longdesc lang="en">
>       OpenSSH per-connection server daemon
>     </longdesc>
>     <shortdesc lang="en">systemd unit file for sshd@</shortdesc>
>     <parameters/>
>     <actions>
>       <action name="start"     timeout="100" />
>       <action name="stop"      timeout="100" />
>       <action name="status"    timeout="100" />
>       <action name="monitor"   timeout="100" interval="60"/>
>       <action name="meta-data" timeout="5"   />
>     </actions>
>     <special tag="systemd"/>
>   </resource-agent>


Metadata for "service" agent `service:systemd-journald-varlink@.socket` can't be fetched. This has been reported as bug2045110

>   [root@virt-029 ~]# crm_resource --show-metadata service:systemd-journald-varlink@.socket
>   crm_resource: Metadata query for service:systemd-journald-varlink@.socket failed: No such device or address
>   Error performing operation: No such object



verified in pacemaker-2.1.2-2.el9, one issue has been found and reported as bug2045110

Comment 7 errata-xmlrpc 2022-05-17 12:20:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: pacemaker), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2293