Bug 1803607

How were the packages installed?

The documentation text flag should only be set after 'doc text' field is provided. Please provide the documentation text and set the flag to '?' again.

(In reply to Ryan Barry from comment #1)
> How were the packages installed?

from rpm packages. In my case, I used this:
https://packagecloud.io/prometheus-rpm/release/packages/el/7/node_exporter-0.18.1-1.el7.centos.x86_64.rpm

I meant: with "rpm -ivh" or yum?

(In reply to Ryan Barry from comment #4)
> I meant: with "rpm -ivh" or yum?

yum, custom repos, not Red Hat repos

The custom repo should not matter. Can you post an sosreport, please?

(In reply to Ryan Barry from comment #6)
> The custom repo should not matter. Can you post an sosreport, please?

Please check my file, sorry it's over attachment size:
https://drive.google.com/open?id=1jM79GXgesfgGwm9fqQ4G9mdVlaHwIkbn

Thanks.

So, imgbased repeatedly does try to reinstall this.

Le, can you please try "yum -y install /var/imgbased/persisted-rpms/node_exporter*" ?

2020-02-17 09:01:16,072 [DEBUG] (MainThread) Running ['systemd-nspawn', '--uuid', '93508011d7c54d6e81e626cdafdcd907', '--machine', 'nuc1.home.local', '-D', u'/tmp/mnt.m4ixC//', 'yum', 'install', '-y', '--noplugins', '/var/imgbased/persisted-rpms/node_exporter-0.18.1-1.el7.centos.x86_64.rpm']

(In reply to Ryan Barry from comment #8)
> So, imgbased repeatedly does try to reinstall this.
> 
> Le, can you please try "yum -y install
> /var/imgbased/persisted-rpms/node_exporter*" ?
> 
> 2020-02-17 09:01:16,072 [DEBUG] (MainThread) Running ['systemd-nspawn',
> '--uuid', '93508011d7c54d6e81e626cdafdcd907', '--machine',
> 'nuc1.home.local', '-D', u'/tmp/mnt.m4ixC//', 'yum', 'install', '-y',
> '--noplugins',
> '/var/imgbased/persisted-rpms/node_exporter-0.18.1-1.el7.centos.x86_64.rpm']

Please check my output:

Examining /var/imgbased/persisted-rpms/node_exporter-0.18.1-1.el7.centos.x86_64.rpm: node_exporter-0.18.1-1.el7.centos.x86_64
/var/imgbased/persisted-rpms/node_exporter-0.18.1-1.el7.centos.x86_64.rpm: does not update installed package.
Error: Nothing to do
Uploading Enabled Repositories Report
Loaded plugins: product-id, subscription-manager, versionlock
Unable to upload Enabled Repositories Report

Thanks!

We need to check if this affects only this specific package or if it affects any 3rd party service on node.

This looks like a subtle regression or missed case from https://bugzilla.redhat.com/show_bug.cgi?id=1380797

See also https://bugzilla.redhat.com/show_bug.cgi?id=1567908 which appears to be a case of a modified file in /etc not making it over

Can you please share the imgbased.log ?

Already part of the sosreport.

2020-02-17 09:00:37,297 [DEBUG] (migrate_etc) Calling binary: (['mount', u'/dev/rhvh/rhvh-4.3.8.1-0.20200126.0+1', u'/tmp/mnt.Hjdsy'],) {}
2020-02-17 09:00:37,297 [DEBUG] (migrate_etc) Calling: (['mount', u'/dev/rhvh/rhvh-4.3.8.1-0.20200126.0+1', u'/tmp/mnt.Hjdsy'],) {'close_fds': True, 'stderr': -2}
....
2020-02-17 09:00:40,114 [DEBUG] (migrate_etc) Calling binary: (['cp', '-a', '-r', u'/tmp/mnt.uDwNv///etc/systemd/system/multi-user.target.wants/node_exporter.service', u'/tmp/mnt.Hjdsy///etc/systemd/system/multi-user.target.wants/node_exporter.service'],) {}
2020-02-17 09:00:40,114 [DEBUG] (migrate_etc) Calling: (['cp', '-a', '-r', u'/tmp/mnt.uDwNv///etc/systemd/system/multi-user.target.wants/node_exporter.service', u'/tmp/mnt.Hjdsy///etc/systemd/system/multi-user.target.wants/node_exporter.service'],) {'close_fds': True, 'stderr': -2}

From the log, we can see node_exporter was installed after node_exporter.service was copied:

2020-02-17 09:00:40,114 [DEBUG] (migrate_etc) Calling: (['cp', '-a', '-r', u'/tmp/mnt.uDwNv///etc/systemd/system/multi-user.target.wants/node_exporter.service', u'/tmp/mnt.Hjdsy///etc/systemd/system/multi-user.target.wants/node_exporter.service'],) {'close_fds': True, 'stderr': -2}

2020-02-17 09:01:16,072 [DEBUG] (MainThread) Running ['systemd-nspawn', '--uuid', '93508011d7c54d6e81e626cdafdcd907', '--machine', 'nuc1.home.local', '-D', u'/tmp/mnt.m4ixC//', 'yum', 'install', '-y', '--noplugins', '/var/imgbased/persisted-rpms/node_exporter-0.18.1-1.el7.centos.x86_64.rpm']


The scripts of node_exporter rpm are:

[root@dell-per740-28 ~]# rpm -q --scripts node_exporter
preinstall scriptlet (using /bin/sh):
getent group prometheus >/dev/null || groupadd -r prometheus
getent passwd prometheus >/dev/null || \
  useradd -r -g prometheus -d /var/lib/prometheus -s /sbin/nologin \
          -c "Prometheus services" prometheus
exit 0
postinstall scriptlet (using /bin/sh):

if [ $1 -eq 1 ] ; then 
        # Initial installation 
        systemctl preset node_exporter.service >/dev/null 2>&1 || : 
fi
preuninstall scriptlet (using /bin/sh):

if [ $1 -eq 0 ] ; then 
        # Package removal, not upgrade 
        systemctl --no-reload disable node_exporter.service > /dev/null 2>&1 || : 
        systemctl stop node_exporter.service > /dev/null 2>&1 || : 
fi
postuninstall scriptlet (using /bin/sh):

systemctl daemon-reload >/dev/null 2>&1 || :


As you can see, in postinstall script, it calls "systemctl preset node_exporter.service", node_exporter.service will be set to disabled when install.

This issue shouldn't be specific to node_exporter, seems any service with disabled vendor preset will encounter this problem.

We could simply revers the actions:

1. Install the rpm
2. Copy the systemd-files back.

That should solve the issue.

Moving out to 4.5 due to capacity.

Is this reproducible on RHV 4.4?

(In reply to Sandro Bonazzola from comment #22)
> Is this reproducible on RHV 4.4?

QE can reproduce this issue on RHV 4.4.

Closing due to capacity. If we'll have capacity we can reopen it.