Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be unavailable on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1491451 - Pushing osad-5.11.63-12.el7sat.noarch or osa-common-5.11.63-12.el7sat.noarch to clients through Satellite aborts the yum transaction, leaving duplicates on the system
Summary: Pushing osad-5.11.63-12.el7sat.noarch or osa-common-5.11.63-12.el7sat.noarch ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite 5
Classification: Red Hat
Component: Client
Version: 580
Hardware: Unspecified
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Tomáš Kašpárek
QA Contact: Pavel Studeník
URL:
Whiteboard:
: 1491362 1494251 (view as bug list)
Depends On: 1494389
Blocks: sat58-errata
TreeView+ depends on / blocked
 
Reported: 2017-09-13 20:42 UTC by jcastran
Modified: 2021-03-11 15:46 UTC (History)
14 users (show)

Fixed In Version: osad-5.11.63-14-sat
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-01-28 14:45:27 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:0137 0 normal SHIPPED_LIVE Red Hat Network Tools bug fix update 2018-01-28 19:44:29 UTC

Description jcastran 2017-09-13 20:42:13 UTC
Description of problem:
  Using the Satellite UI to push osad-5.11.63-12.el7sat.noarch and/or osa-common-5.11.63-12.el7sat.noarch as updates to client will fail. 
  The yum transaction shows "aborted" and the  system is left with duplicates. 

 - If just osad was pushed, the duplicates have only been osad and osa-common.

 - If osad is included in a large update, the entire update fails leaving many duplicates behind.

Version-Release number of selected component (if applicable):
   osad-5.11.63-12.el7sat.noarch
   osa-common-5.11.63-12.el7sat.noarch

How reproducible:
   Everytime

Steps to Reproduce:
  *Selinux is enforcing
  *RHEL 7 client registered to Satellite 5   
   1. In the satellite UI, target the client system
   2. Push updates for osad and osa-common to the client
   3. On the client:

Actual results:
  The update is marked as aborted in yum history. We also now see duplicates
     # yum history info  last
     # rpm -qa  | grep osa
           osa-common-5.11.63-12.el7sat.noarch
           osad-5.11.63-12.el7sat.noarch
           osa-common-5.11.63-11.el7sat.noarch
           osad-5.11.63-11.el7sat.noarch


Expected results:
Update succeeds without interrupting yum.

Additional info:
- When recreating this, if selinux was permissive, the update would be fine
- The client can run "yum update osad" without issue. This does not break
- Running rhn_check to find the job also results with this same issue.

Comment 2 Tomas Lestach 2017-09-14 08:20:14 UTC
This sounds very similar to Bug 1491362. Can anyone confirm Bug 1491362 and Bug 1491451 are duplicates?

Comment 3 Pavel Studeník 2017-09-14 10:38:24 UTC
Hi.
I am trying investigate more about this issue. It is only problem of update latest version osad and osa-common. Old packages are not removed and stay as duplicated.

If update is executed by rhn_check/yum, then it finishes correctly. Problem causes only osad which updates oneself. Probably reason of the issue can be sending signal during updating (restart service).

>> getenforce 
Permissive

.. update osad/osa-common

>> yum history 
...
Warning: RPMDB altered outside of yum.
** Found 4 pre-existing rpmdb problem(s), 'yum check' output follows:
osa-common-5.11.63-11.el7sat.noarch has installed conflicts osad > ('0', '5.11.63', '11.el7sat'): osad-5.11.63-12.el7sat.noarch
osa-common-5.11.63-12.el7sat.noarch has installed conflicts osad < ('0', '5.11.63', '12.el7sat'): osad-5.11.63-11.el7sat.noarch
osa-common-5.11.63-12.el7sat.noarch is a duplicate with osa-common-5.11.63-11.el7sat.noarch
osad-5.11.63-12.el7sat.noarch is a duplicate with osad-5.11.63-11.el7sat.noarch
history list

>> yum history info ..
Loaded plugins: product-id, rhnplugin, search-disabled-repos, subscription-manager
This system is receiving updates from RHN Classic or Red Hat Satellite.
Transaction ID : 19
Begin time     : Thu Sep 14 11:50:01 2017
Begin rpmdb    : 598:1dabd320ee0c74f9034aa3c3f3bdc450948486e0
User           : System <unset>
Return-Code    : ** Aborted **
Transaction performed with:
    Installed     rpm-4.11.3-21.el7.x86_64                @beaker-Client/7.3
    Installed     yum-3.4.3-150.el7.noarch                @beaker-Client/7.3
    Installed     yum-metadata-parser-1.1.4-10.el7.x86_64 @beaker-Client/7.3
    Installed     yum-rhn-plugin-2.0.1-6.el7.noarch       @beaker-Client/7.3
Packages Altered:
 ** Updated osa-common-5.11.63-11.el7sat.noarch @rhn-tools-rhel-x86_64-client-7
 ** Update             5.11.63-12.el7sat.noarch @rhn-tools-rhel-x86_64-client-7
 ** Updated osad-5.11.63-11.el7sat.noarch       @rhn-tools-rhel-x86_64-client-7
 ** Update       5.11.63-12.el7sat.noarch       @rhn-tools-rhel-x86_64-client-7
history info

# rpm -qa | grep osa
osa-common-5.11.63-11.el7sat.noarch
osad-5.11.63-11.el7sat.noarch
osa-common-5.11.63-12.el7sat.noarch
osad-5.11.63-12.el7sat.noarch

In audit I found 3 AVC messages (needed disable dontaudit):
>> semodule -DB
>> tail /var/log/audit/audit.log
...
type=AVC msg=audit(1505383812.993:127): avc:  denied  { rlimitinh } for  pid=13173 comm="rhn_check" scontext=system_u:system_r:osad_t:s0 tcontext=system_u:system_r:rpm_t:s0 tclass=process
type=AVC msg=audit(1505383812.993:127): avc:  denied  { siginh } for  pid=13173 comm="rhn_check" scontext=system_u:system_r:osad_t:s0 tcontext=system_u:system_r:rpm_t:s0 tclass=process
type=AVC msg=audit(1505383812.993:127): avc:  denied  { noatsecure } for  pid=13173 comm="rhn_check" scontext=system_u:system_r:osad_t:s0 tcontext=system_u:system_r:rpm_t:s0 tclass=process

But it looks that it is not problem with selinux, because selinux is in permissive mode.

Comment 4 Pavel Studeník 2017-09-14 12:13:18 UTC
Try to call remote command to update osad/osa-common "yum update -y osad osa-common" by webui and osad. It causes same problem.

>> tail /var/log/up2date
[Thu Sep 14 14:07:52 2017] up2date successfully retrieved authentication token from up2date server
[Thu Sep 14 14:07:58 2017] up2date 
Traceback (most recent call last):
  File "/usr/sbin/rhn_check", line 347, in __run_action
    (status, message, data) = CheckCli.__do_call(method, params, kwargs)
  File "/usr/sbin/rhn_check", line 340, in __do_call
    retval = method(*params, **kwargs)
  File "/usr/share/rhn/actions/script.py", line 260, in run
    input_fds, output_fds, error_fds = select.select([pipe_read], [], [], select_wait)
  File "/usr/share/rhn/actions/script.py", line 58, in handle
    raise SystemExit(SYSEXIT_CODE)
<type 'exceptions.SystemExit'>: 3

[Thu Sep 14 14:07:59 2017] up2date Updating package profile

>> rpm -qa | grep osa
osad-5.11.63-12.el7sat.noarch
osa-common-5.11.63-11.el7sat.noarch
osa-common-5.11.63-12.el7sat.noarch
osad-5.11.63-11.el7sat.noarch

Comment 5 Pavel Studeník 2017-09-14 12:46:18 UTC
So.. next information. Problem is that you try to remove package osad by osad. Reproducer is unistnallation of package osad and this task is picked up by osad.

...
Yum version: 3.4.3
Transaction ID : 51
Begin time     : Thu Sep 14 14:34:55 2017
Begin rpmdb    : 607:f2a1d3715a2b98d6b5c022537d46f6c5a9ed47ec
User           : System <unset>
Return-Code    : ** Aborted **
Transaction performed with:
rpmdb time: 0.000
    Installed     rpm-4.11.3-21.el7.x86_64                @beaker-Client/7.3
    Installed     yum-3.4.3-150.el7.noarch                @beaker-Client/7.3
    Installed     yum-metadata-parser-1.1.4-10.el7.x86_64 @beaker-Client/7.3
    Installed     yum-rhn-plugin-2.0.1-6.el7.noarch       @beaker-Client/7.3
Packages Altered:
 ** Erase osad-5.11.63-12.el7sat.noarch installed
history info

The post install phase isn't finished correct. Update is same issue - yum installs new package then uninstalls old (and restart (stop/start) service).

Comment 6 Tomáš Kašpárek 2017-09-14 14:27:22 UTC
*** Bug 1491362 has been marked as a duplicate of this bug. ***

Comment 7 Baron Lenardson 2017-09-14 18:07:39 UTC
We ran into the same issue as well. I don't think it is related to SELinux but I think it is the %postun script added to osad-5.11.63-11.el7sat.noarch. The postun will restart the service which seems to kill any currently running actions started through osad.

Our current fix is to update osad via 'yum -y --setopt=tsflags=noscripts update osad' so the post uninstall script doesn't run. Not the best fix but it seems to prevent our machines from breaking on patching.

Comment 9 Pavel Studeník 2017-09-15 09:16:13 UTC
This issue is not related to SELinux. I checked behaviour on RHEL6 and it looks that it is not affected. Only RHEL 7.

osa-common-5.11.63-12.el6sat.noarch
osad-5.11.63-12.el6sat.noarch
rhn-check-1.0.0.1-43.el6.noarch

Comment 10 Pavel Studeník 2017-09-15 12:43:03 UTC
Same issue on Spacewalk nightly and RHEL7 (osad-5.11.91-1.el7.centos.noarch)

Comment 11 contact 2017-09-19 11:06:32 UTC
Hello,

same issue:

[Tue Sep 19 04:30:16 2017] up2date updateLoginInfo() login info
[Tue Sep 19 04:30:16 2017] up2date logging into up2date server
[Tue Sep 19 04:30:16 2017] up2date successfully retrieved authentication token from up2date server
[Tue Sep 19 04:36:28 2017] up2date 
Traceback (most recent call last):
  File "/usr/sbin/rhn_check", line 347, in __run_action
    Hit any actions that we want to always run.
  File "/usr/sbin/rhn_check", line 340, in __do_call
    status_report['uptime'][1] = -1
  File "/usr/share/rhn/actions/script.py", line 260, in run
    input_fds, output_fds, error_fds = select.select([pipe_read], [], [], select_wait)
  File "/usr/share/rhn/actions/script.py", line 58, in handle
    raise SystemExit(SYSEXIT_CODE)
<type 'exceptions.SystemExit'>: 3 



rpm -qa | grep osa
osa-common-5.11.63-11.el7sat.noarch
osad-5.11.63-12.el7sat.noarch
osa-common-5.11.63-12.el7sat.noarch
osad-5.11.63-11.el7sat.noarch

so i did this to install osad correctly -->

--> show duplicate:
rpm -qa --queryformat="%{NAME} %{ARCH}\n" | sort | uniq -c | grep -v " 1 " | cut -c 9- | cut -d" " -f1 

--> delete duplicate:
sh -c 'for file in `rpm -qa --queryformat="%{NAME} %{ARCH}\n" | sort | uniq -c | grep -v " 1 " | cut -c 9- | cut -d" " -f1`; do rpm -q --last $file | head -1 | cut -d" " -f1; done | grep -v kernel | grep -v gpg-pubkey | xargs rpm -e --justdb --nodeps'

--> Basic script to check and repair:

#!/bin/sh
# Update OSAD to prevent error
HOSTNAME=`hostname -s`
RH_RELEASE=`cat /etc/redhat-release | awk '{ print $7 }' | cut -d '.' -f 1`
duplicate=$(rpm -qa --queryformat="%{NAME} %{ARCH}\n" | sort | uniq -c | grep -v " 1 " | cut -c 9- | cut -d" " -f1 | wc -l)
i="0"
# twice
while [ $i -lt 2 ]
do
	if [ ${duplicate} -gt 2 ]; then
		sh -c 'for file in `rpm -qa --queryformat="%{NAME} %{ARCH}\n" | sort | uniq -c | grep -v " 1 " | cut -c 9- | cut -d" " -f1`; do rpm -q --last $file | head -1 | cut -d" " -f1; done | grep -v kernel | grep -v gpg-pubkey | xargs rpm -e --justdb --nodeps' &>/dev/null
	fi
	i=$[$i+1]
done
# don't touch to kernel* and gpg-pubkey
lastdup=$(rpm -qa --queryformat="%{NAME} %{ARCH}\n" | sort | uniq -c | grep -v " 1 " | cut -c 9- | cut -d" " -f1 | tr '\n' ' ')

# update OSAD + OSAD-common
yum -y --setopt=tsflags=noscripts update osad --disablerepo=* --enablerepo=your_RHNS_repo* &>/dev/null
RC=$?
echo "${HOSTNAME};${RH_RELEASE};${RC};${lastdup}"


If this can help.
- pco -

Comment 12 Thomas Mueller 2017-09-20 10:29:14 UTC
systemd kills by default the complete control-group - including the rhn_check started by osad. 

an update of osad restarts the daemon - rhn_check executes the yum transaction and leaves a mess behind (like no initramfs for a new kernel) when aborted forcfully by the osad daemon restart

Changing to KillMode=process solved the issue for me. 

https://github.com/spacewalkproject/spacewalk/pull/573

Comment 13 Tomas Lestach 2017-09-20 15:26:22 UTC
Switching to MODIFIED, as there's a patch in upstream ...

spacewalk.git: 68d859da592be2dd4d77ea0409ae71084fb5d290

Comment 14 Tomas Lestach 2017-09-20 15:30:38 UTC
Actually, I let TomasK confirm, no more changes are required. Switching back to ASSIGNED.

Comment 15 Thomas Mueller 2017-09-20 16:05:01 UTC
i suspect the very same problem can exist with rhnsd executing rhn_check updating rhnsd which would reststart rhnsd. but IMHO its a sysv service running with systemd and i dont know what killmode is used by default in that case.

Comment 16 Tomáš Kašpárek 2017-09-21 09:00:02 UTC
(In reply to Thomas Mueller from comment #15)
> i suspect the very same problem can exist with rhnsd executing rhn_check
> updating rhnsd which would reststart rhnsd. but IMHO its a sysv service
> running with systemd and i dont know what killmode is used by default in
> that case.

According to our testing there are no problems (and according to code which implies as you've stated it's sysv service running with systemd there should be no problems).
On the other hand on Fedora clients rhnsd is run as systemd service, which will in my opinion result in such kind of issues with rhnsd (haven't actually tested yet).

Comment 17 Tomas Lestach 2017-09-22 11:54:29 UTC
*** Bug 1494251 has been marked as a duplicate of this bug. ***

Comment 19 Stephan Dühr 2017-09-27 11:12:37 UTC
As of upstream patch https://github.com/spacewalkproject/spacewalk/pull/573/commits/aff0f202eca1cd8ecd2d1ad2e923d0e94a8bfff1 the systemd unit osad.service will probably get
KillMode=process

So I think, it's a good workaround meanwhile to add

/etc/systemd/system/osad.service.d/50-killmode.conf

with content

[Service]
KillMode=process

and run

systemctl deamon-reload

Like that, I was able to successfully update a system via Satellite that had osad-5.11.63-11.el7sat.noarch installed, which got updated to osad-5.11.63-12.el7sat.noarch and also included a new kernel.

This worked fine, dracut has been run to create initramfs and no duplicate packages left behind.

The only thing to notice is that osad does not terminate on sigterm (obviously because osad.py has a signal handler that waits for rhn_check to finish, but that won't happen because rhn_check is running yum which waits for systemctl try-restart osad to finish). So it hits DefaultTimeoutStopSec (90s):

...
Sep 27 11:25:56 test01 yum[30609]: Updated: samba-common-4.6.2-11.el7_4.noarch
Sep 27 11:25:56 test01 yum[30609]: Updated: libwbclient-4.6.2-11.el7_4.x86_64
Sep 27 11:25:56 test01 yum[30609]: Updated: samba-client-libs-4.6.2-11.el7_4.x86_64
Sep 27 11:25:57 test01 yum[30609]: Updated: kernel-tools-libs-3.10.0-693.2.2.el7.x86_64
Sep 27 11:25:57 test01 yum[30609]: Updated: osa-common-5.11.63-12.el7sat.noarch
Sep 27 11:25:57 test01 yum[30609]: Updated: osad-5.11.63-12.el7sat.noarch
Sep 27 11:25:57 test01 yum[30609]: Updated: kernel-tools-3.10.0-693.2.2.el7.x86_64
Sep 27 11:25:57 test01 yum[30609]: Updated: libsmbclient-4.6.2-11.el7_4.x86_64
Sep 27 11:25:57 test01 yum[30609]: Updated: 1:emacs-filesystem-24.3-20.el7_4.noarch
Sep 27 11:26:01 test01 yum[30609]: Installed: kernel-3.10.0-693.2.2.el7.x86_64
Sep 27 11:26:01 test01 yum[30609]: Updated: augeas-libs-1.4.0-2.el7_4.1.x86_64
Sep 27 11:26:01 test01 systemd: Reloading.
Sep 27 11:26:01 test01 systemd: Stopping OSAD daemon...
Sep 27 11:27:31 test01 systemd: osad.service stop-sigterm timed out. Killing.
Sep 27 11:27:31 test01 systemd: osad.service: main process exited, code=killed, status=9/KILL
Sep 27 11:27:31 test01 systemd: Unit osad.service entered failed state.
Sep 27 11:27:31 test01 systemd: osad.service failed.
Sep 27 11:27:31 test01 systemd: Starting OSAD daemon...
Sep 27 11:27:31 test01 systemd: PID file /var/run/osad.pid not readable (yet?) after start.
Sep 27 11:27:31 test01 systemd: Started OSAD daemon.
Sep 27 11:27:39 test01 dracut: dracut-
Sep 27 11:27:39 test01 dracut: Executing: /sbin/dracut -f /boot/initramfs-3.10.0-693.2.2.el7.x86_64.img 3.10.0-693.2.2.el7.x86_64
...

So adding something like
TimeoutStopSec=30s
could be added to reduce useless waiting time.

Comment 20 Tomáš Kašpárek 2017-09-27 11:22:41 UTC
(In reply to Stephan Dühr from comment #19)
> As of upstream patch
> https://github.com/spacewalkproject/spacewalk/pull/573/commits/
> aff0f202eca1cd8ecd2d1ad2e923d0e94a8bfff1 the systemd unit osad.service will
> probably get
> KillMode=process

The issue with this patch is that it re-introduces https://bugzilla.redhat.com/show_bug.cgi?id=1260527

Comment 21 Stephan Dühr 2017-09-27 12:41:03 UTC
(In reply to Tomáš Kašpárek from comment #20)
> (In reply to Stephan Dühr from comment #19)
> > As of upstream patch
> > https://github.com/spacewalkproject/spacewalk/pull/573/commits/
> > aff0f202eca1cd8ecd2d1ad2e923d0e94a8bfff1 the systemd unit osad.service will
> > probably get
> > KillMode=process
> 
> The issue with this patch is that it re-introduces
> https://bugzilla.redhat.com/show_bug.cgi?id=1260527

I can only see https://bugzilla.redhat.com/show_bug.cgi?id=1409562,
yes, that's bad, indeed.

Comment 22 Tomáš Kašpárek 2017-10-05 14:00:58 UTC
spacewalk.git(master): 96697c3f887e2010109a562863cc81ce40f86004

Reboot loop depends on fixes made in: BZ#1494389

Comment 28 Pavel Studeník 2018-01-08 16:40:15 UTC
Verified with packages:

osa-common-5.11.63-14.el7sat.noarch.rpm
osad-5.11.63-14.el7sat.noarch.rpm

Client execution returned "Update Succeeded" (code 0)

Comment 30 Pavel Studeník 2018-01-11 12:13:20 UTC
Some testing scenarios on RHEL 7.5

1) upgrade to newer version of osad - higher than osad-5.11.63-14-sat
 - Done
2) remove osad by osad
 - Done
3) downgrade (remote command) to older version of osad - must be higher than osad-5.11.63-14-sat
 - Done
4) remote script upgrade/remove 
 - Done

Comment 33 errata-xmlrpc 2018-01-28 14:45:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0137

Comment 34 BugMasta 2018-08-08 08:29:46 UTC
PROBLEM IS BACK WHEN UPGRADING TO osa-common-5.11.89-1.el7.noarch

# package-cleanup --dupes
Loaded plugins: fastestmirror, rhnplugin, susemanagerplugin, yumnotify
This system is receiving updates from RHN Classic or Red Hat Satellite.
rhnsd-5.0.30-1.el7.x86_64
rhnsd-5.0.25-1.el7.x86_64
yum-rhn-plugin-2.6.4-1.el7.noarch
yum-rhn-plugin-2.7.7-1.el7.noarch
rhnlib-2.7.2.2-24.1.noarch
rhnlib-2.7.5-1.el7.noarch
osad-5.11.89-1.el7.noarch
osad-5.11.74-1.el7.noarch
rhn-check-2.6.8-1.el7.noarch
rhn-check-2.7.16-1.el7.noarch
rhn-client-tools-2.6.8-1.el7.noarch
rhn-client-tools-2.7.16-1.el7.noarch
osa-common-5.11.74-1.el7.noarch
osa-common-5.11.89-1.el7.noarch
rhn-setup-2.7.16-1.el7.noarch
rhn-setup-2.6.8-1.el7.noarch
[root@audccfots809 ~]# cat /etc/redhat-release 
CentOS Linux release 7.5.1804 (Core)

Comment 35 BugMasta 2018-08-08 08:35:06 UTC
I've just added repo "Spacewalk Client 2.7 for CentOS7" to a bunch of my machines, and pushed out these updates. So now I've got a bunch of cleaning up to do.

Thanks a lot.

Comment 36 BugMasta 2018-08-08 09:22:14 UTC
Lucky i didn't push it out to our Red Hat prod vms, or I'd really be spewing.

Comment 37 Tomas Lestach 2018-08-08 09:58:04 UTC
Issue described in this BZ has been fixed and shipped and cannot be changed/undone any more. If you see any Satellite 5 issues, please open a case with Red Hat Support.

Just note Spacewalk Client packages are not being tested and supported to work with Satellite 5.
Note: both osad-5.11.89-1.el7.noarch and osad-5.11.74-1.el7.noarch are both Spacewalk packages.


Note You need to log in before you can comment on or make changes to this bug.