Description of problem: Using the Satellite UI to push osad-5.11.63-12.el7sat.noarch and/or osa-common-5.11.63-12.el7sat.noarch as updates to client will fail. The yum transaction shows "aborted" and the system is left with duplicates. - If just osad was pushed, the duplicates have only been osad and osa-common. - If osad is included in a large update, the entire update fails leaving many duplicates behind. Version-Release number of selected component (if applicable): osad-5.11.63-12.el7sat.noarch osa-common-5.11.63-12.el7sat.noarch How reproducible: Everytime Steps to Reproduce: *Selinux is enforcing *RHEL 7 client registered to Satellite 5 1. In the satellite UI, target the client system 2. Push updates for osad and osa-common to the client 3. On the client: Actual results: The update is marked as aborted in yum history. We also now see duplicates # yum history info last # rpm -qa | grep osa osa-common-5.11.63-12.el7sat.noarch osad-5.11.63-12.el7sat.noarch osa-common-5.11.63-11.el7sat.noarch osad-5.11.63-11.el7sat.noarch Expected results: Update succeeds without interrupting yum. Additional info: - When recreating this, if selinux was permissive, the update would be fine - The client can run "yum update osad" without issue. This does not break - Running rhn_check to find the job also results with this same issue.
This sounds very similar to Bug 1491362. Can anyone confirm Bug 1491362 and Bug 1491451 are duplicates?
Hi. I am trying investigate more about this issue. It is only problem of update latest version osad and osa-common. Old packages are not removed and stay as duplicated. If update is executed by rhn_check/yum, then it finishes correctly. Problem causes only osad which updates oneself. Probably reason of the issue can be sending signal during updating (restart service). >> getenforce Permissive .. update osad/osa-common >> yum history ... Warning: RPMDB altered outside of yum. ** Found 4 pre-existing rpmdb problem(s), 'yum check' output follows: osa-common-5.11.63-11.el7sat.noarch has installed conflicts osad > ('0', '5.11.63', '11.el7sat'): osad-5.11.63-12.el7sat.noarch osa-common-5.11.63-12.el7sat.noarch has installed conflicts osad < ('0', '5.11.63', '12.el7sat'): osad-5.11.63-11.el7sat.noarch osa-common-5.11.63-12.el7sat.noarch is a duplicate with osa-common-5.11.63-11.el7sat.noarch osad-5.11.63-12.el7sat.noarch is a duplicate with osad-5.11.63-11.el7sat.noarch history list >> yum history info .. Loaded plugins: product-id, rhnplugin, search-disabled-repos, subscription-manager This system is receiving updates from RHN Classic or Red Hat Satellite. Transaction ID : 19 Begin time : Thu Sep 14 11:50:01 2017 Begin rpmdb : 598:1dabd320ee0c74f9034aa3c3f3bdc450948486e0 User : System <unset> Return-Code : ** Aborted ** Transaction performed with: Installed rpm-4.11.3-21.el7.x86_64 @beaker-Client/7.3 Installed yum-3.4.3-150.el7.noarch @beaker-Client/7.3 Installed yum-metadata-parser-1.1.4-10.el7.x86_64 @beaker-Client/7.3 Installed yum-rhn-plugin-2.0.1-6.el7.noarch @beaker-Client/7.3 Packages Altered: ** Updated osa-common-5.11.63-11.el7sat.noarch @rhn-tools-rhel-x86_64-client-7 ** Update 5.11.63-12.el7sat.noarch @rhn-tools-rhel-x86_64-client-7 ** Updated osad-5.11.63-11.el7sat.noarch @rhn-tools-rhel-x86_64-client-7 ** Update 5.11.63-12.el7sat.noarch @rhn-tools-rhel-x86_64-client-7 history info # rpm -qa | grep osa osa-common-5.11.63-11.el7sat.noarch osad-5.11.63-11.el7sat.noarch osa-common-5.11.63-12.el7sat.noarch osad-5.11.63-12.el7sat.noarch In audit I found 3 AVC messages (needed disable dontaudit): >> semodule -DB >> tail /var/log/audit/audit.log ... type=AVC msg=audit(1505383812.993:127): avc: denied { rlimitinh } for pid=13173 comm="rhn_check" scontext=system_u:system_r:osad_t:s0 tcontext=system_u:system_r:rpm_t:s0 tclass=process type=AVC msg=audit(1505383812.993:127): avc: denied { siginh } for pid=13173 comm="rhn_check" scontext=system_u:system_r:osad_t:s0 tcontext=system_u:system_r:rpm_t:s0 tclass=process type=AVC msg=audit(1505383812.993:127): avc: denied { noatsecure } for pid=13173 comm="rhn_check" scontext=system_u:system_r:osad_t:s0 tcontext=system_u:system_r:rpm_t:s0 tclass=process But it looks that it is not problem with selinux, because selinux is in permissive mode.
Try to call remote command to update osad/osa-common "yum update -y osad osa-common" by webui and osad. It causes same problem. >> tail /var/log/up2date [Thu Sep 14 14:07:52 2017] up2date successfully retrieved authentication token from up2date server [Thu Sep 14 14:07:58 2017] up2date Traceback (most recent call last): File "/usr/sbin/rhn_check", line 347, in __run_action (status, message, data) = CheckCli.__do_call(method, params, kwargs) File "/usr/sbin/rhn_check", line 340, in __do_call retval = method(*params, **kwargs) File "/usr/share/rhn/actions/script.py", line 260, in run input_fds, output_fds, error_fds = select.select([pipe_read], [], [], select_wait) File "/usr/share/rhn/actions/script.py", line 58, in handle raise SystemExit(SYSEXIT_CODE) <type 'exceptions.SystemExit'>: 3 [Thu Sep 14 14:07:59 2017] up2date Updating package profile >> rpm -qa | grep osa osad-5.11.63-12.el7sat.noarch osa-common-5.11.63-11.el7sat.noarch osa-common-5.11.63-12.el7sat.noarch osad-5.11.63-11.el7sat.noarch
So.. next information. Problem is that you try to remove package osad by osad. Reproducer is unistnallation of package osad and this task is picked up by osad. ... Yum version: 3.4.3 Transaction ID : 51 Begin time : Thu Sep 14 14:34:55 2017 Begin rpmdb : 607:f2a1d3715a2b98d6b5c022537d46f6c5a9ed47ec User : System <unset> Return-Code : ** Aborted ** Transaction performed with: rpmdb time: 0.000 Installed rpm-4.11.3-21.el7.x86_64 @beaker-Client/7.3 Installed yum-3.4.3-150.el7.noarch @beaker-Client/7.3 Installed yum-metadata-parser-1.1.4-10.el7.x86_64 @beaker-Client/7.3 Installed yum-rhn-plugin-2.0.1-6.el7.noarch @beaker-Client/7.3 Packages Altered: ** Erase osad-5.11.63-12.el7sat.noarch installed history info The post install phase isn't finished correct. Update is same issue - yum installs new package then uninstalls old (and restart (stop/start) service).
*** Bug 1491362 has been marked as a duplicate of this bug. ***
We ran into the same issue as well. I don't think it is related to SELinux but I think it is the %postun script added to osad-5.11.63-11.el7sat.noarch. The postun will restart the service which seems to kill any currently running actions started through osad. Our current fix is to update osad via 'yum -y --setopt=tsflags=noscripts update osad' so the post uninstall script doesn't run. Not the best fix but it seems to prevent our machines from breaking on patching.
This issue is not related to SELinux. I checked behaviour on RHEL6 and it looks that it is not affected. Only RHEL 7. osa-common-5.11.63-12.el6sat.noarch osad-5.11.63-12.el6sat.noarch rhn-check-1.0.0.1-43.el6.noarch
Same issue on Spacewalk nightly and RHEL7 (osad-5.11.91-1.el7.centos.noarch)
Hello, same issue: [Tue Sep 19 04:30:16 2017] up2date updateLoginInfo() login info [Tue Sep 19 04:30:16 2017] up2date logging into up2date server [Tue Sep 19 04:30:16 2017] up2date successfully retrieved authentication token from up2date server [Tue Sep 19 04:36:28 2017] up2date Traceback (most recent call last): File "/usr/sbin/rhn_check", line 347, in __run_action Hit any actions that we want to always run. File "/usr/sbin/rhn_check", line 340, in __do_call status_report['uptime'][1] = -1 File "/usr/share/rhn/actions/script.py", line 260, in run input_fds, output_fds, error_fds = select.select([pipe_read], [], [], select_wait) File "/usr/share/rhn/actions/script.py", line 58, in handle raise SystemExit(SYSEXIT_CODE) <type 'exceptions.SystemExit'>: 3 rpm -qa | grep osa osa-common-5.11.63-11.el7sat.noarch osad-5.11.63-12.el7sat.noarch osa-common-5.11.63-12.el7sat.noarch osad-5.11.63-11.el7sat.noarch so i did this to install osad correctly --> --> show duplicate: rpm -qa --queryformat="%{NAME} %{ARCH}\n" | sort | uniq -c | grep -v " 1 " | cut -c 9- | cut -d" " -f1 --> delete duplicate: sh -c 'for file in `rpm -qa --queryformat="%{NAME} %{ARCH}\n" | sort | uniq -c | grep -v " 1 " | cut -c 9- | cut -d" " -f1`; do rpm -q --last $file | head -1 | cut -d" " -f1; done | grep -v kernel | grep -v gpg-pubkey | xargs rpm -e --justdb --nodeps' --> Basic script to check and repair: #!/bin/sh # Update OSAD to prevent error HOSTNAME=`hostname -s` RH_RELEASE=`cat /etc/redhat-release | awk '{ print $7 }' | cut -d '.' -f 1` duplicate=$(rpm -qa --queryformat="%{NAME} %{ARCH}\n" | sort | uniq -c | grep -v " 1 " | cut -c 9- | cut -d" " -f1 | wc -l) i="0" # twice while [ $i -lt 2 ] do if [ ${duplicate} -gt 2 ]; then sh -c 'for file in `rpm -qa --queryformat="%{NAME} %{ARCH}\n" | sort | uniq -c | grep -v " 1 " | cut -c 9- | cut -d" " -f1`; do rpm -q --last $file | head -1 | cut -d" " -f1; done | grep -v kernel | grep -v gpg-pubkey | xargs rpm -e --justdb --nodeps' &>/dev/null fi i=$[$i+1] done # don't touch to kernel* and gpg-pubkey lastdup=$(rpm -qa --queryformat="%{NAME} %{ARCH}\n" | sort | uniq -c | grep -v " 1 " | cut -c 9- | cut -d" " -f1 | tr '\n' ' ') # update OSAD + OSAD-common yum -y --setopt=tsflags=noscripts update osad --disablerepo=* --enablerepo=your_RHNS_repo* &>/dev/null RC=$? echo "${HOSTNAME};${RH_RELEASE};${RC};${lastdup}" If this can help. - pco -
systemd kills by default the complete control-group - including the rhn_check started by osad. an update of osad restarts the daemon - rhn_check executes the yum transaction and leaves a mess behind (like no initramfs for a new kernel) when aborted forcfully by the osad daemon restart Changing to KillMode=process solved the issue for me. https://github.com/spacewalkproject/spacewalk/pull/573
Switching to MODIFIED, as there's a patch in upstream ... spacewalk.git: 68d859da592be2dd4d77ea0409ae71084fb5d290
Actually, I let TomasK confirm, no more changes are required. Switching back to ASSIGNED.
i suspect the very same problem can exist with rhnsd executing rhn_check updating rhnsd which would reststart rhnsd. but IMHO its a sysv service running with systemd and i dont know what killmode is used by default in that case.
(In reply to Thomas Mueller from comment #15) > i suspect the very same problem can exist with rhnsd executing rhn_check > updating rhnsd which would reststart rhnsd. but IMHO its a sysv service > running with systemd and i dont know what killmode is used by default in > that case. According to our testing there are no problems (and according to code which implies as you've stated it's sysv service running with systemd there should be no problems). On the other hand on Fedora clients rhnsd is run as systemd service, which will in my opinion result in such kind of issues with rhnsd (haven't actually tested yet).
*** Bug 1494251 has been marked as a duplicate of this bug. ***
As of upstream patch https://github.com/spacewalkproject/spacewalk/pull/573/commits/aff0f202eca1cd8ecd2d1ad2e923d0e94a8bfff1 the systemd unit osad.service will probably get KillMode=process So I think, it's a good workaround meanwhile to add /etc/systemd/system/osad.service.d/50-killmode.conf with content [Service] KillMode=process and run systemctl deamon-reload Like that, I was able to successfully update a system via Satellite that had osad-5.11.63-11.el7sat.noarch installed, which got updated to osad-5.11.63-12.el7sat.noarch and also included a new kernel. This worked fine, dracut has been run to create initramfs and no duplicate packages left behind. The only thing to notice is that osad does not terminate on sigterm (obviously because osad.py has a signal handler that waits for rhn_check to finish, but that won't happen because rhn_check is running yum which waits for systemctl try-restart osad to finish). So it hits DefaultTimeoutStopSec (90s): ... Sep 27 11:25:56 test01 yum[30609]: Updated: samba-common-4.6.2-11.el7_4.noarch Sep 27 11:25:56 test01 yum[30609]: Updated: libwbclient-4.6.2-11.el7_4.x86_64 Sep 27 11:25:56 test01 yum[30609]: Updated: samba-client-libs-4.6.2-11.el7_4.x86_64 Sep 27 11:25:57 test01 yum[30609]: Updated: kernel-tools-libs-3.10.0-693.2.2.el7.x86_64 Sep 27 11:25:57 test01 yum[30609]: Updated: osa-common-5.11.63-12.el7sat.noarch Sep 27 11:25:57 test01 yum[30609]: Updated: osad-5.11.63-12.el7sat.noarch Sep 27 11:25:57 test01 yum[30609]: Updated: kernel-tools-3.10.0-693.2.2.el7.x86_64 Sep 27 11:25:57 test01 yum[30609]: Updated: libsmbclient-4.6.2-11.el7_4.x86_64 Sep 27 11:25:57 test01 yum[30609]: Updated: 1:emacs-filesystem-24.3-20.el7_4.noarch Sep 27 11:26:01 test01 yum[30609]: Installed: kernel-3.10.0-693.2.2.el7.x86_64 Sep 27 11:26:01 test01 yum[30609]: Updated: augeas-libs-1.4.0-2.el7_4.1.x86_64 Sep 27 11:26:01 test01 systemd: Reloading. Sep 27 11:26:01 test01 systemd: Stopping OSAD daemon... Sep 27 11:27:31 test01 systemd: osad.service stop-sigterm timed out. Killing. Sep 27 11:27:31 test01 systemd: osad.service: main process exited, code=killed, status=9/KILL Sep 27 11:27:31 test01 systemd: Unit osad.service entered failed state. Sep 27 11:27:31 test01 systemd: osad.service failed. Sep 27 11:27:31 test01 systemd: Starting OSAD daemon... Sep 27 11:27:31 test01 systemd: PID file /var/run/osad.pid not readable (yet?) after start. Sep 27 11:27:31 test01 systemd: Started OSAD daemon. Sep 27 11:27:39 test01 dracut: dracut- Sep 27 11:27:39 test01 dracut: Executing: /sbin/dracut -f /boot/initramfs-3.10.0-693.2.2.el7.x86_64.img 3.10.0-693.2.2.el7.x86_64 ... So adding something like TimeoutStopSec=30s could be added to reduce useless waiting time.
(In reply to Stephan Dühr from comment #19) > As of upstream patch > https://github.com/spacewalkproject/spacewalk/pull/573/commits/ > aff0f202eca1cd8ecd2d1ad2e923d0e94a8bfff1 the systemd unit osad.service will > probably get > KillMode=process The issue with this patch is that it re-introduces https://bugzilla.redhat.com/show_bug.cgi?id=1260527
(In reply to Tomáš Kašpárek from comment #20) > (In reply to Stephan Dühr from comment #19) > > As of upstream patch > > https://github.com/spacewalkproject/spacewalk/pull/573/commits/ > > aff0f202eca1cd8ecd2d1ad2e923d0e94a8bfff1 the systemd unit osad.service will > > probably get > > KillMode=process > > The issue with this patch is that it re-introduces > https://bugzilla.redhat.com/show_bug.cgi?id=1260527 I can only see https://bugzilla.redhat.com/show_bug.cgi?id=1409562, yes, that's bad, indeed.
spacewalk.git(master): 96697c3f887e2010109a562863cc81ce40f86004 Reboot loop depends on fixes made in: BZ#1494389
Verified with packages: osa-common-5.11.63-14.el7sat.noarch.rpm osad-5.11.63-14.el7sat.noarch.rpm Client execution returned "Update Succeeded" (code 0)
Some testing scenarios on RHEL 7.5 1) upgrade to newer version of osad - higher than osad-5.11.63-14-sat - Done 2) remove osad by osad - Done 3) downgrade (remote command) to older version of osad - must be higher than osad-5.11.63-14-sat - Done 4) remote script upgrade/remove - Done
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:0137
PROBLEM IS BACK WHEN UPGRADING TO osa-common-5.11.89-1.el7.noarch # package-cleanup --dupes Loaded plugins: fastestmirror, rhnplugin, susemanagerplugin, yumnotify This system is receiving updates from RHN Classic or Red Hat Satellite. rhnsd-5.0.30-1.el7.x86_64 rhnsd-5.0.25-1.el7.x86_64 yum-rhn-plugin-2.6.4-1.el7.noarch yum-rhn-plugin-2.7.7-1.el7.noarch rhnlib-2.7.2.2-24.1.noarch rhnlib-2.7.5-1.el7.noarch osad-5.11.89-1.el7.noarch osad-5.11.74-1.el7.noarch rhn-check-2.6.8-1.el7.noarch rhn-check-2.7.16-1.el7.noarch rhn-client-tools-2.6.8-1.el7.noarch rhn-client-tools-2.7.16-1.el7.noarch osa-common-5.11.74-1.el7.noarch osa-common-5.11.89-1.el7.noarch rhn-setup-2.7.16-1.el7.noarch rhn-setup-2.6.8-1.el7.noarch [root@audccfots809 ~]# cat /etc/redhat-release CentOS Linux release 7.5.1804 (Core)
I've just added repo "Spacewalk Client 2.7 for CentOS7" to a bunch of my machines, and pushed out these updates. So now I've got a bunch of cleaning up to do. Thanks a lot.
Lucky i didn't push it out to our Red Hat prod vms, or I'd really be spewing.
Issue described in this BZ has been fixed and shipped and cannot be changed/undone any more. If you see any Satellite 5 issues, please open a case with Red Hat Support. Just note Spacewalk Client packages are not being tested and supported to work with Satellite 5. Note: both osad-5.11.89-1.el7.noarch and osad-5.11.74-1.el7.noarch are both Spacewalk packages.