Bug 503090

Cliff and I understand why this is happening, but we think this case should be taken care of specially for the RHN client packages.  These packages should be ignored/removed from the package profile list when associated with a kickstart session.  These packages should be removed from the array of items to be used in the sync.

Am I correct in assuming that the kickstart completes fine, just the subsequent package sync generates this error?

Brandon could you clarify what you and Cliff understand about why this is happening? In a normal package sync we don't try to re-install packages that we already have installed as per the database. Is this is a case where (because the box was just kickstarted) we have no package data in the database yet and thus try to install things the system got during the kickstart anyhow? If so, why would this just be relevant to our client packages?

Several days of testing later I finally can reproduce this, RHEL 4 u8 system, create a profile, associate it to a u4 kickstart profile.

The action shows as failed in the event history, and to further aid in testing you can reschedule it right from here rather than update and re-kickstart the box.

rhnlib shows up as an "install" in the event history, whereas most packages are set to replace. This is incorrect as the package is already installed.

Current theory: among the list of rpms we attempt to magically wget and rpm -Uvh in kickstart post scripts is rhnlib. Because this happens here the Satellite has not yet had time to realize the system actually has rhnlib installed (as it's done externally), and when comparing the system's current packages to those in the profile we wish to sync with, we see rhnlib is missing and attempt to "install" it fresh, which up2date (or perhaps rpm) chokes on.

I do not know how best to solve this, we could just stop comparing these packages but this implies we'll have a disjoint list of magic packages in two places that need to be kept in sync. (one in java where we render kickstart files with the wget + rpm -Uvh commands, and now one in python where we compare the packages and schedule the updates)

Did some more investigation, it does appear that rhnlib is *not* installed by default in some kickstart trees, perhaps one of the reasons we do this wget + rpm -Uvh hackery. Once the ks script runs it does get installed but Satellite does not yet know it, and thus it looks like the above theory is what's actually taking place.

We decided to solve by excluding rhnlib from the package profile sync list, as we always install the latest version anyhow. 

spacewalk.git: ef545ff
satellite.git: 3e43732

Cancelling NEEDINFO.

While this fix does work, and the logic which allows us to do it is valid, there may be another issue at play. rhnlib does appear to be installed by default in the kickstart tree, according to install.log it is. During the kickstart the client makes an xmlrpc backend call for registration.add_packges and offers up a list of all it's installed packages. rhnlib is mysteriously missing from this list, whereas all other packages appear to be present.

IMO this issue warrants investigation in a future release, so will file a new bugzilla for this.

Filed bug #504416 to investigate the issue in the future.

fails_qa
profile-sync doesn't complete. no error message
from
ks-post.log
Traceback (most recent call last):
  File "/usr/sbin/rhn_check", line 345, in ?
    handle_action(action)
  File "/usr/sbin/rhn_check", line 218, in handle_action
    (status, message, data) = run_action(method, params)
  File "/usr/sbin/rhn_check", line 174, in run_action
    (status, message, data) = do_call(method, params)   
  File "/usr/sbin/rhn_check", line 91, in do_call
    retval = apply(method, params)
  File "/usr/share/rhn/actions/packages.py", line 428, in runTransaction
    (ts, added, removed) = up2date.genTransaction(tsd)
  File "/usr/share/rhn/up2date_client/up2date.py", line 589, in genTransaction
    resolveRemovalDeps(ts)
  File "/usr/share/rhn/up2date_client/up2date.py", line 1146, in resolveRemovalDeps
    raise up2dateErrors.RpmRemoveSkipListError, depName
up2date_client.up2dateErrors.RpmRemoveSkipListError: Could not remove package "kernel-utils-2.4-13.1.99". It was on the RemoveSkipList
Traceback (most recent call last):
  File "/usr/sbin/rhn_check", line 345, in ?
    handle_action(action)
  File "/usr/sbin/rhn_check", line 218, in handle_action
    (status, message, data) = run_action(method, params)
  File "/usr/sbin/rhn_check", line 174, in run_action
    (status, message, data) = do_call(method, params)   
  File "/usr/sbin/rhn_check", line 91, in do_call
    retval = apply(method, params)
  File "/usr/share/rhn/actions/packages.py", line 428, in runTransaction
    (ts, added, removed) = up2date.genTransaction(tsd)
  File "/usr/share/rhn/up2date_client/up2date.py", line 589, in genTransaction
    resolveRemovalDeps(ts)
  File "/usr/share/rhn/up2date_client/up2date.py", line 1146, in resolveRemovalDeps
    raise up2dateErrors.RpmRemoveSkipListError, depName
up2date_client.up2dateErrors.RpmRemoveSkipListError: Could not remove package "kernel-utils-2.4-13.1.99". It was on the RemoveSkipList

from /var/log/up2date
[Mon Jun 15 14:17:11 2009] up2date  kernel-utils-2.4-13.1.99 in the skipList, skipping from the transaction
[Mon Jun 15 14:17:11 2009] up2date Could not remove package "kernel-utils-2.4-13.1.99". It was on the RemoveSkipList

Looks like a completely new issue. Preethi could you post any specifics on how you caused this to happen, specifically the versions of RHEL 4 used to create the package profile, and then on the machine that was being synced during kickstart. Thanks.

1. Installed an RHEL4 U6 system and updated it to RHEL4 Update 8. Created a
system profile from this system .

2. In RHN 5.3 satellite, created a kickstart file. rhel4u5 

3. In the kickstart file -> Software -> Package Profiles -> Chosen the created
system profile [step 1] to which the system should be synced at the time of
installation.

4. Generated the activation key with Monitoring and Provisioning entitlements.
Also selected relevant Base Channel and Child Channels. 

5. In the ks file -> Activation keys -> Choose the activation key which as been
created in step 4.

6. Kickstart the system using the kickstart file created.

New issue filed as bug #506167, moving this back to modified but blocked by the new.

Other issue was closed as notabug, more details there but basically everything working as expected. (and profile syncing between rhel u releases is a bit of an edge case and probably not a great idea)

To test this, you should be able to verify with a more normal usage of profile sync, such as creating a RHEL 4u8 system, adding some extra packages to it (i.e. zsh), *then* creating the package profile.

Use the same RHEL 4u8 release for the kickstart, and zsh should be installed after kickstart.

Back to ON_QA.

fails_qa
rhel4u7 system failed to sync

Details:  	This action will be executed after 06/16/09 10:19:02 AM EDT.

This action's status is: Failed.
The client picked up this action on 06/16/09 10:19:02 AM EDT.
The client completed this action on 06/16/09 10:19:08 AM EDT.
Client execution returned "Failed: The transaction failed for the following reasons: [('package up2date-4.8.1-33.el4 is already installed', (2, '', 0L))]" (code 40)
Changes Scheduled:

    * Install up2date-4.8.1-33.el4
    * Install ypserv-2.13-19
    * Install zsh-4.2.0-4.EL.4.5
    * Install zsh-html-4.2.0-4.EL.4.5

This is a clear indication our bandaid attempt at a fix is not going to cut it. There is definitely a problem with kickstarts where the initial package list the server reports in ks-post is drastically incorrect. (investigating Preethi's Satellite it looks like initial profile contains 120 packages, whereas later it suddenly jumps up to > 300) 

Unfortunately this looks like something we cannot fix server side, need to understand and investigate why the client list being sent is incorrect. Moving back to triage.

I believe we have traced this down and it is indeed a client side problem.

During the kickstart the package list is updated twice, the first time appears to be during rhnreg_ks which calls the backend registration.add_packages. This list was coming through with 303 packages, and is missing rhnlib and up2date ONLY.

A few seconds later during (we believe) rhn_check, we update again with the registration.update_packages call, this list comes through with 305 packages and now includes rhnlib and up2date.

The problem appears to be related to this:

[root@localhost ~]# rpm -qa --qf "%{NAME} %{ARCH} %{COOKIE}\n" | grep none 
rhnlib noarch (none)
up2date i386 (none)
gpg-pubkey (none) (none)

rhnlib and up2date are the *only* two packages that have an arch, but no cookie.

The culprit appears to be client/rhel/rhn-client-tools/src/up2date_client/rpmUtils.py:

        if getArch:
            arch = h['arch']
            # the arch on gpg-pubkeys is "None"...
            if arch:
                pkg_list.append([name, version, release, epoch, arch])
        elif getInfo:
            arch = h['arch']
            cookie = h['cookie']
            if arch and cookie:
                pkg_list.append([name, version, release, epoch, arch, cookie])
            elif arch:
                pkg_list.append([name, version, release, epoch, arch])
        else:
            pkg_list.append([name, version, release, epoch])

In the second block if cookie is missing, the package is skipped.

This issue is *not* related to bug #506651. In our case the package snapshot is stored correctly, it's what we have associated with the server that's wrong.

Will attach a proposed patch, proceeding to attempt to reproduce on latest RHEL 4 so we can address the client side changes that need to be made.

Problem is even worse on RHEL 4 u8, as I believe the build system no longer populates these RPM cookies.

Reproducer:
- Kickstart fresh RHEL4u8 system.
- Install zsh on it.
- Create package profile for the system.
- Create kickstart profile, set up to sync to the package profile just created.

The initial add_packages call comes back with just 220 rpms. Shortly after the update_packages call has 307. rhnlib and up2date are missing in the first as well as many others:

[root@localhost ~]# rpm -qa --qf "%{NAME} %{ARCH} %{COOKIE}\n" | grep none | wc -l
88

Patch committed to spacewalk.git: 050793bb214e260ebff734275c2188d5fb529866

I did not port this to satellite.git at this time. Need input from Prad.

Also the first attempt to solve the problem by filtering on Satellite side has been reverted in both Spacewalk/Satellite.

Tested by (on the Satellite) setting debug=5 in rhn.conf, then running rhnreg_ks on the client again to re-register. Then examining rhn_server_xmlrpc.log and look for the new add_packages call. In this scenario that list is 220 packages each time, missing rhnlib, up2date, and a variety of other packages.

After applying the patch, 307 packages come through and this list includes rhnlib and up2date.

Prad, handing this over to you to get included in RHEL 4. The fix is in spacewalk master, but I understand this is not where our packages are actually built from. Thanks!

*** Bug 543942 has been marked as a duplicate of this bug. ***

    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Package synchronization no longer fails after a kickstart installation.

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0240.html