Created attachment 471286 [details]
screenshot of satellite event page showing success

Additional output to show that yum metadata caching is the cause of the packages not being installed:

[root@jrummy6-64 ~]# yum check-update
Loaded plugins: fs-snapshot, refresh-packagekit, rhnplugin
[root@jrummy6-64 ~]# yum clean all
Loaded plugins: fs-snapshot, refresh-packagekit, rhnplugin
Cleaning up Everything
[root@jrummy6-64 ~]# yum check-update
Loaded plugins: fs-snapshot, refresh-packagekit, rhnplugin
rhel-6-test                                                                                    |  871 B     00:00     
rhel-6-test/primary                                                                            | 4.6 kB     00:00     
rhel-6-test                                                                                                       8/8

git.x86_64                                            1.7.1-2.el6_0.1                                      rhel-6-test
initscripts.x86_64                                    9.03.17-1.el6_0.1                                    rhel-6-test

Here's the deal:

In the situation described in the initial comment, we're not able to
simply re-schedule the installation of the package (which is what
the described expected results suggest) for two reasons:

1. From the yum exception we receive in this case, we're not able to
tell whether we got the exception because the package really does not
exist in the yum (Satellite) repo or the package does exist on the
server and our metadata cache is just out of date.

yum libraries simply trust the local cache as long as it's not expired.
Nothing nice we can do about this (other than deleting the local cache
and downloading it from server again every time we run rhn_check, which
is not acceptable).

2. Even if the previous point would be feasible, we're simply not able
to automatically schedule an event as a result of another (failed) event.


There's one solution to the problem that I see though.

Every Satellite channel has a last_modified flag set (indeed the time
of last channel modification) which is being sent to the registered client.

What we can do on the client is:
1. Check the time stamp on the locally cached metadata (more specifically
the time stamp of a 'cachecookie' file)

2. Compare the time stamp from point 1 with the last_modified value of
the Satellite channel. If the Satellite channel is more recent than
the local cachecookie, we delete the cookie, which in turn will convince
yum libraries to refresh the cache from the server.

From this point on, the scheduled installation will work OK.

There's one major problem with this approach though -- the channel time stamp
which is being sent from Satellite to the client is in 'YYYYMMDDHH24MISS' 
format, i.e. local time of the Satellite machine and the time stamp comparison
simply wouldn't work if the client and the server would be in different
time zones.

So to accomplish the proposed plan, we'd have to
1. Modify yum-rhn-plugin according to the schema above
2. Modify Satellite to send one more flag with the channel -- channel's
last_modified info as a unix epoch.

Both fixes (server & client) shouldn't be too invasive.

About the part of the bug report, where rhn_check (and Satellite webui) say
the package install / update was successful, when in fact it wasn't:

When installing a new package (i.e. no older version on the client system),
the yum libraries try to find the requested package info in the local cache;
when not successful, the yum library throws an exception which is then
caught by yum-rhn-plugin / rhn_check and correctly propagated to the parent
Satellite as a failure.

When updating a package and the requested package does not exist in
the local cache, no exception is thrown and we simply execute an
empty transaction (empty, as in nothing changed). This looks as a success,
which is also reported as such to the parent Satellite.

The behavior above is how yum libraries are designed.

We are able to fix the problem with package update by checking the
length of YumAction's tsInfo before and after the operation takes
place. If the length is zero -> action failure.

Yes, this mixup looks to be a result of me generalizing the use case I was testing.  In my tests, I was installing an updated version of a package that was already installed on the system, but for some reason I chose to describe it as install/update.  My apologies for the confusion.  

Either way, it sounds like you have identified the issue and a potential solution, so the customer will be pleased.

That said, the customer did actually explain to me their wish to see yum behave the way you described in https://bugzilla.redhat.com/show_bug.cgi?id=666545#c3 (it checks if its metadata is out-of-date, and pulls down a new one if it is), so if that is no longer the intended action plan then I will likely be filing an RFE in a separate case to request this functionality.  

Thanks for your help.
-John

John - if you wish to report a bug / RFE for yum - please do so. This bug is in how the yum-rhn-plugin and/or rhn-client-tools interacts with yum when we are performing a scheduled event to update a package - as outlined in comment #5. We as maintainers of the RHN Client pieces can update/change our code as Milan suggests to avoid the confusion caused by incorrectly saying a package was updated when it was not and be done with this bug report. 

We could then go beyond that - most likely creating a new bug/RFE, as outlined in comment #3 and make changes to both yum-rhn-plugin and Satellite for cache expire checks. Are you wanting yum to change? OR let our code control this as proposed in comment #3. 

Cliff

solution in comment #3 would only work for only if customer had newest satellite. 
solution in comment #5 would work for anyone using newest client.

my prefernce for this bug report is #5, more isolated fix.

We can look at #3 - but would have to split it off. If we do it.

Thanks Cliff.  The solution proposed in comment #5 is fine for the purposes of resolving the issue reported in this bug.

I have gone ahead and sent up a separate RFE case to the GSS representatives for triage (case 402319, if you're interested) to request the functionality that the metadata be discarded if a new copy is available, regardless of whether it has expired.  If it is found to be agreeable to them, they will file a new bugzilla.  

As to your question about whether this should happen in yum or our code (as
proposed in comment #3), I'm not sure.  If I ask the customer, I'm sure they'll
say they'd like to see it happen in yum, so it would behave as they wish for
rhnplugin-based channels and standard repos alike (since they currently use
both).  However if a case could be made for only doing it in yum-rhn-plugin
(due to simplicity, or for other reasons), then I could probably get them to accept that as well.  I'll have to think it over.  For now, the RFE states that what's described in comment #3 is fine, but if that changes I'll note it in the RFE that gets filed.

Thanks for your help.

-John

Fix for the problem where we report empty transactions as a success:

spacewalk.git master: 15e1dca13350ae3e9a0eb8fbf52287db1d59ddd3
satellite.git CLIENT-RHEL-6: b0d0b34f8b2f6d3800c196c650c19d71b5e04ac7

This in fact is a change in yum-rhn-plugin -- changing the component.

Additional fixes:
satellite.git CLIENT-RHEL-6: 793b940f44faa81ef23e4df79d1eaea6c67d538e

Additional fixes:

spacewalk.git master: fd657459014a569811dd382bf009cc0db39143f0
satellite.git CLIENT-RHEL-6: 39a59794de2f1e1d6f83188766119581e97c678d

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0565.html

(In reply to comment #3)
> 
> There's one solution to the problem that I see though.
> 
> Every Satellite channel has a last_modified flag set (indeed the time
> of last channel modification) which is being sent to the registered client.
> 
> What we can do on the client is:
> 1. Check the time stamp on the locally cached metadata (more specifically
> the time stamp of a 'cachecookie' file)
> 
> 2. Compare the time stamp from point 1 with the last_modified value of
> the Satellite channel. If the Satellite channel is more recent than
> the local cachecookie, we delete the cookie, which in turn will convince
> yum libraries to refresh the cache from the server.
> 
> From this point on, the scheduled installation will work OK.
> 
> There's one major problem with this approach though -- the channel time stamp
> which is being sent from Satellite to the client is in 'YYYYMMDDHH24MISS' 
> format, i.e. local time of the Satellite machine and the time stamp comparison
> simply wouldn't work if the client and the server would be in different
> time zones.
> 
> So to accomplish the proposed plan, we'd have to
> 1. Modify yum-rhn-plugin according to the schema above
> 2. Modify Satellite to send one more flag with the channel -- channel's
> last_modified info as a unix epoch.
> 
> Both fixes (server & client) shouldn't be too invasive.

Just for the record here: the last_modified value is not guaranteed not to change if something else than package list changes. For example, see bug 743833. It would seem better to send the actual repodata timestamp with the channel -- if there is new repodata available, it makes sense to re-download them again. And yes, we'd need to make this UTC value to work correctly across timezones.