Bug 1215671

Created attachment 1019321 [details]
Helper script I'm using to clone channel or do some cleanup

Setup:

$ ./bz1205306.py <user> <pass> <fqdn> parent-twxqq $( seq 1000010000 1000010099 | tr "\n" ',' | sed 's/,$//' )

Cleanup:

$ ./bz1205306.py <user> <pass> <fqdn> parent-twxqq $( seq 1000010000 1000010099 | tr "\n" ',' | sed 's/,$//' ) --cleanup-only

I think that there are two root causes for this bad behavior.

One is that there are two ways of doing asynchronous errata cache updates: one is through taskomatic (normal), and the other is the way that the "Publish Erratum" process happens to use which is to schedule a java MessageQueue event inside of tomcat to be picked up later. The Errata Queue taskomatic task, which is what schedules auto-errata-updates, can only see things that are actually written to the database (like the queued errata-cache taskomatic tasks), and cannot see things that only exist in tomcat's MessageQueue. So when Errata Queue goes to schedule the auto-errata-update actions, it doesn't realize that there are errata cache changes pending and doesn't know it has to wait for them.

Problem 2 is that the Errata Queue task only currently waits for "update errata cache for entire channel" type Errata Cache actions. There are two other types of Errata Cache updates possible, "update errata cache for this server" and "update errata cache for this erratum". Publishing an erratum into a channel triggers the third type. So even we fixed problem #1 so that the pending task information is actually written to the database and visible across processes, Errata Queue still needs to be updated to pay attention to all types of Errata Cache updates.

I will attempt to make this process work properly in every case. Not scheduling auto-errata updates is a serious problem.

I have decided to fix this problem by moving the auto-errata update scheduling out into its own taskomatic task that runs by default once every 10 minutes. This way we are no longer hindered by the run-once nature of Errata Queue that requires all the proper ducks to be lined up for the auto-errata scheduling to work correctly.

As its own Auto Errata task, it will just schedule auto errata updates as they become available. So whenever the Errata Cache taskomatic task or the java MessageQueue errata cache updates get around to finishing, the next time the Auto Errata taskomatic task runs it will schedule the auto errata update actions.

The logic that drives the new Auto Errata task is, schedule update actions for:
 * servers that have auto errata updates turned on
 * servers that have errata updates that apply to them (as specified by the errata cache, the thing that drives "X errata available" type information in the web ui)
 * don't schedule yet if the channel's yum metadata is still regenerating (as yum would not be able to find the rpms yet if it tried to update)
 * don't schedule if we have ever had another errata update action for this errata / server combination

Note the last bullet, which is equivalent to current behavior and necessary so that we don't get in schedule-update, fail-for-some-reason, schedule-update-again loops. If an update fails, the admin will need to fix whatever was wrong and then manually schedule the update again, it will only ever automatically be scheduled once.

Committing to Spacewalk master:
d4c2840997c8fb3f8fdc670a2f36c53bf7eec9ae
8fa6aa316ec91877394cc3e0278536a238b24751

Spacewalk builds:
spacewalk-java-2.4.9-1
spacewalk-schema-2.4.5-1

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-1040.html