[snip a lot discussion where we were trying to find the problem]

--- Additional comment from Stephen Herr on 2014-01-20 16:43:41 EST ---

Okay, I believe I have managed to reproduce and debug the problem.

The problem, in short, is as neha says in comment 29, the cloned version of the erratum does not contain all the packages it should. When spacewalk-clone-by-date clones errata into a cloned channel, the existing cloned erratum is re-used (normal) and linked with the new channel, and all the packages in the cloned erratum that are also delivered into the original channel through the original erratum are added to the cloned channel. (This is normal, you don't want for example s390 packages that are associated with the erratum getting delivered into an x86_64 channel. Looking up what the original erratum delivered is the only way to know if you should put the i386 packages in an x86_64 channel, etc.)

The problem comes in because the cloned erratum already exists, and is missing the packages of the architecture we care about. So the erratum gets linked with the channel, but no packages get added.

It is possible for a customer (intentionally or unintentionally) to have gotten themselves in this situation by removing packages from a cloned erratum (either through the webui or api). If that is the only way that this data corruption is occurring, then there actually is no code bug here. I also have a couple of ideas on how this might possibly be happening through the code, in which case there would be a bug we'd need to fix.

Either way, we will need to fix the currently broken data for these customers, and then their clone-by-date should function as expected. Tomorrow I will see if I can find a path through the existing code that would explain this behavior (beyond customers removing packages from errata).

--- Additional comment from Stephen Herr on 2014-01-21 08:54:50 EST ---

I have confirmed that there is a way to get into this situation through normal Satellite usage. The problem path looks like:

1) Satellite-sync a channel that contains an erratum in a given architecture, say s390. This erratum will contain the s390 packages only at this point since that's all the Satellite knows about.
2) Create a clone of that channel. This creates a clone of the erratum that contains the s390 packages.
3) Satellite-sync a different architecture of the same channel, say x86_64. The Red Hat Erratum is updated to also contain x86_64 packages, but the cloned erratum is not.
4) Create a clone of the x86_64 channel. The cloned erratum will get associated with the x86_64 cloned channel, but since the cloned erratum contains no x86_64 packages there will be no packages added to the channel.

It turns out that this has been a problem in the past, and we have a webui page specifically dedicated to fixing problems of this type. The Channels -> Manage Software Channels -> <channel> -> Errata -> Sync page exists so that you can easily re-synchronize the packages in cloned errata contained in that channel with their originals. You can have customers run that and it should immediately fix their problems.