Dennis,
Could you take a look at this? 

Thanks,
Keqin

[Errno -1] Metadata file does not match checksum
Some analysis:
The productid (of a freshly installed os) got removed during first yum update from cdn-internal.rcm-test.redhat.com when there's a checksum error of productid.gz

e.g.
1. cat http://cdn-internal.rcm-test.redhat.com/content/dist/rhel/server/6/6Server/i386/os/repodata/repomd.xml
...skip...
<data type="productid">
<location href="repodata/productid.gz"/>
<checksum type="sha">f1f28a9c882159109b479df0737ae1cffc8d0785</checksum>
                          ^^^^
<timestamp>1315944951</timestamp>
<open-checksum type="sha">d21d7be9999c257a8e394fbc846702e8b7f79c9c</open-checksum>
</data>

2.
$ wget http://cdn-internal.rcm-test.redhat.com/content/dist/rhel/server/6/6Server/i386/os/repodata/productid.gz
$ sha1sum productid.gz
0b4f80d14a08a6f0e85a9d625efbdb91385dc845  productid.gz
 ^^^^
   mismatch

I've reproduces this and I think I see what is happening, and have a potential fix.

To quote from irc:

<alikins> jsefler: I think I see what is causing the productid cert disappearnce... the cert is getting installed associated with the name of the repo at install time (something like "anaconda-RedHatEnterpriseLinux-201109300151.i386") while yum is thinking it should be something like "rhel6-server-rpms"
<alikins> jsefler: and the anaconda repo isn't "active" post install
<jsefler> alikins, that makes sense based on what you said yesterday about the product_id plugins algorythm for removing product certs on rhel6
<alikins> jsefler: and I bet the anaconda repo name is sticking around in the cert->repo map file because of the metadata mismatch
<alikins> jsefler: anaconda repo is what's mapped in anaconda, it goes to write the proper cert->repo map, can't get the productid from the metadata because of the checksum. if it could, it would rewrite that map with the new enabled repo instead of the anaconda one
<alikins> but it keeps the anaconda one, on first product installed, then it thinks that 69.pem is the cert for the anaconda repo, which is no longer active
<alikins> and delete's it
<alikins> I'm not entirely sure how to fix though, since it kind of needs the info in that bogus metadata to do the map correctly. a few options: 1) don't break the metadata 2) don't ever delete certs
<alikins> 3) maybe ignore cert->repo maps written during installation, which kind of invalidates the point of the plugin working in anaconda
<alikins> guess a reasonable workaround is "if we get metadata exceptions, don't delete anything, we are confused"


Aka, anaconda creates the initial cert->repo mapping based on install media. Normally, if the metadata could be read, it would then replace that with the mapping from cert-> installed name of repo. 

the productid plugin will delete certs if they are not "active", aka, no packages from the repo's associated with that repo are installed. To do that,
it has to look up the repo associated with the repo. Because of the above mentioned incorrect cert->repo map created during anaconda, it thinks the repo associated with the cert (69.pem in this case) is associated with an inactive repo (anaconda created "anaconda-RedHatEnterpriseLinux-201109300151.i386" in this case), and deletes it. 

So this shouldn't happen if the metadata is correct. But to handle the case where it is wrong, I've changed the code to not delete anything if there are metadata loading issues.

Wouldn't 69.pem also get removed then if the user installs a package from a custom repo and doesn't have any of the rhel-* repos enabled?  If so, that seems like an issue.

commit 2ee535b52089e5b0443132c191a1831d916bb8e7
Author: Adrian Likins <alikins>
Date:   Thu Sep 29 16:36:59 2011 -0400

    740773: Do not delete certs if we have repo metadata errors
    
    products cert's installed by anaconda were getting deleted
    on first product install by yum (via product-id plugin)
    if we have repo metadata issues (in this case, a checksum
    mismatch for the product file).
    
    product-id plugin was not able to create the proper
    cert->repo mapping in productid.js because of the
    chksum errors. So the productid.js had the cert
    associated with a repo name that only exists in anaconda.
    This repo appeared "inactive", and then would be
    deleted.
    
    This fix prevents any cert deletions from happening
    if the metadata presents any errors.

Verified on subscription-manager-0.96.13-1.el6.i686, PASS.
Original Product cert was not removed after the fix and client got clear error prompt for metadata error.

# yum install tigervnc
 ...<skip>...
  Installing : tigervnc-1.0.90-0.15.20110314svn4359.el6_1.1.i686            1/1 
rhel-6-server-rpms/productid                             | 1.7 kB     00:00     
http://cdn-internal.rcm-test.redhat.com/content/dist/rhel/server/6/6Server/i386/os/repodata/productid.gz: [Errno -1] Metadata file does not match checksum
                                    ^^^^^^^^
Trying other mirror.
Installed products updated.

Installed:
  tigervnc.i686 0:1.0.90-0.15.20110314svn4359.el6_1.1                           

Complete!


#rhsm.log
2011-10-09 07:17:57,992 [ERROR]  @productid.py:220 - failure: repodata/productid.gz from rhel-6-server-rpms: [Errno 256] No more mirrors to try.
Traceback (most recent call last):
  File "/usr/share/rhsm/subscription_manager/productid.py", line 213, in getEnabled
    fn = repo.retrieveMD(self.PRODUCTID)
  File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1559, in retrieveMD
    return self._retrieveMD(mdtype)
  File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1615, in _retrieveMD
    size=thisdata.size)
  File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 842, in _getFile
    raise Errors.NoMoreMirrorsRepoError, errstr
NoMoreMirrorsRepoError: failure: repodata/productid.gz from rhel-6-server-rpms: [Errno 256] No more mirrors to try.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2011-1695.html