Created attachment 928056 [details] output from running pulp-manage-db with extra instrumentation Description of problem: While running pulp-manage-db as part of a 2.3.1 to 2.4 upgrade. I ran into multiple failures while it was running the step "Applying pulp_rpm.plugins.migrations version 16" Version-Release number of selected component (if applicable): 2.4 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: When running pulp-manage-db, it would fail with the error: "Applying migration pulp_rpm.plugins.migrations.0016_new_yum_distributor failed. See log for details." A paste of /var/log/messages of one of the failures in the version 16 runs is here: http://pastebin.com/xmQksf06 Barnaby Court ask me to output the repo.id of each of the repos as the script attempted the migration and then to output the full contents of erratum_unit.metadata when an exception in add_unit_metadata() occurred. The output of that is attached.
The odd thing is multiple runs of pulp-manage-db appeared to fix the problem. On the first few runs, the script would error out as mentioned above. Subsequent runs appeared to get farther (more repo.ids outputted before failure.) One later run, errored with: Aug 18 10:42:51 dvpuap02 pulp: pulp.server.db.manage:CRITICAL: Applying migration pulp_rpm.plugins.migrations.0016_new_yum_distribu short = pkglist.get('short') tor failed. Aug 18 10:42:51 dvpuap02 pulp: pulp.server.db.manage:CRITICAL: Link path [/var/lib/pulp/working/repos/rhel-5-x86_64-os/distributors/yum_distributor/repodata/productid.gz] exists, but is not a symbolic link Aug 18 10:42:51 dvpuap02 pulp: pulp.server.db.manage:CRITICAL: Traceback (most recent call last): Aug 18 10:42:51 dvpuap02 pulp: pulp.server.db.manage:CRITICAL: File "/usr/lib/python2.6/site-packages/pulp/server/db/manage.py", line 86, in migrate_database Aug 18 10:42:51 dvpuap02 pulp: pulp.server.db.manage:CRITICAL: update_current_version=not options.test) Aug 18 10:42:51 dvpuap02 pulp: pulp.server.db.manage:CRITICAL: File "/usr/lib/python2.6/site-packages/pulp/server/db/migrate/models.py", line 161, in apply_migration Aug 18 10:42:51 dvpuap02 pulp: pulp.server.db.manage:CRITICAL: migration.migrate() Aug 18 10:42:51 dvpuap02 pulp: pulp.server.db.manage:CRITICAL: File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/migrations/0016_new_yum_distributor.py", line 59, in migrate Aug 18 10:42:51 dvpuap02 pulp: pulp.server.db.manage:CRITICAL: _re_publish_repository(repo, d) Aug 18 10:42:51 dvpuap02 pulp: pulp.server.db.manage:CRITICAL: File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/migrations/0016_new_yum_distributor.py", line 165, in _re_publish_repository Aug 18 10:42:51 dvpuap02 pulp: pulp.server.db.manage:CRITICAL: publisher.publish() Aug 18 10:42:51 dvpuap02 pulp: pulp.server.db.manage:CRITICAL: File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 323, in publish Aug 18 10:42:51 dvpuap02 pulp: pulp.server.db.manage:CRITICAL: self.process_lifecycle() Aug 18 10:42:51 dvpuap02 pulp: pulp.server.db.manage:CRITICAL: File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 92, in process_lifecycle Aug 18 10:42:51 dvpuap02 pulp: pulp.server.db.manage:CRITICAL: step.process() Aug 18 10:42:51 dvpuap02 pulp: pulp.server.db.manage:CRITICAL: File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 150, in process Aug 18 10:42:51 dvpuap02 pulp: pulp.server.db.manage:CRITICAL: self._process_block() Aug 18 10:42:51 dvpuap02 pulp: pulp.server.db.manage:CRITICAL: File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 562, in _process_block Aug 18 10:42:51 dvpuap02 pulp: pulp.server.db.manage:CRITICAL: self.process_unit(package_unit) Aug 18 10:42:51 dvpuap02 pulp: pulp.server.db.manage:CRITICAL: File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/distributors/yum/publish.py", line 418, in process_unit Aug 18 10:42:51 dvpuap02 pulp: pulp.server.db.manage:CRITICAL: self._create_symlink(unit.storage_path, link_path) Aug 18 10:42:51 dvpuap02 pulp: pulp.server.db.manage:CRITICAL: File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 421, in _create_symlink Aug 18 10:42:51 dvpuap02 pulp: pulp.server.db.manage:CRITICAL: raise RuntimeError(msg % {'l': link_path}) Aug 18 10:42:51 dvpuap02 pulp: pulp.server.db.manage:CRITICAL: RuntimeError: Link path [/var/lib/pulp/working/repos/rhel-5-x86_64-os/distributors/yum_distributor/repodata/productid.gz] exists, but is not a symbolic link Aug 18 10:42:51 dvpuap02 pulp: pulp.server.db.manage:CRITICAL: Aug 18 10:42:51 dvpuap02 pulp: pulp.server.db.manage:INFO: Database migrations complete. But then the next run of pulp-manage-db finished successfully and I was able to start the pulp_workers, etc. and connect to pulp via the admin client on the same VM. In all, I probably had to run pulp-manage-db 6-8 times for it to get all the way through the process.
The first problem reported is because of None values stored for the package src attribute in the erratum unit or from attribute in the from section that was not being translated to an empty string for json serialization.
PR: https://github.com/pulp/pulp_rpm/pull/554
build: 2.5.0-0.6.beta
verified Upgraded from 2.3 ->2.5 with different types for content. pulp-manage-db ran without any errors