Bug 1016612

Summary:	Pulp needs to have support for uploading srpms
Product:	[Retired] Pulp	Reporter:	Laurent Bigonville <bigon>
Component:	rpm-support	Assignee:	Barnaby Court <bcourt>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Preethi Thomas <pthomas>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	2.2	CC:	bcourt, jbaird, kris.buytaert, mhrivnak, pthomas, skarmark
Target Milestone:	---	Keywords:	Triaged
Target Release:	2.4.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2014-08-09 06:57:13 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Laurent Bigonville 2013-10-08 12:10:12 UTC

Hi,

Description of problem:
Today I tried to update pulp to 2.2 from 2.1 and pulp-manage-db is failing with a backtrace

Version-Release number of selected component (if applicable):
2.2

How reproducible:
Always

Steps to Reproduce:
1. Update pulp RPM's to 2.2
2. run pulp-manage-db

Actual results:
It fails with the following backtrace:


# pulp-manage-db 
Beginning database migrations.
Migration package pulp.server.db.migrations is up to date at version 4
Migration package pulp_puppet.plugins.migrations is up to date at version 0
Applying pulp_rpm.migrations version 8
Migration to pulp_rpm.migrations version 8 complete.
Applying pulp_rpm.migrations version 9
Migration to pulp_rpm.migrations version 9 complete.
Applying pulp_rpm.migrations version 10
Migration to pulp_rpm.migrations version 10 complete.
Applying pulp_rpm.migrations version 11
Applying migration pulp_rpm.migrations.0011_new_importer failed.  See log for details.
2013-10-08 11:52:52,924 db:CRITICAL: Applying migration pulp_rpm.migrations.0011_new_importer failed.
2013-10-08 11:52:52,925 db:CRITICAL: 'NoneType' object has no attribute 'text'
2013-10-08 11:52:52,998 db:CRITICAL: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/db/manage.py", line 79, in migrate_database
    update_current_version=not options.test)
  File "/usr/lib/python2.6/site-packages/pulp/server/db/migrate/models.py", line 161, in apply_migration
    migration.migrate()
  File "/usr/lib/python2.6/site-packages/pulp_rpm/migrations/0011_new_importer.py", line 29, in migrate
    _migrate_collection(type_id)
  File "/usr/lib/python2.6/site-packages/pulp_rpm/migrations/0011_new_importer.py", line 56, in _migrate_collection
    package['sourcerpm'] = format_element.find('sourcerpm').text
AttributeError: 'NoneType' object has no attribute 'text'

Database migrations complete.
Loading content types.
Content types loaded.

Expected results:
Upgrade being complete

Additional info:

After some investigation it seems that I have packages with arch=src and with (obviously) and empty "sourcerpm" in the units_rpm collection.

> db.units_rpm.find({arch: 'src', sourcerpm: ''}).count()
3

Is this expected? Shouldn't they be in the units_srpm instead.

To work around this issue, I changed the following line in 0011_new_importer.py:

package['sourcerpm'] = format_element.find('sourcerpm').text

by

try:
    package['sourcerpm'] = format_element.find('sourcerpm').text
except AttributeError:
    package['sourcerpm'] = ''

I'm not really sure this is the proper way to fix this

Comment 1 Michael Hrivnak 2013-10-09 16:02:08 UTC

This turned out to be that the user had SRPMs in the RPM collection. This does point out that we lack a srpm upload command, so we should add that to resolve this issue.

Comment 2 Sayli Karmarkar 2013-10-09 16:13:30 UTC

Changing the summary according to #c1

Comment 3 Barnaby Court 2013-12-02 12:43:34 UTC

We already support uploading RPMs manually using "pulp-admin rpm repo uploads srpm --repo-id <repoid> --file <filename>"

I verified this command using the current master branch.

Comment 4 Preethi Thomas 2014-04-16 14:37:52 UTC

[root@pulp-24-server upload]# rpm -qa pulp-server
pulp-server-2.4.0-0.9.beta.el6.noarch
[root@pulp-24-server upload]# 

[root@pulp-24-server upload]# pulp-admin rpm repo uploads srpm  --repo-id srpm -f /root/upload/test-srpm01-1.0-1.src.rpm -v
+----------------------------------------------------------------------+
                              Unit Upload
+----------------------------------------------------------------------+

Extracting necessary metadata for each request...
[==================================================] 100%
Analyzing: test-srpm01-1.0-1.src.rpm
... completed

Files to be uploaded:
  test-srpm01-1.0-1.src.rpm

Creating upload requests on the server...
[==================================================] 100%
Initializing: test-srpm01-1.0-1.src.rpm
... completed

Starting upload of selected units. If this process is stopped through ctrl+c,
the uploads will be paused and may be resumed later using the resume command or
cancelled entirely using the cancel command.

Uploading: test-srpm01-1.0-1.src.rpm
[==================================================] 100%
1607/1607 bytes
... completed

Importing into the repository...
Import postponed due to queued operations against the repository. The progress
of this import can be viewed in the repository tasks list.

[root@pulp-24-server upload]#

Comment 5 Randy Barlow 2014-08-09 06:57:13 UTC

This has been fixed in Pulp 2.4.0-1.

Comment 6 Josh Baird 2014-09-29 23:03:40 UTC

I think I may be hitting the same bug.  I'm migrating from 2.3.1 to 2.4.1.

The pulp-manage-db fails:

...
Applying pulp_rpm.plugins.migrations version 9
Migration to pulp_rpm.plugins.migrations version 9 complete.
Applying pulp_rpm.plugins.migrations version 10
Migration to pulp_rpm.plugins.migrations version 10 complete.
Applying pulp_rpm.plugins.migrations version 11
Applying migration pulp_rpm.plugins.migrations.0011_new_importer failed.

Halting migrations due to a migration failure.  See log for details.
'NoneType' object has no attribute 'text'

The logs:

 pulp: pulp.server.db.manage:CRITICAL: Applying migration pulp_rpm.plugins.migrations.0011_new_importer failed.
 pulp: pulp.server.db.manage:CRITICAL:
 pulp: pulp.server.db.manage:CRITICAL: Halting migrations due to a migration failure.
 pulp: pulp.server.db.manage:CRITICAL: 'NoneType' object has no attribute 'text'
 pulp: pulp.server.db.manage:CRITICAL: Traceback (most recent call last):
 pulp: pulp.server.db.manage:CRITICAL:   File "/usr/lib/python2.6/site-packages/pulp/server/db/manage.py", line 111, in main
 pulp: pulp.server.db.manage:CRITICAL:     _auto_manage_db(options)
 pulp: pulp.server.db.manage:CRITICAL:   File "/usr/lib/python2.6/site-packages/pulp/server/db/manage.py", line 157, in _auto_manage_db
 pulp: pulp.server.db.manage:CRITICAL:     migrate_database(options)
 pulp: pulp.server.db.manage:CRITICAL:   File "/usr/lib/python2.6/site-packages/pulp/server/db/manage.py", line 86, in migrate_database
 pulp: pulp.server.db.manage:CRITICAL:     update_current_version=not options.test)
 pulp: pulp.server.db.manage:CRITICAL:   File "/usr/lib/python2.6/site-packages/pulp/server/db/migrate/models.py", line 161, in apply_m

 pulp: pulp.server.db.manage:CRITICAL:     migration.migrate()
 pulp: pulp.server.db.manage:CRITICAL:   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/migrations/0011_new_importer.py", line

 pulp: pulp.server.db.manage:CRITICAL:     _migrate_collection(type_id)
 pulp: pulp.server.db.manage:CRITICAL:   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/migrations/0011_new_importer.py", line

 pulp: pulp.server.db.manage:CRITICAL:     package['sourcerpm'] = format_element.find('sourcerpm').text
 pulp: pulp.server.db.manage:CRITICAL: AttributeError: 'NoneType' object has no attribute 'text'
 pulp: pulp.server.db.manage:CRITICAL: