Bug 1338742

Summary: Upgrade Step: remove_docker_v1_content failed (on RHEL6)
Product: Red Hat Satellite Reporter: Lukas Pramuk <lpramuk>
Component: UpgradesAssignee: Justin Sherrill <jsherril>
Status: CLOSED ERRATA QA Contact: Lukas Pramuk <lpramuk>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.2.0CC: bbuckingham, ehelms
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: rubygem-katello-3.0.0.38-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-07-27 11:12:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1335807    

Description Lukas Pramuk 2016-05-23 11:25:11 UTC
Description of problem:
Upgrade Step: remove_docker_v1_content failed

Version-Release number of selected component (if applicable):
Sat6.2.0-Snap12

How reproducible:
always

Steps to Reproduce:
# satellite-installer --upgrade
...
Upgrade Step: db_seed...
Upgrade Step: import_package_groups (this may take a while) ...
Upgrade Step: import_rpms (this may take a while) ...
Upgrade Step: import_distributions (this may take a while) ...
Upgrade Step: import_puppet_modules (this may take a while) ...
Upgrade Step: import_subscriptions (this may take a while) ...
Upgrade Step: remove_elasticsearch...
Elasticsearch has been removed as a dependency, the database files can be removed manually with #rm -rf /var/lib/elasticsearch
Upgrade Step: remove_docker_v1_content (this may take a while) ...
Upgrade step remove_docker_v1_content failed. Check logs for more information.
/usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/logger.rb:108:in `error': wrong number of arguments (2 for 1) (ArgumentError)
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/logger.rb:108:in `send'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/logger.rb:108:in `dump_buffer'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/logger.rb:108:in `each'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/logger.rb:108:in `dump_buffer'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/logger.rb:107:in `each'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/logger.rb:107:in `dump_buffer'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/logger.rb:98:in `dump_errors'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/logger.rb:103:in `dump_errors'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/exit_handler.rb:28:in `exit'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/kafo_configure.rb:162:in `exit'
	from /usr/share/katello-installer-base/hooks/post/30-upgrade.rb:58:in `fail_and_exit'
	from /usr/share/katello-installer-base/hooks/post/30-upgrade.rb:52:in `upgrade_step'
	from /usr/share/katello-installer-base/hooks/post/30-upgrade.rb:72:in `load'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/hook_context.rb:13:in `instance_exec'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/hook_context.rb:13:in `execute'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/hooking.rb:51:in `execute'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/hooking.rb:49:in `each'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/hooking.rb:49:in `execute'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/kafo_configure.rb:412:in `run_installation'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/exit_handler.rb:26:in `call'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/exit_handler.rb:26:in `exit'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/kafo_configure.rb:162:in `exit'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/kafo_configure.rb:411:in `run_installation'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/kafo_configure.rb:149:in `execute'
	from /usr/lib/ruby/gems/1.8/gems/clamp-0.6.2/lib/clamp/command.rb:67:in `run'
	from /usr/lib/ruby/gems/1.8/gems/clamp-0.6.2/lib/clamp/command.rb:125:in `run'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/kafo_configure.rb:156:in `run'
	from /usr/sbin/foreman-installer:12


Actual results:
upgrade step failed

Expected results:
upgrade step passes successfully

Comment 1 Lukas Pramuk 2016-05-23 11:26:52 UTC
/var/log/foreman-installer/satellite.log

[ INFO 2016-05-21 07:21:35 main] Upgrade Step: remove_docker_v1_content (this may take a while) ...
[ERROR 2016-05-21 07:21:52 main] rake aborted!
ForemanTasks::TaskError: Task 09589c9a-30f5-4bc0-80ff-11420ba39ec1: Katello::Errors::CandlepinError: Runtime Error null at org.candlepin.model.AbstractHibernateCurator.delete:326
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.14.4/lib/foreman_tasks.rb:40:in `block in sync_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.14.4/lib/foreman_tasks.rb:39:in `tap'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.14.4/lib/foreman_tasks.rb:39:in `sync_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.32/lib/katello/tasks/upgrades/3.0/delete_docker_v1_content.rake:19:in `block (5 levels) in <top (required)>'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.32/lib/katello/tasks/upgrades/3.0/delete_docker_v1_content.rake:15:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.32/lib/katello/tasks/upgrades/3.0/delete_docker_v1_content.rake:15:in `block (4 levels) in <top (required)>'
Tasks: TOP => katello:upgrades:3.0:delete_docker_v1_content
(See full trace by running task with --trace)
Deleting repositories that only contain Docker v1 content....
[ERROR 2016-05-21 07:21:52 main] Upgrade step remove_docker_v1_content failed. Check logs for more information.
[DEBUG 2016-05-21 07:21:52 main] Exit with status code: 1 (signal was 1)

Comment 2 Lukas Pramuk 2016-05-23 11:27:51 UTC
Successive runs:

Comment 3 Lukas Pramuk 2016-05-23 11:28:48 UTC
Deleting repositories that only contain Docker v1 content....rake aborted!
ForemanTasks::TaskError: Task 49adad1c-d216-4861-a238-68591d0cbd8b: RestClient::BadRequest: Katello::Resources::Candlepin::Content: 400 Bad Request <html><head><title>Apache Tomcat/6.0.24 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 400 - </h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>The request sent by the client was syntactically incorrect ().</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/6.0.24</h3></body></html> (DELETE /candlepin/content/1463151906334)
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.14.4/lib/foreman_tasks.rb:40:in `block in sync_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.14.4/lib/foreman_tasks.rb:39:in `tap'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.14.4/lib/foreman_tasks.rb:39:in `sync_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.32/lib/katello/tasks/upgrades/3.0/delete_docker_v1_content.rake:19:in `block (5 levels) in <top (required)>'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.32/lib/katello/tasks/upgrades/3.0/delete_docker_v1_content.rake:15:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.32/lib/katello/tasks/upgrades/3.0/delete_docker_v1_content.rake:15:in `block (4 levels) in <top (required)>'
Tasks: TOP => katello:upgrades:3.0:delete_docker_v1_content
(See full trace by running task with --trace)


# foreman-rake katello:upgrades:3.0:delete_docker_v1_content
Deleting repositories that only contain Docker v1 content....Done
Orphaned content deletion started in background.

Comment 4 Lukas Pramuk 2016-05-23 11:29:57 UTC
in comment #3 is missing 1st line:

# foreman-rake katello:upgrades:3.0:delete_docker_v1_content

Comment 8 Brad Buckingham 2016-05-24 14:50:08 UTC
I have not been able to reproduce this issue on my environment.  I am testing with changes that will be coming in snap 13; therefore, it could be related to those changes or environmental or procedural differences.  For now, I am going to triage this in, as we need to ensure the bug is resolved.

Comment 9 Lukas Pramuk 2016-05-26 11:11:55 UTC
Hi Brad,

to reproduce try to use Sat @ RHEL6 and populate some basic docker stuff
- compute resource (internal)
- docker CV
- registry
- containers (hub, cv, ext.registry)

Comment 10 Brad Buckingham 2016-05-26 19:34:14 UTC
I tried reproducing this again on my environment with Snap 13 and have not been able to.  My install is RHEL6 / Satellite 6.2 SNAP 13.  Prior to the upgrade from 6.1, I did populate some data, such as:

- product: docker
  - repo: centos
  - repo: ubuntu
  - repo: busybox

- all repos synced

- content view: centos w/ centos docker repo, published to Library
- content view: busybox w/ busybox docker repo, published to Library

I also created a few small custom yum repositories.

Since the behavior appears to be a race condition and there is a fix proposed in the upstream for an issue with the same stack, I am going to link the 2 up (upstream issue w/ this bz).

Comment 11 Bryan Kearney 2016-05-26 20:17:27 UTC
Upstream bug component is Content Management

Comment 12 Lukas Pramuk 2016-06-02 13:24:51 UTC
VERIFIED.

@satellite-6.2.0-14.1.el6sat.noarch
tfm-rubygem-katello-3.0.0.41-1.el6sat.noarch

# satellite-installer --upgrade
...
Upgrade Step: remove_docker_v1_content (this may take a while) ...
...

satellite.log:

[ INFO 2016-06-02 08:20:27 main] Upgrade Step: remove_docker_v1_content (this may take a while) ...
[DEBUG 2016-06-02 08:20:57 main] Deleting repositories that only contain Docker v1 content....Done
Orphaned content deletion started in background.

>>> no race condition duiring docker v1 content deletion

Comment 13 Bryan Kearney 2016-07-27 11:12:37 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1501