Bug 1410252 - [upgrade]content-view publish on upgraded setup fails: undefined method `[]' for nil:NilClass (NoMethodError) [NEEDINFO]
Summary: [upgrade]content-view publish on upgraded setup fails: undefined method `[]' ...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Content Management
Version: 6.2.6
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified vote
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Katello QA List
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-04 22:18 UTC by Francisco Javier Lopez Y Grueber
Modified: 2020-05-14 15:30 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-08-14 19:27:19 UTC
Target Upstream Version:
bbuckingham: needinfo? (flg)
pmoravec: needinfo? (flg)


Attachments (Terms of Use)
foreman-debug (13.78 MB, application/x-xz)
2017-01-04 22:18 UTC, Francisco Javier Lopez Y Grueber
no flags Details
Foreman Debug 6.2.6 cci01-sat6 (4.95 MB, application/x-xz)
2017-01-10 13:14 UTC, Francisco Javier Lopez Y Grueber
no flags Details

Description Francisco Javier Lopez Y Grueber 2017-01-04 22:18:49 UTC
Created attachment 1237370 [details]
foreman-debug

Description of problem:

The server hsa been updated to satellite 6.2.6 successfully. 
All existing content views publish tasks fail with:

undefined method `[]' for nil:NilClass (NoMethodError) 

Version-Release number of selected component (if applicable):

6.2.6

How reproducible:

Always

Steps to Reproduce:
1. for i in $(hammer content-view list --organization $ORG|grep -v ccv|grep cv-rhel-7 | awk {'print $3'});do hammer content-view publish --name=$i --organization $ORG --async;done
2.  tasks are queued
3.  Same happens when sending individual jobs

Actual results:

publishing fails with error 

Expected results:

publishing and promotion succeed

Additional info:

Addind foreman-debug for further details.

Comment 1 Francisco Javier Lopez Y Grueber 2017-01-10 13:05:57 UTC
Hello, 

I ran into the same problem on a second machine. 

So this issue is 100% reproducible and critical due to it's impact.

I'll add second foreman-debug as reference.

Comment 2 Francisco Javier Lopez Y Grueber 2017-01-10 13:14:29 UTC
Created attachment 1239106 [details]
Foreman Debug 6.2.6 cci01-sat6

Comment 3 pm-sat@redhat.com 2017-01-11 15:03:22 UTC
Upstream bug assigned to bbuckingham@redhat.com

Comment 4 pm-sat@redhat.com 2017-01-11 15:03:25 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/15297 has been resolved.

Comment 5 Brad Buckingham 2017-01-20 13:34:52 UTC
Removing upstream issue 15297 from the bugzilla.  That issue was delivered as part of Satellite 6.2 GA.

Comment 6 Brad Buckingham 2017-01-20 20:31:43 UTC
Note: from attachment foreman-debug-oD0C9: 

more of the stack trace:

2017-01-10 13:34:37 [app] [I] Processing by Katello::Api::V2::ContentViewsController#show as JSON
2017-01-10 13:34:37 [app] [I]   Parameters: {"api_version"=>"v2", "id"=>"106", "content_view"=>{}}
2017-01-10 13:34:37 [app] [I] Authorized user admin(Admin User)
2017-01-10 13:34:37 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.90/app/views/katello/api/v2/content_views/show.json.rabl within katello/api/v2/layouts/resource (34.1ms)
2017-01-10 13:34:37 [app] [I] Completed 200 OK in 79ms (Views: 28.8ms | ActiveRecord: 31.6ms)
2017-01-10 13:34:39 [foreman-tasks/action] [E] undefined method `[]' for nil:NilClass (NoMethodError)
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.90/app/lib/actions/pulp/repository/distributor_publish.rb:24:in `distributor_id'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.90/app/lib/actions/pulp/repository/distributor_publish.rb:16:in `invoke_external_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.3/lib/dynflow/action/polling.rb:83:in `initiate_external_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.3/lib/dynflow/action/polling.rb:18:in `run'
...
...

Comment 7 Brad Buckingham 2017-01-20 20:33:50 UTC
Note: from attachment foreman-debug-oD0C9: 

from var/log/foreman-installer/satellite.log:

[ERROR 2017-01-10 03:25:06 main] Upgrade step import_rpms failed. Check logs for more information.
[DEBUG 2017-01-10 03:25:06 main] Exit with status code: 1 (signal was 1)
[ERROR 2017-01-10 03:25:06 main] Errors encountered during run:
[ERROR 2017-01-10 03:25:06 main] rake aborted!
RestClient::InternalServerError: 500 Internal Server Error


This error message may be unrelated; however, it could be, if the upgrade was unsuccessful.

Comment 8 Pavel Moravec 2017-01-23 09:18:22 UTC
(In reply to Brad Buckingham from comment #7)
> Note: from attachment foreman-debug-oD0C9: 
> 
> from var/log/foreman-installer/satellite.log:
> 
> [ERROR 2017-01-10 03:25:06 main] Upgrade step import_rpms failed. Check logs
> for more information.
> [DEBUG 2017-01-10 03:25:06 main] Exit with status code: 1 (signal was 1)
> [ERROR 2017-01-10 03:25:06 main] Errors encountered during run:
> [ERROR 2017-01-10 03:25:06 main] rake aborted!
> RestClient::InternalServerError: 500 Internal Server Error
> 
> 
> This error message may be unrelated; however, it could be, if the upgrade
> was unsuccessful.

This was IMHO triggered by httpd restart, per /var/log/httpd/error_log :

[Tue Jan 10 03:25:03.141383 2017] [mpm_prefork:notice] [pid 25767] AH00171: Graceful restart requested, doing restart

(followed by usual startup logs)

Since import_rpms executes a rake script requiring httpd running, I expect the restart did the rake abort.


Further, checking foreman-debug from #c0, that is from a Sat6.1 system (foreman-1.7.2.62-1.el7sat.noarch installed) that also hit some upgrade error:

[ INFO 2016-12-05 22:42:42 main] Upgrade Step: update_repository_metadata (this may take a while) ...
[ERROR 2016-12-05 22:46:51 main] rake aborted!
Task 78ba6cf7-69d2-48b3-ad87-13e7906e9f4f: NoMethodError: undefined method `[]' for nil:NilClass

I see nothing common between the 2 foreman-debugs / how the internal reproducer matches the original scenario.

I suggest focusing on the upgrade error from #c0 since:
- that quite probably triggered the CV publish issues
- internal reproducer is flawed by httpd restart during upgrade that broke the upgrade that broke the CV publish later on

Comment 9 Brad Buckingham 2017-01-23 14:08:24 UTC
Francisco,

Is the foreman-debug attached in #c0 the correct log that led to the error mentioned in #c0?  As Pavel mentioned, that appears to contain the logs from a 6.1 server.

If the error was after performing an upgrade to 6.2:

1. Did anyone investigate and resolve the error that occurred in the 6.1 upgrade before performing the 6.2 upgrade?

2. Would it be possible to attach the foreman-debug from the 6.2 upgrade on that server?  As Pavel mentioned, the 6.2 upgrade on that user's install may have encountered something different than the reproducer.

Comment 11 Pavel Moravec 2017-04-22 07:24:09 UTC
(In reply to Harshad More from comment #10)
> Customer facing issue after upgrading to sat 6.2.8

This case has different root cause - KCS 1453313 was applied to Sat6.2 while it is applicable to 6.1 only (tricky to find it out). Upgrade to 6.2.8 went fine here, while it was the cause of problems in the original BZ report case.

I would suggest closing the BZ since upgrade went wrong - rather investigate why the upgrade failed than why something is broken after that failure. But that would be a different issue / bz.

Do you Francisco agree?

Comment 12 Bryan Kearney 2017-08-14 19:27:19 UTC
Closing this out as the case is resolved. If you feel this was done incorrectly, please feel free to re-open with additional information.


Note You need to log in before you can comment on or make changes to this bug.