Bug 1410252

Hello, 

I ran into the same problem on a second machine. 

So this issue is 100% reproducible and critical due to it's impact.

I'll add second foreman-debug as reference.

Created attachment 1239106 [details]
Foreman Debug 6.2.6 cci01-sat6

Upstream bug assigned to bbuckingham

Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/15297 has been resolved.

Removing upstream issue 15297 from the bugzilla.  That issue was delivered as part of Satellite 6.2 GA.

Note: from attachment foreman-debug-oD0C9: 

more of the stack trace:

2017-01-10 13:34:37 [app] [I] Processing by Katello::Api::V2::ContentViewsController#show as JSON
2017-01-10 13:34:37 [app] [I]   Parameters: {"api_version"=>"v2", "id"=>"106", "content_view"=>{}}
2017-01-10 13:34:37 [app] [I] Authorized user admin(Admin User)
2017-01-10 13:34:37 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.90/app/views/katello/api/v2/content_views/show.json.rabl within katello/api/v2/layouts/resource (34.1ms)
2017-01-10 13:34:37 [app] [I] Completed 200 OK in 79ms (Views: 28.8ms | ActiveRecord: 31.6ms)
2017-01-10 13:34:39 [foreman-tasks/action] [E] undefined method `[]' for nil:NilClass (NoMethodError)
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.90/app/lib/actions/pulp/repository/distributor_publish.rb:24:in `distributor_id'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.90/app/lib/actions/pulp/repository/distributor_publish.rb:16:in `invoke_external_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.3/lib/dynflow/action/polling.rb:83:in `initiate_external_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.3/lib/dynflow/action/polling.rb:18:in `run'
...
...

Note: from attachment foreman-debug-oD0C9: 

from var/log/foreman-installer/satellite.log:

[ERROR 2017-01-10 03:25:06 main] Upgrade step import_rpms failed. Check logs for more information.
[DEBUG 2017-01-10 03:25:06 main] Exit with status code: 1 (signal was 1)
[ERROR 2017-01-10 03:25:06 main] Errors encountered during run:
[ERROR 2017-01-10 03:25:06 main] rake aborted!
RestClient::InternalServerError: 500 Internal Server Error


This error message may be unrelated; however, it could be, if the upgrade was unsuccessful.

(In reply to Brad Buckingham from comment #7)
> Note: from attachment foreman-debug-oD0C9: 
> 
> from var/log/foreman-installer/satellite.log:
> 
> [ERROR 2017-01-10 03:25:06 main] Upgrade step import_rpms failed. Check logs
> for more information.
> [DEBUG 2017-01-10 03:25:06 main] Exit with status code: 1 (signal was 1)
> [ERROR 2017-01-10 03:25:06 main] Errors encountered during run:
> [ERROR 2017-01-10 03:25:06 main] rake aborted!
> RestClient::InternalServerError: 500 Internal Server Error
> 
> 
> This error message may be unrelated; however, it could be, if the upgrade
> was unsuccessful.

This was IMHO triggered by httpd restart, per /var/log/httpd/error_log :

[Tue Jan 10 03:25:03.141383 2017] [mpm_prefork:notice] [pid 25767] AH00171: Graceful restart requested, doing restart

(followed by usual startup logs)

Since import_rpms executes a rake script requiring httpd running, I expect the restart did the rake abort.


Further, checking foreman-debug from #c0, that is from a Sat6.1 system (foreman-1.7.2.62-1.el7sat.noarch installed) that also hit some upgrade error:

[ INFO 2016-12-05 22:42:42 main] Upgrade Step: update_repository_metadata (this may take a while) ...
[ERROR 2016-12-05 22:46:51 main] rake aborted!
Task 78ba6cf7-69d2-48b3-ad87-13e7906e9f4f: NoMethodError: undefined method `[]' for nil:NilClass

I see nothing common between the 2 foreman-debugs / how the internal reproducer matches the original scenario.

I suggest focusing on the upgrade error from #c0 since:
- that quite probably triggered the CV publish issues
- internal reproducer is flawed by httpd restart during upgrade that broke the upgrade that broke the CV publish later on

Francisco,

Is the foreman-debug attached in #c0 the correct log that led to the error mentioned in #c0?  As Pavel mentioned, that appears to contain the logs from a 6.1 server.

If the error was after performing an upgrade to 6.2:

1. Did anyone investigate and resolve the error that occurred in the 6.1 upgrade before performing the 6.2 upgrade?

2. Would it be possible to attach the foreman-debug from the 6.2 upgrade on that server?  As Pavel mentioned, the 6.2 upgrade on that user's install may have encountered something different than the reproducer.

(In reply to Harshad More from comment #10)
> Customer facing issue after upgrading to sat 6.2.8

This case has different root cause - KCS 1453313 was applied to Sat6.2 while it is applicable to 6.1 only (tricky to find it out). Upgrade to 6.2.8 went fine here, while it was the cause of problems in the original BZ report case.

I would suggest closing the BZ since upgrade went wrong - rather investigate why the upgrade failed than why something is broken after that failure. But that would be a different issue / bz.

Do you Francisco agree?

Closing this out as the case is resolved. If you feel this was done incorrectly, please feel free to re-open with additional information.

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days