Bug 1130225 - Can't log in to Satellite 6 on RHEL 7 after reboot
Summary: Can't log in to Satellite 6 on RHEL 7 after reboot
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Other
Version: 6.0.4
Hardware: x86_64
OS: Linux
high
high vote
Target Milestone: Unspecified
Assignee: Jason Montleon
QA Contact: Og Maciel
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-08-14 15:04 UTC by Jason Montleon
Modified: 2014-09-11 12:28 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-09-11 12:28:44 UTC


Attachments (Terms of Use)
foreman-debug (225.34 KB, application/x-xz)
2014-08-14 15:04 UTC, Jason Montleon
no flags Details
logs (246.03 KB, application/x-xz)
2014-08-14 15:04 UTC, Jason Montleon
no flags Details

Description Jason Montleon 2014-08-14 15:04:01 UTC
Created attachment 926850 [details]
foreman-debug

Description of problem:
It's possible to log into Satellite 7 until you reboot.
After that you get a connection refused error upon logging in


Version-Release number of selected component (if applicable):
candlepin-0.9.23-1.el7.noarch
pulp-server-2.4.0-0.30.beta.el7sat.noarch
foreman-1.6.0.40-1.el7sat.noarch
katello-1.5.0-28.el7sat.noarch
ruby193-rubygem-katello-1.5.0-82.el7sat.noarch


How reproducible:
Seems always

Steps to Reproduce:
1. Install Satellite 6.0.4 on RHEL 7
2. Reboot
3. Log in

Actual results:
Error
Connection refused - connect(2)

Errno::ECONNREFUSED
Connection refused - connect(2)
app/models/user.rb:209:in `block in post_successful_login'
app/models/concerns/foreman/thread_session.rb:73:in `as'
app/models/concerns/foreman/thread_session.rb:79:in `as_anonymous_admin'
app/models/user.rb:208:in `post_successful_login'
app/models/user.rb:199:in `try_to_login'
app/controllers/users_controller.rb:72:in `login'
app/models/concerns/foreman/thread_session.rb:33:in `clear_thread'
lib/middleware/catch_json_parse_errors.rb:9:in `call' 

Expected results:
Should be able to log in without an error.

Additional info:

Comment 1 Jason Montleon 2014-08-14 15:04:26 UTC
Created attachment 926851 [details]
logs

Comment 5 Dominic Cleal 2014-08-15 13:01:26 UTC
I turned on debugging and got this stack trace:

Operation FAILED: Connection refused - connect(2)
/opt/rh/ruby193/root/usr/share/ruby/net/http.rb:763:in `initialize'
/opt/rh/ruby193/root/usr/share/ruby/net/http.rb:763:in `open'
/opt/rh/ruby193/root/usr/share/ruby/net/http.rb:763:in `block in connect'
/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:55:in `timeout'
/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:100:in `timeout'
/opt/rh/ruby193/root/usr/share/ruby/net/http.rb:763:in `connect'
/opt/rh/ruby193/root/usr/share/ruby/net/http.rb:756:in `do_start'
/opt/rh/ruby193/root/usr/share/ruby/net/http.rb:745:in `start'
/opt/rh/ruby193/root/usr/share/gems/gems/rbovirt-0.0.29/lib/restclient_ext/request.rb:44:in `transmit'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in `execute'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in `execute'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient.rb:72:in `post'
/opt/rh/ruby193/root/usr/share/gems/gems/tire-0.6.2/lib/tire/http/client.rb:19:in `post'
/opt/rh/ruby193/root/usr/share/gems/gems/tire-0.6.2/lib/tire/index.rb:146:in `store'
/opt/rh/ruby193/root/usr/share/gems/gems/tire-0.6.2/lib/tire/model/search.rb:148:in `block in update_index'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:403:in `_run__252718943274004389__update_elasticsearch_index__1536697697987125702__callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_update_elasticsearch_index_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/tire-0.6.2/lib/tire/model/search.rb:144:in `update_index'
/opt/rh/ruby193/root/usr/share/gems/gems/tire-0.6.2/lib/tire/model/callbacks.rb:21:in `block in included'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:468:in `_run__252718943274004389__save__1536697697987125702__callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_save_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/callbacks.rb:264:in `create_or_update'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/persistence.rb:84:in `save'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/validations.rb:50:in `save'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/attribute_methods/dirty.rb:22:in `save'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:241:in `block (2 levels) in save'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:295:in `block in with_transaction_returning_status'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in `transaction'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:208:in `transaction'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:293:in `with_transaction_returning_status'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:241:in `block in save'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:252:in `rollback_active_record_state!'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:240:in `save'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/katello/lazy_accessor.rb:96:in `save'
/opt/rh/ruby193/root/usr/share/gems/gems/foreman-tasks-0.6.6/app/models/foreman_tasks/concerns/action_triggering.rb:26:in `block in save'
/opt/rh/ruby193/root/usr/share/gems/gems/foreman-tasks-0.6.6/app/models/foreman_tasks/concerns/action_triggering.rb:106:in `dynflow_task_wrap'
/opt/rh/ruby193/root/usr/share/gems/gems/foreman-tasks-0.6.6/app/models/foreman_tasks/concerns/action_triggering.rb:26:in `save'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/persistence.rb:184:in `update_attribute'
/usr/share/foreman/app/models/user.rb:209:in `block in post_successful_login'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:73:in `as'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:79:in `as_anonymous_admin'
/usr/share/foreman/app/models/user.rb:208:in `post_successful_login'
/usr/share/foreman/app/models/user.rb:199:in `try_to_login'
/usr/share/foreman/app/controllers/users_controller.rb:72:in `login'

It looks like elasticsearch is down, and I can't restart it using the init script/systemd - it exits immediately.  After some debugging, it seems /var/run/elasticsearch isn't present - the package hasn't been updated properly for EL7 to use tmpfiles.

[root@satellite1 ~]# runuser -s /bin/bash elasticsearch -c 'ulimit -S -c 0 >/dev/null 2>&1 ; ES_HOME=/usr/share/java/elasticsearch ES_INCLUDE=/usr/share/java/elasticsearch/bin/elasticsearch.in.sh ES_MIN_MEM=256m ES_MAX_MEM=256m /usr/share/java/elasticsearch/bin/elasticsearch -p /var/run/elasticsearch/elasticsearch.pid -Des.config=/etc/elasticsearch/elasticsearch.yml -Des.path.conf=/etc/elasticsearch -Des.path.home=/usr/share/java/elasticsearch -Des.path.logs=/var/log/elasticsearch -Des.path.data=/var/lib/elasticsearch -Des.path.work=/tmp/elasticsearch -Des.path.plugins=/usr/share/java/elasticsearch/plugins'
[root@satellite1 ~]# {0.90.10}: pid Failed ...
- FileNotFoundException[/var/run/elasticsearch/elasticsearch.pid (No such file or directory)]

Something like this is needed in /usr/lib/tmpfiles.d/elasticsearch.conf:

d /run/elasticsearch 0755 elasticsearch elasticsearch -

(I can't tell what the upstream is for the package to check if one already exists.)

Comment 8 Og Maciel 2014-08-31 00:17:53 UTC
Verified:

Browser:
=====
* Firefox 31.0 (MacOS)

Build:  
====
* Satellite/Satellite-6.0.4-RHEL-7-20140829.0

Packages:
======
* candlepin-0.9.23-1.el7.noarch
* candlepin-common-1.0.1-1.el7.noarch
* candlepin-guice-3.0-2_redhat_1.el7.noarch
* candlepin-scl-1-5.el7.noarch
* candlepin-scl-quartz-2.1.5-6.el7.noarch
* candlepin-scl-rhino-1.7R3-3.el7.noarch
* candlepin-scl-runtime-1-5.el7.noarch
* candlepin-selinux-0.9.23-1.el7.noarch
* candlepin-tomcat-0.9.23-1.el7.noarch
* elasticsearch-0.90.10-6.el7sat.noarch
* foreman-1.6.0.42-1.el7sat.noarch
* foreman-compute-1.6.0.42-1.el7sat.noarch
* foreman-gce-1.6.0.42-1.el7sat.noarch
* foreman-libvirt-1.6.0.42-1.el7sat.noarch
* foreman-ovirt-1.6.0.42-1.el7sat.noarch
* foreman-postgresql-1.6.0.42-1.el7sat.noarch
* foreman-proxy-1.6.0.30-1.el7sat.noarch
* foreman-selinux-1.6.0.14-1.el7sat.noarch
* foreman-vmware-1.6.0.42-1.el7sat.noarch
* katello-1.5.0-30.el7sat.noarch
* katello-certs-tools-1.5.6-1.el7sat.noarch
* katello-default-ca-1.0-1.noarch
* katello-installer-0.0.64-1.el7sat.noarch
* katello-server-ca-1.0-1.noarch
* openldap-2.4.39-3.el7.x86_64
* pulp-katello-0.3-4.el7sat.noarch
* pulp-nodes-common-2.4.1-0.5.rc1.el7sat.noarch
* pulp-nodes-parent-2.4.1-0.5.rc1.el7sat.noarch
* pulp-puppet-plugins-2.4.1-0.5.rc1.el7sat.noarch
* pulp-puppet-tools-2.4.1-0.5.rc1.el7sat.noarch
* pulp-rpm-plugins-2.4.1-0.6.beta.el7sat.noarch
* pulp-selinux-2.4.1-0.5.rc1.el7sat.noarch
* pulp-server-2.4.1-0.5.rc1.el7sat.noarch
* python-ldap-2.4.6-6.el7.x86_64
* ruby193-rubygem-net-ldap-0.3.1-3.el7sat.noarch
* ruby193-rubygem-runcible-1.1.0-2.el7sat.noarch
* rubygem-hammer_cli-0.1.1-12.el7sat.noarch
* rubygem-hammer_cli_foreman-0.1.1-16.el7sat.noarch
* rubygem-hammer_cli_foreman_tasks-0.0.3-3.el7sat.noarch
* rubygem-hammer_cli_import-0.10.2-1.2.el7sat.noarch
* rubygem-hammer_cli_katello-0.0.4-14.el7sat.noarch

Comment 9 Bryan Kearney 2014-09-11 12:28:44 UTC
This was delivered with Satellite 6.0 which was released on 10 September 2014.


Note You need to log in before you can comment on or make changes to this bug.