Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1130225

Summary: Can't log in to Satellite 6 on RHEL 7 after reboot
Product: Red Hat Satellite Reporter: Jason Montleon <jmontleo>
Component: OtherAssignee: Jason Montleon <jmontleo>
Status: CLOSED CURRENTRELEASE QA Contact: Og Maciel <omaciel>
Severity: high Docs Contact:
Priority: high    
Version: 6.0.4CC: bbuckingham, bkearney, omaciel, sthirugn
Target Milestone: Unspecified   
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-09-11 12:28:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
foreman-debug
none
logs none

Description Jason Montleon 2014-08-14 15:04:01 UTC 
Created attachment 926850 [details]
foreman-debug

Description of problem:
It's possible to log into Satellite 7 until you reboot.
After that you get a connection refused error upon logging in


Version-Release number of selected component (if applicable):
candlepin-0.9.23-1.el7.noarch
pulp-server-2.4.0-0.30.beta.el7sat.noarch
foreman-1.6.0.40-1.el7sat.noarch
katello-1.5.0-28.el7sat.noarch
ruby193-rubygem-katello-1.5.0-82.el7sat.noarch


How reproducible:
Seems always

Steps to Reproduce:
1. Install Satellite 6.0.4 on RHEL 7
2. Reboot
3. Log in

Actual results:
Error
Connection refused - connect(2)

Errno::ECONNREFUSED
Connection refused - connect(2)
app/models/user.rb:209:in `block in post_successful_login'
app/models/concerns/foreman/thread_session.rb:73:in `as'
app/models/concerns/foreman/thread_session.rb:79:in `as_anonymous_admin'
app/models/user.rb:208:in `post_successful_login'
app/models/user.rb:199:in `try_to_login'
app/controllers/users_controller.rb:72:in `login'
app/models/concerns/foreman/thread_session.rb:33:in `clear_thread'
lib/middleware/catch_json_parse_errors.rb:9:in `call' 

Expected results:
Should be able to log in without an error.

Additional info:
Comment 1 Jason Montleon 2014-08-14 15:04:26 UTC 
Created attachment 926851 [details]
logs
Comment 5 Dominic Cleal 2014-08-15 13:01:26 UTC 
I turned on debugging and got this stack trace:

Operation FAILED: Connection refused - connect(2)
/opt/rh/ruby193/root/usr/share/ruby/net/http.rb:763:in `initialize'
/opt/rh/ruby193/root/usr/share/ruby/net/http.rb:763:in `open'
/opt/rh/ruby193/root/usr/share/ruby/net/http.rb:763:in `block in connect'
/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:55:in `timeout'
/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:100:in `timeout'
/opt/rh/ruby193/root/usr/share/ruby/net/http.rb:763:in `connect'
/opt/rh/ruby193/root/usr/share/ruby/net/http.rb:756:in `do_start'
/opt/rh/ruby193/root/usr/share/ruby/net/http.rb:745:in `start'
/opt/rh/ruby193/root/usr/share/gems/gems/rbovirt-0.0.29/lib/restclient_ext/request.rb:44:in `transmit'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in `execute'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in `execute'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient.rb:72:in `post'
/opt/rh/ruby193/root/usr/share/gems/gems/tire-0.6.2/lib/tire/http/client.rb:19:in `post'
/opt/rh/ruby193/root/usr/share/gems/gems/tire-0.6.2/lib/tire/index.rb:146:in `store'
/opt/rh/ruby193/root/usr/share/gems/gems/tire-0.6.2/lib/tire/model/search.rb:148:in `block in update_index'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:403:in `_run__252718943274004389__update_elasticsearch_index__1536697697987125702__callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_update_elasticsearch_index_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/tire-0.6.2/lib/tire/model/search.rb:144:in `update_index'
/opt/rh/ruby193/root/usr/share/gems/gems/tire-0.6.2/lib/tire/model/callbacks.rb:21:in `block in included'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:468:in `_run__252718943274004389__save__1536697697987125702__callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_save_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/callbacks.rb:264:in `create_or_update'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/persistence.rb:84:in `save'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/validations.rb:50:in `save'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/attribute_methods/dirty.rb:22:in `save'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:241:in `block (2 levels) in save'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:295:in `block in with_transaction_returning_status'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in `transaction'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:208:in `transaction'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:293:in `with_transaction_returning_status'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:241:in `block in save'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:252:in `rollback_active_record_state!'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:240:in `save'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/katello/lazy_accessor.rb:96:in `save'
/opt/rh/ruby193/root/usr/share/gems/gems/foreman-tasks-0.6.6/app/models/foreman_tasks/concerns/action_triggering.rb:26:in `block in save'
/opt/rh/ruby193/root/usr/share/gems/gems/foreman-tasks-0.6.6/app/models/foreman_tasks/concerns/action_triggering.rb:106:in `dynflow_task_wrap'
/opt/rh/ruby193/root/usr/share/gems/gems/foreman-tasks-0.6.6/app/models/foreman_tasks/concerns/action_triggering.rb:26:in `save'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/persistence.rb:184:in `update_attribute'
/usr/share/foreman/app/models/user.rb:209:in `block in post_successful_login'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:73:in `as'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:79:in `as_anonymous_admin'
/usr/share/foreman/app/models/user.rb:208:in `post_successful_login'
/usr/share/foreman/app/models/user.rb:199:in `try_to_login'
/usr/share/foreman/app/controllers/users_controller.rb:72:in `login'

It looks like elasticsearch is down, and I can't restart it using the init script/systemd - it exits immediately.  After some debugging, it seems /var/run/elasticsearch isn't present - the package hasn't been updated properly for EL7 to use tmpfiles.

[root@satellite1 ~]# runuser -s /bin/bash elasticsearch -c 'ulimit -S -c 0 >/dev/null 2>&1 ; ES_HOME=/usr/share/java/elasticsearch ES_INCLUDE=/usr/share/java/elasticsearch/bin/elasticsearch.in.sh ES_MIN_MEM=256m ES_MAX_MEM=256m /usr/share/java/elasticsearch/bin/elasticsearch -p /var/run/elasticsearch/elasticsearch.pid -Des.config=/etc/elasticsearch/elasticsearch.yml -Des.path.conf=/etc/elasticsearch -Des.path.home=/usr/share/java/elasticsearch -Des.path.logs=/var/log/elasticsearch -Des.path.data=/var/lib/elasticsearch -Des.path.work=/tmp/elasticsearch -Des.path.plugins=/usr/share/java/elasticsearch/plugins'
[root@satellite1 ~]# {0.90.10}: pid Failed ...
- FileNotFoundException[/var/run/elasticsearch/elasticsearch.pid (No such file or directory)]

Something like this is needed in /usr/lib/tmpfiles.d/elasticsearch.conf:

d /run/elasticsearch 0755 elasticsearch elasticsearch -

(I can't tell what the upstream is for the package to check if one already exists.)
Comment 8 Og Maciel 2014-08-31 00:17:53 UTC 
Verified:

Browser:
=====
* Firefox 31.0 (MacOS)

Build:  
====
* Satellite/Satellite-6.0.4-RHEL-7-20140829.0

Packages:
======
* candlepin-0.9.23-1.el7.noarch
* candlepin-common-1.0.1-1.el7.noarch
* candlepin-guice-3.0-2_redhat_1.el7.noarch
* candlepin-scl-1-5.el7.noarch
* candlepin-scl-quartz-2.1.5-6.el7.noarch
* candlepin-scl-rhino-1.7R3-3.el7.noarch
* candlepin-scl-runtime-1-5.el7.noarch
* candlepin-selinux-0.9.23-1.el7.noarch
* candlepin-tomcat-0.9.23-1.el7.noarch
* elasticsearch-0.90.10-6.el7sat.noarch
* foreman-1.6.0.42-1.el7sat.noarch
* foreman-compute-1.6.0.42-1.el7sat.noarch
* foreman-gce-1.6.0.42-1.el7sat.noarch
* foreman-libvirt-1.6.0.42-1.el7sat.noarch
* foreman-ovirt-1.6.0.42-1.el7sat.noarch
* foreman-postgresql-1.6.0.42-1.el7sat.noarch
* foreman-proxy-1.6.0.30-1.el7sat.noarch
* foreman-selinux-1.6.0.14-1.el7sat.noarch
* foreman-vmware-1.6.0.42-1.el7sat.noarch
* katello-1.5.0-30.el7sat.noarch
* katello-certs-tools-1.5.6-1.el7sat.noarch
* katello-default-ca-1.0-1.noarch
* katello-installer-0.0.64-1.el7sat.noarch
* katello-server-ca-1.0-1.noarch
* openldap-2.4.39-3.el7.x86_64
* pulp-katello-0.3-4.el7sat.noarch
* pulp-nodes-common-2.4.1-0.5.rc1.el7sat.noarch
* pulp-nodes-parent-2.4.1-0.5.rc1.el7sat.noarch
* pulp-puppet-plugins-2.4.1-0.5.rc1.el7sat.noarch
* pulp-puppet-tools-2.4.1-0.5.rc1.el7sat.noarch
* pulp-rpm-plugins-2.4.1-0.6.beta.el7sat.noarch
* pulp-selinux-2.4.1-0.5.rc1.el7sat.noarch
* pulp-server-2.4.1-0.5.rc1.el7sat.noarch
* python-ldap-2.4.6-6.el7.x86_64
* ruby193-rubygem-net-ldap-0.3.1-3.el7sat.noarch
* ruby193-rubygem-runcible-1.1.0-2.el7sat.noarch
* rubygem-hammer_cli-0.1.1-12.el7sat.noarch
* rubygem-hammer_cli_foreman-0.1.1-16.el7sat.noarch
* rubygem-hammer_cli_foreman_tasks-0.0.3-3.el7sat.noarch
* rubygem-hammer_cli_import-0.10.2-1.2.el7sat.noarch
* rubygem-hammer_cli_katello-0.0.4-14.el7sat.noarch
Comment 9 Bryan Kearney 2014-09-11 12:28:44 UTC 
This was delivered with Satellite 6.0 which was released on 10 September 2014.