Hide Forgot
Description of problem: Following exception raised in production.log while performing "yum update" to update packages from sat6.1.9 -> Sat6.2 snap12. Though the log-level seems "warning" but following exception is being logged in production.log ==> /var/log/foreman/production.log <== 2016-05-19 04:32:23 [app] [W] failed to detect boot server: ERF12-5139 [ProxyAPI::ProxyException]: Unable to detect TFTP boot server ([Errno::ECONNREFUSED]: Connection refused - connect(2) for "qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.co...) for Capsule https://qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com:9090/tftp | ProxyAPI::ProxyException: ERF12-5139 [ProxyAPI::ProxyException]: Unable to detect TFTP boot server ([Errno::ECONNREFUSED]: Connection refused - connect(2) for "qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.co...) for Capsule https://qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com:9090/tftp | /usr/share/foreman/lib/proxy_api/tftp.rb:49:in `rescue in bootServer' | /usr/share/foreman/lib/proxy_api/tftp.rb:42:in `bootServer' | /usr/share/foreman/app/models/concerns/orchestration/dhcp.rb:72:in `boot_server' | /usr/share/foreman/app/models/concerns/orchestration/dhcp.rb:99:in `dhcp_attrs' | /usr/share/foreman/app/models/concerns/orchestration/dhcp.rb:20:in `dhcp_record' | /usr/share/foreman/app/models/concerns/orchestration/dhcp.rb:179:in `dhcp_conflict_detected?' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:424:in `block in make_lambda' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:231:in `call' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:231:in `block in halting' Version-Release number of selected component (if applicable): Sat[6.1.9 -> 6.2 snap12] How reproducible: Steps to Reproduce: 1. set sat6.2 repo 2. katello-service stop 3. yum update -y Actual results: Expected results: No such exception in logs. If its a real issue then we need to fix it otherwise, we should suppress the exception, if possible. Additional info:
Created attachment 1159329 [details] foreman-debug logs
Ok, This warning/exception is reproducible with following upgrade steps: 1. Disable sat6.1 repo 2. Set/enable sat6.2 repo 3. yum install satellite -y 4. yum update -y 5. satellite-installer --scenario satellite --upgrade 2016-05-20 04:15:56 [app] [W] failed to detect boot server: ERF12-5139 [ProxyAPI::ProxyException]: Unable to detect TFTP boot server ([Errno::ECONNREFUSED]: Connection refused - connect(2) for "qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.co...) for Capsule https://qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com:9090/tftp | ProxyAPI::ProxyException: ERF12-5139 [ProxyAPI::ProxyException]: Unable to detect TFTP boot server ([Errno::ECONNREFUSED]: Connection refused - connect(2) for "qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.co...) for Capsule https://qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com:9090/tftp | /usr/share/foreman/lib/proxy_api/tftp.rb:49:in `rescue in bootServer' | /usr/share/foreman/lib/proxy_api/tftp.rb:42:in `bootServer' | /usr/share/foreman/app/models/concerns/orchestration/dhcp.rb:72:in `boot_server' | /usr/share/foreman/app/models/concerns/orchestration/dhcp.rb:99:in `dhcp_attrs' | /usr/share/foreman/app/models/concerns/orchestration/dhcp.rb:20:in `dhcp_record' | /usr/share/foreman/app/models/concerns/orchestration/dhcp.rb:179:in `dhcp_conflict_detected?' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:424:in `block in make_lambda' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:231:in `call'
I have not been able to reproduce this issue on my environment. I am testing with changes that will be coming in snap 13; therefore, it could be related to those changes or environmental or procedural differences. For now, I am going to triage this in, as we need to ensure the bug is resolved.
Reproducible with upgrade from sat6.1.9 -> sat6.2 snap13.1 steps to reproduce: ================== 1. Set sat6.2 repo on existing sat6.1 setup 2. yum update -y ==> /var/log/foreman/production.log <== 2016-05-26 00:38:44 [app] [W] failed to detect boot server: ERF12-5139 [ProxyAPI::ProxyException]: Unable to detect TFTP boot server ([Errno::ECONNREFUSED]: Connection refused - connect(2) for "qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.co...) for Capsule https://qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com:9090/tftp | ProxyAPI::ProxyException: ERF12-5139 [ProxyAPI::ProxyException]: Unable to detect TFTP boot server ([Errno::ECONNREFUSED]: Connection refused - connect(2) for "qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.co...) for Capsule https://qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com:9090/tftp | /usr/share/foreman/lib/proxy_api/tftp.rb:49:in `rescue in bootServer' | /usr/share/foreman/lib/proxy_api/tftp.rb:42:in `bootServer' | /usr/share/foreman/app/models/concerns/orchestration/dhcp.rb:72:in `boot_server' | /usr/share/foreman/app/models/concerns/orchestration/dhcp.rb:99:in `dhcp_attrs' | /usr/share/foreman/app/models/concerns/orchestration/dhcp.rb:20:in `dhcp_record' | /usr/share/foreman/app/models/concerns/orchestration/dhcp.rb:179:in `dhcp_conflict_detected?' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:424:in `block in make_lambda' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:231:in `call' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:231:in `block in halting' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:229:in `call' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:229:in `block in halting' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:229:in `call' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:229:in `block in halting' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:229:in `call' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:229:in `block in halting' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:229:in `call' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:229:in `block in halting' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:229:in `call' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:229:in `block in halting' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:86:in `call' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:86:in `run_callbacks' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activemodel-4.1.5/lib/active_model/validations/callbacks.rb:111:in `run_validations!' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activemodel-4.1.5/lib/active_model/validations.rb:318:in `valid?' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/validations.rb:70:in `valid?' | /usr/share/foreman/app/models/concerns/orchestration.rb:69:in `valid?' | /usr/share/foreman/app/models/host/managed.rb:1044:in `trigger_nic_orchestration'
the issue here is that a host object is being updated while being upgrade, triggering a proxy/capsule api call, which should not happen within a db migration. if you look at the logs, it looks like the following line triggered the host save /usr/share/foreman/db/migrate/20150612135546_create_host_status.rb:13:in `up'
I would assume this is not a blocker (besides the spam in the logs) but would investigate regardless.
Created redmine issue http://projects.theforeman.org/issues/15213 from this bug
Upstream bug component is DHCP & DNS
It appears that this bug will also require changes introduced in the upstream with http://projects.theforeman.org/issues/12425, which has been cloned to bug 1341860. Without this change, the following is observed during upgrade: [ERROR 2016-06-01 02:28:26 main] rake aborted! StandardError: An error has occurred, this and all later migrations canceled: undefined method `importing_facts=' for #<Host::Managed:0x000000098c19b0>/opt/rh/rh-ror41/root/usr/share/gems/gems/activemodel-4.1.5/lib/active_model/attribute_methods.rb:435:in `method_missing' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/attribute_methods.rb:208:in `method_missing' /usr/share/foreman/db/migrate/20150612135546_create_host_status.rb:19:in `ensure in block in up' /usr/share/foreman/db/migrate/20150612135546_create_host_status.rb:19:in `block in up' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:52:in `block (2 levels) in find_each' ...
Thanks Brad. You beat me on the issue in comment12. I just filed another bz for same issue: https://bugzilla.redhat.com/show_bug.cgi?id=1341974 (marked as duplicate)
Verified with upgrade from sat6.1.9 -> sat6.2 GA snap14 on rhel7. The reported issue related boot_server is no longer reproducible with upgrade performed with above builds. I don't find any exception during "yum update -y step". thank you for the fix. However I do see another issue as mentioned in comment12. But separate bz 1341860 is filed for this.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1501