Bug 1406486
| Summary: | Deployment of HE stuck on "Waiting for the host to become operational in the engine. This may take several minutes..." for 14 minutes and both ha-agent and broker died. | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [oVirt] ovirt-engine | Reporter: | Nikolai Sednev <nsednev> | ||||||||
| Component: | BLL.HostedEngine | Assignee: | Doron Fediuck <dfediuck> | ||||||||
| Status: | CLOSED DUPLICATE | QA Contact: | Nikolai Sednev <nsednev> | ||||||||
| Severity: | urgent | Docs Contact: | |||||||||
| Priority: | unspecified | ||||||||||
| Version: | 4.1.0 | CC: | bugs, dfediuck, stirabos, ylavi | ||||||||
| Target Milestone: | ovirt-4.1.0-beta | Keywords: | Regression | ||||||||
| Target Release: | --- | Flags: | rule-engine:
ovirt-4.1?
rule-engine: blocker? nsednev: planning_ack? nsednev: devel_ack? nsednev: testing_ack? |
||||||||
| Hardware: | x86_64 | ||||||||||
| OS: | Linux | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||
| Doc Text: | Story Points: | --- | |||||||||
| Clone Of: | Environment: | ||||||||||
| Last Closed: | 2017-01-09 11:15:59 UTC | Type: | Bug | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | Integration | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Attachments: |
|
||||||||||
Created attachment 1233934 [details]
Screenshot from 2016-12-20 18-50-05.png
Created attachment 1233937 [details]
logs from the engine
Created attachment 1233938 [details]
sosreport from puma19
I suspect that could be related to:
Dec 20 17:23:02 puma19 sshd[20506]: Server listening on 0.0.0.0 port 2223.
Dec 20 17:23:02 puma19 sshd[20506]: Server listening on :: port 2223.
Dec 20 17:23:03 puma19 journal: vdsm vds WARN Not ready yet, ignoring event u'|virt|VM_status|2fc3ee66-d7e2-4858-84a0-2bcdda11efb9' args={u'2fc3ee66-d7e2-4858-84a0-2bcdda11efb9': {'status': 'Up', 'displayInfo': [{'tlsPort': '-1', 'ipAddress': '0', 'type': u'vnc', 'port': '5900'}], 'hash': '-5667167303080685580', 'displayIp': '0', 'displayPort': '5900', 'displaySecurePort': '-1', 'timeOffset': '0', 'pauseCode': 'NOERR', 'vcpuQuota': '-1', 'cpuUser': '0.00', 'monitorResponse': '0', 'elapsedTime': '241', 'displayType': 'vnc', 'cpuSys': '0.00', 'clientIp': '', 'vcpuPeriod': 100000L}}
Dec 20 17:23:04 puma19 journal: vdsm vds WARN Not ready yet, ignoring event u'|virt|VM_status|2fc3ee66-d7e2-4858-84a0-2bcdda11efb9' args={u'2fc3ee66-d7e2-4858-84a0-2bcdda11efb9': {'status': 'Up', 'username': 'Unknown', 'memUsage': '17', 'guestFQDN': '', 'memoryStats': {u'swap_out': '0', u'majflt': '0', u'swap_usage': '0', u'mem_cached': '768368', u'mem_free': '13835024', u'mem_buffers': '33840', u'swap_in': '0', u'swap_total': '0', u'pageflt': '2995', u'mem_total': '16268136', u'mem_unused': '13032816'}, 'session': 'Unknown', 'netIfaces': [], 'guestCPUCount': -1, 'appsList': (), 'guestIPs': '', 'disksUsage': []}}
This bug report has Keywords: Regression or TestBlocker. Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP. Any news here? The last sucesfull interaction between hosted-engine-setup and the engine using the oVirt python SDK was at 2016-12-20 17:22:56 DEBUG otopi.plugins.gr_he_setup.engine.add_host add_host._wait_host_ready:92 VDSM host in installing state The next poll cycle will be one second after but it got stuck forever. This is because at the same time host-deploy restarted iptables configured with nf_conntrack and the connection got probably stuck there. Dec 20 17:22:57 puma19 systemd: Starting IPv4 firewall with iptables... Dec 20 17:22:57 puma19 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team Dec 20 17:22:57 puma19 kernel: nf_conntrack version 0.5.0 (65536 buckets, 262144 max) Dec 20 17:22:57 puma19 kernel: Bridge firewalling registered Dec 20 17:22:57 puma19 iptables.init: iptables: Applying firewall rules: [ OK ] Dec 20 17:22:57 puma19 systemd: Started IPv4 firewall with iptables. At the same time the oVirt python SDK doesn't provide a default timeout and so also hosted-engine-setup got stuck forever. *** This bug has been marked as a duplicate of bug 1410501 *** https://bugzilla.redhat.com/show_bug.cgi?id=1406486 Reported: 2016-12-20 11:50 EST by Nikolai Sednev https://bugzilla.redhat.com/show_bug.cgi?id=1410501 Reported: 2017-01-05 10:47 EST by Simone Tiraboschi My bug was opened much earlier, may you please consider a change for this bug's status? Sorry Nikolai, I closed this one as a duplicate by mistake but I should have closed 1410501 being opened after this one. I'd prefer to keep now like this since 1410501 is already in the errata process and so on. |
Description of problem: Deployment of HE stuck on "Waiting for the host to become operational in the engine. This may take several minutes..." for 14 minutes and both ha-agent and broker died. I've noticed that engine is accessable via WEBADMIN, host is present and active inside, while deployment got stuck, also hostname of the engine is incorrect, it should've been nsednev-he-4.scl.lab.tlv.redhat.com, but got nsednev-he-4, as I saw via CLI. Eventually I had to abort the deployment from the shell. as it got stuck. puma19 ~]# hosted-engine --deploy [ INFO ] Stage: Initializing [ INFO ] Generating a temporary VNC password. [ INFO ] Stage: Environment setup During customization use CTRL-D to abort. Continuing will configure this host for serving as hypervisor and create a VM where you have to install the engine afterwards. Are you sure you want to continue? (Yes, No)[Yes]: It has been detected that this program is executed through an SSH connection without using screen. Continuing with the installation may lead to broken installation if the network connection fails. It is highly recommended to abort the installation and run it inside a screen session using command "screen". Do you want to continue anyway? (Yes, No)[No]: yes [ INFO ] Hardware supports virtualization Configuration files: [] Log file: /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20161220171158-2av0mm.log Version: otopi-1.6.0_master (otopi-1.6.0-0.0.master.20160928073940.git7499a54.el7.centos) [ INFO ] Detecting available oVirt engine appliances [ INFO ] Stage: Environment packages setup [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment setup [ INFO ] Generating libvirt-spice certificates [WARNING] Cannot locate gluster packages, Hyper Converged setup support will be disabled. [ INFO ] Please abort the setup and install vdsm-gluster, glusterfs-server >= 3.7.2 and restart vdsmd service in order to gain Hyper Converged setup support. [ INFO ] Stage: Environment customization --== STORAGE CONFIGURATION ==-- Please specify the storage you would like to use (glusterfs, iscsi, fc, nfs3, nfs4)[nfs3]: Please specify the full shared storage connection path to use (example: host:/path): 10.35.110.11:/Compute_NFS/nsednev_he_4 If needed, specify additional mount options for the connection to the hosted-engine storage domain []: --== HOST NETWORK CONFIGURATION ==-- iptables was detected on your computer, do you wish setup to configure it? (Yes, No)[Yes]: Please indicate a pingable gateway IP address [10.35.160.254]: Please indicate a nic to set ovirtmgmt bridge on: (enp5s0f1, enp5s0f0, enp4s0f0, enp4s0f1) [enp5s0f1]: enp4s0f0 --== VM CONFIGURATION ==-- The following appliance have been found on your system: [1] - The oVirt Engine Appliance image (OVA) - 4.1-20161202.1.el7.centos [2] - Directly select an OVA file Please select an appliance (1, 2) [1]: [ INFO ] Verifying its sha1sum [ INFO ] Checking OVF archive content (could take a few minutes depending on archive size) [ INFO ] Checking OVF XML content (could take a few minutes depending on archive size) Please specify the console type you would like to use to connect to the VM (vnc, spice) [vnc]: [ INFO ] Detecting host timezone. Would you like to use cloud-init to customize the appliance on the first boot (Yes, No)[Yes]? Would you like to generate on-fly a cloud-init ISO image (of no-cloud type) or do you have an existing one (Generate, Existing)[Generate]? Please provide the FQDN you would like to use for the engine appliance. Note: This will be the FQDN of the engine VM you are now going to launch, it should not point to the base host or to any other existing machine. Engine VM FQDN: (leave it empty to skip): []: nsednev-he-4.scl.lab.tlv.redhat.com Please provide the domain name you would like touse for the engine appliance. Engine VM domain: [scl.lab.tlv.redhat.com] Automatically execute engine-setup on the engine appliance on first boot (Yes, No)[Yes]? Automatically restart the engine VM as a monitored service after engine-setup (Yes, No)[Yes]? Enter root password that will be used for the engine appliance (leave it empty to skip): Confirm appliance root password: Enter ssh public key for the root user that will be used for the engine appliance (leave it empty to skip): [WARNING] Skipping appliance root ssh public key Do you want to enable ssh access for the root user (yes, no, without-password) [yes]: Please specify the size of the VM disk in GB: [25]: 50 Please specify the memory size of the VM in MB (Defaults to appliance OVF value): [16384]: The following CPU types are supported by this host: - model_Westmere: Intel Westmere Family - model_Nehalem: Intel Nehalem Family - model_Penryn: Intel Penryn Family - model_Conroe: Intel Conroe Family Please specify the CPU type to be used by the VM [model_Westmere]: Please specify the number of virtual CPUs for the VM (Defaults to appliance OVF value): [4]: 6 You may specify a unicast MAC address for the VM or accept a randomly generated default [00:16:3e:27:73:ab]: 00:16:3E:7D:DD:DD How should the engine VM network be configured (DHCP, Static)[DHCP]? Add lines for the appliance itself and for this host to /etc/hosts on the engine VM? Note: ensuring that this host could resolve the engine VM hostname is still up to you (Yes, No)[No] yes --== HOSTED ENGINE CONFIGURATION ==-- Enter engine admin password: Confirm engine admin password: Please provide the name of the SMTP server through which we will send notifications [localhost]: Please provide the TCP port number of the SMTP server [25]: Please provide the email address from which notifications will be sent [root@localhost]: Please provide a comma-separated list of email addresses which will get notifications [root@localhost]: [ INFO ] Stage: Setup validation --== CONFIGURATION PREVIEW ==-- Bridge interface : enp4s0f0 Engine FQDN : nsednev-he-4.scl.lab.tlv.redhat.com Bridge name : ovirtmgmt Host address : puma19.scl.lab.tlv.redhat.com SSH daemon port : 22 Firewall manager : iptables Gateway address : 10.35.160.254 Storage Domain type : nfs3 Image size GB : 50 Host ID : 1 Storage connection : 10.35.110.11:/Compute_NFS/nsednev_he_4 Console type : vnc Memory size MB : 16384 MAC address : 00:16:3E:7D:DD:DD Number of CPUs : 6 OVF archive (for disk boot) : /usr/share/ovirt-engine-appliance/ovirt-engine-appliance-4.1-20161202.1.el7.centos.ova Appliance version : 4.1-20161202.1.el7.centos Restart engine VM after engine-setup: True Engine VM timezone : Asia/Jerusalem CPU Type : model_Westmere Please confirm installation settings (Yes, No)[Yes]: [ INFO ] Stage: Transaction setup [ INFO ] Stage: Misc configuration [ INFO ] Stage: Package installation [ INFO ] Stage: Misc configuration [ INFO ] Configuring libvirt [ INFO ] Configuring VDSM [ INFO ] Starting vdsmd [ INFO ] Configuring the management bridge [ INFO ] Creating Storage Domain [ INFO ] Creating Storage Pool [ INFO ] Connecting Storage Pool [ INFO ] Verifying sanlock lockspace initialization [ INFO ] Creating Image for 'hosted-engine.lockspace' ... [ INFO ] Image for 'hosted-engine.lockspace' created successfully [ INFO ] Creating Image for 'hosted-engine.metadata' ... [ INFO ] Image for 'hosted-engine.metadata' created successfully [ INFO ] Creating VM Image [ INFO ] Extracting disk image from OVF archive (could take a few minutes depending on archive size) [ INFO ] Validating pre-allocated volume size [ INFO ] Uploading volume to data domain (could take a few minutes depending on archive size) [ INFO ] Image successfully imported from OVF [ INFO ] Destroying Storage Pool [ INFO ] Start monitoring domain [ INFO ] Configuring VM [ INFO ] Updating hosted-engine configuration [ INFO ] Stage: Transaction commit [ INFO ] Stage: Closing up [ INFO ] Creating VM You can now connect to the VM with the following command: hosted-engine --console You can also graphically connect to the VM from your system with the following command: remote-viewer vnc://puma19.scl.lab.tlv.redhat.com:5900 Use temporary password "7367YNKO" to connect to vnc console. Please ensure that your Guest OS is properly configured to support serial console according to your distro documentation. Follow http://www.ovirt.org/Serial_Console_Setup#I_need_to_access_the_console_the_old_way for more info. If you need to reboot the VM you will need to start it manually using the command: hosted-engine --vm-start You can then set a temporary password using the command: hosted-engine --add-console-password [ INFO ] Running engine-setup on the appliance |- [ INFO ] Stage: Initializing |- [ INFO ] Stage: Environment setup |- Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf', '/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/root/ovirt-engine-answers', '/root/heanswers.conf'] |- Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20161220171920-z3j0m3.log |- Version: otopi-1.6.0_master (otopi-1.6.0-0.0.master.20160928073940.git7499a54.el7.centos) |- [ INFO ] Stage: Environment packages setup |- [ INFO ] Stage: Programs detection |- [ INFO ] Stage: Environment setup |- [ INFO ] Stage: Environment customization |- |- --== PRODUCT OPTIONS ==-- |- |- Configure Image I/O Proxy on this host? (Yes, No) [Yes]: |- |- --== PACKAGES ==-- |- |- |- --== NETWORK CONFIGURATION ==-- |- |- [ INFO ] firewalld will be configured as firewall manager. |- |- --== DATABASE CONFIGURATION ==-- |- |- |- --== OVIRT ENGINE CONFIGURATION ==-- |- |- |- --== STORAGE CONFIGURATION ==-- |- |- |- --== PKI CONFIGURATION ==-- |- |- |- --== APACHE CONFIGURATION ==-- |- |- |- --== SYSTEM CONFIGURATION ==-- |- |- |- --== MISC CONFIGURATION ==-- |- |- Please choose Data Warehouse sampling scale: |- (1) Basic |- (2) Full |- (1, 2)[1]: |- |- --== END OF CONFIGURATION ==-- |- |- [ INFO ] Stage: Setup validation |- |- --== CONFIGURATION PREVIEW ==-- |- |- Application mode : both |- Default SAN wipe after delete : False |- Firewall manager : firewalld |- Update Firewall : True |- Host FQDN : nsednev-he-4.scl.lab.tlv.redhat.com |- Configure local Engine database : True |- Set application as default page : True |- Configure Apache SSL : True |- Engine database secured connection : False |- Engine database host : localhost |- Engine database user name : engine |- Engine database name : engine |- Engine database port : 5432 |- Engine database host name validation : False |- Engine installation : True |- PKI organization : scl.lab.tlv.redhat.com |- DWH installation : True |- DWH database secured connection : False |- DWH database host : localhost |- DWH database user name : ovirt_engine_history |- DWH database name : ovirt_engine_history |- DWH database port : 5432 |- DWH database host name validation : False |- Configure local DWH database : True |- Configure Image I/O Proxy : True |- Configure VMConsole Proxy : True |- Configure WebSocket Proxy : True |- [ INFO ] Stage: Transaction setup |- [ INFO ] Stopping engine service |- [ INFO ] Stopping ovirt-fence-kdump-listener service |- [ INFO ] Stopping dwh service |- [ INFO ] Stopping Image I/O Proxy service |- [ INFO ] Stopping websocket-proxy service |- [ INFO ] Stage: Misc configuration |- [ INFO ] Stage: Package installation |- [ INFO ] Stage: Misc configuration |- [ INFO ] Upgrading CA |- [ INFO ] Initializing PostgreSQL |- [ INFO ] Creating PostgreSQL 'engine' database |- [ INFO ] Configuring PostgreSQL |- [ INFO ] Creating PostgreSQL 'ovirt_engine_history' database |- [ INFO ] Configuring PostgreSQL |- [ INFO ] Creating CA |- [ INFO ] Creating/refreshing Engine database schema |- [ INFO ] Creating/refreshing DWH database schema |- [ INFO ] Configuring Image I/O Proxy |- [ INFO ] Setting up ovirt-vmconsole proxy helper PKI artifacts |- [ INFO ] Setting up ovirt-vmconsole SSH PKI artifacts |- [ INFO ] Configuring WebSocket Proxy |- [ INFO ] Creating/refreshing Engine 'internal' domain database schema |- [ INFO ] Generating post install configuration file '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf' |- [ INFO ] Stage: Transaction commit |- [ INFO ] Stage: Closing up |- [ INFO ] Starting engine service |- [ INFO ] Starting dwh service |- [ INFO ] Restarting ovirt-vmconsole proxy service |- |- --== SUMMARY ==-- |- |- [ INFO ] Restarting httpd |- Please use the user 'admin@internal' and password specified in order to login |- Web access is enabled at: |- http://nsednev-he-4.scl.lab.tlv.redhat.com:80/ovirt-engine |- https://nsednev-he-4.scl.lab.tlv.redhat.com:443/ovirt-engine |- Internal CA 4E:01:3C:90:76:7B:79:AB:FF:85:19:DB:D9:D0:8E:00:C9:4B:D2:E0 |- SSH fingerprint: 1c:3a:02:b8:83:14:02:57:c3:9b:d3:aa:00:a2:70:65 |- |- --== END OF SUMMARY ==-- |- |- [ INFO ] Stage: Clean up |- Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20161220171920-z3j0m3.log |- [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20161220172154-setup.conf' |- [ INFO ] Stage: Pre-termination |- [ INFO ] Stage: Termination |- [ INFO ] Execution of setup completed successfully |- HE_APPLIANCE_ENGINE_SETUP_SUCCESS [ INFO ] Engine-setup successfully completed [ INFO ] Engine is still unreachable [ INFO ] Engine is still not reachable, waiting... [ INFO ] Engine replied: DB Up!Welcome to Health Status! [ INFO ] Acquiring internal CA cert from the engine [ INFO ] The following CA certificate is going to be used, please immediately interrupt if not correct: [ INFO ] Issuer: C=US, O=scl.lab.tlv.redhat.com, CN=nsednev-he-4.scl.lab.tlv.redhat.com.93239, Subject: C=US, O=scl.lab.tlv.redhat.com, CN=nsednev-he-4.scl.lab.tlv.redhat.com.93239, Fingerprint (SHA-1): 4E013C90767B79ABFF8519DBD9D08E00C94BD2E0 [ INFO ] Connecting to the Engine [ INFO ] Waiting for the host to become operational in the engine. This may take several minutes... Version-Release number of selected component (if applicable): On host: ovirt-engine-appliance-4.1-20161202.1.el7.centos.noarch vdsm-4.18.999-1020.git1ff41b1.el7.centos.x86_64 libvirt-client-2.0.0-10.el7_3.2.x86_64 ovirt-vmconsole-1.0.4-1.el7ev.noarch ovirt-release41-pre-4.1.0-0.0.beta.20161201085255.git731841c.el7.centos.noarch sanlock-3.4.0-1.el7.x86_64 qemu-kvm-rhev-2.6.0-28.el7_3.2.x86_64 ovirt-hosted-engine-ha-2.1.0-0.0.master.20161130135331.20161130135328.git3541725.el7.centos.noarch ovirt-engine-appliance-4.1-20161202.1.el7.centos.noarch ovirt-host-deploy-1.6.0-0.0.master.20161107121647.gitfd7ddcd.el7.centos.noarch ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch ovirt-imageio-common-0.5.0-0.201611201242.gitb02532b.el7.centos.noarch ovirt-vmconsole-host-1.0.4-1.el7ev.noarch mom-0.5.8-1.el7ev.noarch ovirt-hosted-engine-setup-2.1.0-0.0.master.20161130101611.gitb3ad261.el7.centos.noarch ovirt-setup-lib-1.1.0-0.0.master.20161107100014.gitb73abeb.el7.centos.noarch ovirt-imageio-daemon-0.5.0-0.201611201242.gitb02532b.el7.centos.noarch Linux version 3.10.0-514.2.2.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Wed Nov 16 13:15:13 EST 2016 Linux 3.10.0-514.2.2.el7.x86_64 #1 SMP Wed Nov 16 13:15:13 EST 2016 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.3 (Maipo) Engine: Linux version 3.10.0-327.36.3.el7.x86_64 (builder.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Mon Oct 24 16:09:20 UTC 2016 Linux 3.10.0-327.36.3.el7.x86_64 #1 SMP Mon Oct 24 16:09:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux CentOS Linux release 7.2.1511 (Core) How reproducible: Steps to Reproduce: 1.Deploy HE over NFS on clean el7.3 host. 2. 3. Actual results: Got stuck with deployment. Expected results: ha-agent and broker should keep running and deployment should end successfully. Additional info: sosreport from host being attached.