Bug 1411491
|
Description
Derek Whatley
2017-01-09 20:28:49 UTC
According to the logs [1], it's an SSL issue in the daemon. When restarting the daemon the certificate reloads, hence works as expected.
@Derek - is it a completely clean installation? fresh engine and vdsm?
@Nir - can we automate the reload of the daemon's certificate? or it requires a daemon restart?
[1]
SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
(Thread-24 ) ERROR 2017-01-04 14:31:04,710 web:84:web:(log_response) 192.168.200.213 - PUT 503 209 (0.01s)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/web.py", line 43, in __call__
resp = self.dispatch(request)
File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/web.py", line 68, in dispatch
return method(*match.groups())
File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/http_helper.py", line 88, in wrapper
ret = func(self, *args)
File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/http_helper.py", line 59, in wrapper
ret = func(self, *args)
File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_handler.py", line 75, in put
return self.send_data(self.request)
File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_handler.py", line 116, in send_data
request.method, imaged_url, headers, body, stream)
File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_handler.py", line 187, in make_imaged_request
raise exc.HTTPServiceUnavailable(s)
HTTPServiceUnavailable: Failed communicating with vdsm-imaged: An SSL error occurred.
(In reply to Daniel Erez from comment #1) > According to the logs [1], it's an SSL issue in the daemon. When restarting > the daemon the certificate reloads, hence works as expected. > > @Derek - is it a completely clean installation? fresh engine and vdsm? Yes, completely clean installation. The sequence is as follows: 1) Run "hosted-engine --deploy" to deploy engine and hypervisor (docs reference linked below) 2) Create disks 3) Attempt disk image upload, fails https://access.redhat.com/documentation/en/red-hat-virtualization/4.0/paged/self-hosted-engine-guide/chapter-2-deploying-self-hosted-engine @Nir - another duplicate of bug 1401901? (In reply to Daniel Erez from comment #3) > @Nir - another duplicate of bug 1401901? Do you see the same error in the log? ovirt-imageio-daemon fail to start with permission error? @Derek - can you please also attach ovirt-imageio-daemon / vdsm / engine logs Created attachment 1239571 [details]
Collection of logs from hypervisor and engine.
If you have any specific additional requests for log files, please let me know.
@Derek - seems that you using an old version without logs for imageio. Which version of imageio daemon are you using? Can you try the described scenario with the latest one? I'm running ovirt-imageio-daemon-0.4.0-0.el7ev.noarch on the hypervisor. This seems to be the most up-to-date version according to the access.redhat.com package search. What is the name/path of the log file you were expecting to be included? We still seem to be seeing this when deploying multiple hypervisors. When we have a self-hosted engine deployment with multiple hypervisors, we have to restart the imageio-daemon on all of them before image uploads work, not just the one hosting the engine. (In reply to Fabian von Feilitzsch from comment #9) > We still seem to be seeing this when deploying multiple hypervisors. When we > have a self-hosted engine deployment with multiple hypervisors, we have to > restart the imageio-daemon on all of them before image uploads work, not > just the one hosting the engine. This is expected, you should restart ovirt-imageio-daemon after enrolling certificates on a host. The patch Daniel submitted should fix this issue during host-deploy. Does it matter much if verification process will use hosted-engine on RHEL7.3? (instead of using RHV hypervisior) (In reply to Natalie Gavrielov from comment #11) > Does it matter much if verification process will use hosted-engine on > RHEL7.3? (instead of using RHV hypervisior) Shouldn't matter, the flow is similar. Scenario performed: ------------------- 1. On a clean RHV hypervisor, performed hosted engine installation. 2. Configured certificates for both Chrome and Firefox, with all required (and possible) permissions. 3. Tried to upload an image via GUI (disks tab -> upload) * Tried both RAW and Qcow2 v3 formats. Result: ------- All attempts to upload failed (switching status to "Paused by the system"). Important notes: ---------------- 1. After manually restarting the ovirt-imageio-daemon upload image works! 2. I looked for the restart of the ovirt-imageio-daemon, that host deploy should do in ovirt-imageio-daemon.log and couldn't find any evidence for this restart (only the one I performed manually). 3. ovirt-imageio-proxy was not installed (never worked with hosted engine before but I would guess it's not supposed to be installed..) Builds used: RHVM 4.1.1.2-0.1.el7 ovirt-imageio-daemon-1.0.0-0.el7ev.noarch Will attach logs in no time. Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release. Created attachment 1265457 [details]
logs: setup, agent, broker, daemon, vdsm and more...
@Natalie - which version of ovirt-host-deploy is installed? (In reply to Daniel Erez from comment #16) > @Natalie - which version of ovirt-host-deploy is installed? ovirt-host-deploy-1.6.3-1.el7ev.noarch (In reply to Natalie Gavrielov from comment #17) > (In reply to Daniel Erez from comment #16) > > @Natalie - which version of ovirt-host-deploy is installed? > > ovirt-host-deploy-1.6.3-1.el7ev.noarch @Simone - how can I check if the commit is included in this version? According to the daemon log [1], it's seems that there wasn't a certificate issue, but a different network error - resolved as part of bug 1430243. A workaround to this issue is to manually increase the value of UploadImageXhrTimeoutInSeconds config value in the vdc_options table (e.g. from 10 to 120). @Natalie - can you please try that? [1] Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/web.py", line 48, in __call__ resp = self.dispatch(request) File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/web.py", line 73, in dispatch return method(*match.groups()) File "/usr/lib/python2.7/site-packages/ovirt_imageio_daemon/server.py", line 175, in put op.run() File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/directio.py", line 138, in run self._receive_chunk(dst, buf, count) File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/directio.py", line 174, in _receive_chunk raise errors.PartialContent(self.size, self.done) PartialContent: Requested 8388608 bytes, available 1433600 bytes (In reply to Daniel Erez from comment #19) > According to the daemon log [1], it's seems that there wasn't a certificate > issue, but a different network error - resolved as part of bug 1430243. > A workaround to this issue is to manually increase the value of > UploadImageXhrTimeoutInSeconds config value in the vdc_options table (e.g. > from 10 to 120). > > @Natalie - can you please try that? > > > [1] > Traceback (most recent call last): > File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/web.py", line > 48, in __call__ > resp = self.dispatch(request) > File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/web.py", line > 73, in dispatch > return method(*match.groups()) > File "/usr/lib/python2.7/site-packages/ovirt_imageio_daemon/server.py", > line 175, in put > op.run() > File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/directio.py", > line 138, in run > self._receive_chunk(dst, buf, count) > File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/directio.py", > line 174, in _receive_chunk > raise errors.PartialContent(self.size, self.done) > PartialContent: Requested 8388608 bytes, available 1433600 bytes This looks like a different issue, see the timeline in the log, the restart I performed to the daemon was at: 2017-03-22 17:49:54,151 These traceback errors your referring to happened later on (when upload actually started working). And if I recall correctly, disk size that was set in the UI was 8gb but the actual data that was transferred is around 1.5gb - so it might be just that (probably a bug.. but unrelated to this topic). If you still need this workaround to be tested It's best if this needinfo will move to its relevant bug. (In reply to Natalie Gavrielov from comment #20) > (In reply to Daniel Erez from comment #19) > > According to the daemon log [1], it's seems that there wasn't a certificate > > issue, but a different network error - resolved as part of bug 1430243. > > A workaround to this issue is to manually increase the value of > > UploadImageXhrTimeoutInSeconds config value in the vdc_options table (e.g. > > from 10 to 120). > > > > @Natalie - can you please try that? > > > > > > [1] > > Traceback (most recent call last): > > File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/web.py", line > > 48, in __call__ > > resp = self.dispatch(request) > > File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/web.py", line > > 73, in dispatch > > return method(*match.groups()) > > File "/usr/lib/python2.7/site-packages/ovirt_imageio_daemon/server.py", > > line 175, in put > > op.run() > > File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/directio.py", > > line 138, in run > > self._receive_chunk(dst, buf, count) > > File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/directio.py", > > line 174, in _receive_chunk > > raise errors.PartialContent(self.size, self.done) > > PartialContent: Requested 8388608 bytes, available 1433600 bytes > > This looks like a different issue, see the timeline in the log, the restart > I performed to the daemon was at: 2017-03-22 17:49:54,151 > These traceback errors your referring to happened later on (when upload > actually started working). > > And if I recall correctly, disk size that was set in the UI was 8gb but the > actual data that was transferred is around 1.5gb - so it might be just that > (probably a bug.. but unrelated to this topic). > > If you still need this workaround to be tested It's best if this needinfo > will move to its relevant bug. I still don't se an SSL error.. Can you please attach the proxy log? (In reply to Natalie Gavrielov from comment #13) > Scenario performed: > ------------------- > > 1. On a clean RHV hypervisor, performed hosted engine installation. > 2. Configured certificates for both Chrome and Firefox, with all required > (and > possible) permissions. > 3. Tried to upload an image via GUI (disks tab -> upload) > * Tried both RAW and Qcow2 v3 formats. > > Result: > ------- > > All attempts to upload failed (switching status to "Paused by the system"). > > > Important notes: > ---------------- > > 1. After manually restarting the ovirt-imageio-daemon upload image works! > 2. I looked for the restart of the ovirt-imageio-daemon, that host deploy > should do in ovirt-imageio-daemon.log and couldn't find any evidence for > this restart (only the one I performed manually). > 3. ovirt-imageio-proxy was not installed (never worked with hosted engine > before but I would guess it's not supposed to be installed..) ovirt-imageio-proxy should be installed during ovirt-hosted-engine-setup wizard. Have you installed the proxy manually? > > Builds used: > RHVM 4.1.1.2-0.1.el7 > ovirt-imageio-daemon-1.0.0-0.el7ev.noarch > > Will attach logs in no time. Created attachment 1265673 [details] image proxy, engine, server (In reply to Daniel Erez from comment #21) > I still don't se an SSL error.. Can you please attach the proxy log? Attached. (In reply to Daniel Erez from comment #22) > ovirt-imageio-proxy should be installed during ovirt-hosted-engine-setup > wizard. Have you installed the proxy manually? It was installed automatically - I missed it because the engine is nested.. and I'm not used it :\ (In reply to Natalie Gavrielov from comment #23) > Created attachment 1265673 [details] > image proxy, engine, server > > (In reply to Daniel Erez from comment #21) > > I still don't se an SSL error.. Can you please attach the proxy log? > > Attached. > > (In reply to Daniel Erez from comment #22) > > ovirt-imageio-proxy should be installed during ovirt-hosted-engine-setup > wizard. Have you installed the proxy manually? > > It was installed automatically - I missed it because the engine is nested.. > and I'm not used it :\ (So did you manually reinstalled the proxy then?) What are the exact installation steps of hosted engine? Also, please attach the ovirt-host-deploy logs.. Created attachment 1265686 [details] host-deploy logs (In reply to Daniel Erez from comment #25) > Also, please attach the ovirt-host-deploy logs.. Attached. (In reply to Daniel Erez from comment #24) > (In reply to Natalie Gavrielov from comment #23) > > Created attachment 1265673 [details] > > image proxy, engine, server > > > > (In reply to Daniel Erez from comment #21) > > > I still don't se an SSL error.. Can you please attach the proxy log? > > > > Attached. > > > > (In reply to Daniel Erez from comment #22) > > > ovirt-imageio-proxy should be installed during ovirt-hosted-engine-setup > wizard. Have you installed the proxy manually? > > > > It was installed automatically - I missed it because the engine is nested.. > > and I'm not used it :\ > > (So did you manually reinstalled the proxy then?) > What are the exact installation steps of hosted engine? No, I did not install / reinstall the proxy - it was installed by the hosted engine installation. I used a GUI installation provided by the Cockpit and rhevm-appliance, the flow is next-next-next-wait-done similar to this flow: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.0/html/self-hosted_engine_guide/deploying_self-hosted_engine_on_rhvh . more about Cockpit: https://bugzilla.redhat.com/show_bug.cgi?id=1190758 http://cockpit-project.org/ @Sandro - is a new rhvh build planned for 4.1.1 (which includes ovirt-host-deploy-1.6.3)? (In reply to Daniel Erez from comment #18) > @Simone - how can I check if the commit is included in this version? https://gerrit.ovirt.org/#/q/status:merged+project:ovirt-host-deploy+branch:ovirt-host-deploy-1.6 It should be there since ovirt-host-deploy-1.6.1 This should be fixed in 4.1.1-1, moving to MODIFIED If it was in 1.6.1, it's already in last week build. I've deployed a clean HE environment last week with Red Hat Virtualization Manager Version: 4.1.1.6 and then upgraded to 4.1.1.7-0.1.el7, then tried to upload two qcow2 images of rhel-guest-image, I've got the same result as described in https://bugzilla.redhat.com/show_bug.cgi?id=1411491#c13. ovirt-imageio-daemon.service was running on both my hosted-engine-hosts. Following https://bugzilla.redhat.com/show_bug.cgi?id=1411491#c22, ovirt-imageio-proxy is not automatically installed on hoststed-engine-hosts, not in 3.5, not in 3.6.11, not in 4.0.x, neither in latest 4.1.1.7-0.1.el7,so I had to install them manually. I've manually restarted ovirt-imageio-daemon on both hosts and retried image upload, for this time it wass successful. Can you check if ovirt-imageio-daemon should automatically get installed during ovirt-hosted-engine-setup please? 1)Deployed fresh and clean hosted-engine over NFS and added two data storage domains over NFS. 2)Added certificate http://<engine_url>/ovirt-engine/services/pki-resource?resource=ca-certificate&format=X509-PEM-CA using Firefox 52.0 (64-bit). 3)Tried to upload two qcow2 images and failed with "Paused by system" in UI. Components on engine: ovirt-imageio-proxy-setup-1.0.0-0.el7ev.noarch ovirt-imageio-proxy-1.0.0-0.el7ev.noarch rhev-guest-tools-iso-4.1-5.el7ev.noarch rhevm-branding-rhev-4.1.0-1.el7ev.noarch rhevm-4.1.1.7-0.1.el7.noarch rhevm-doc-4.1.0-3.el7ev.noarch rhevm-setup-plugins-4.1.1-1.el7ev.noarch rhevm-dependencies-4.1.1-1.el7ev.noarch Linux version 3.10.0-514.6.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Sat Dec 10 11:15:38 EST 2016 Linux 3.10.0-514.6.1.el7.x86_64 #1 SMP Sat Dec 10 11:15:38 EST 2016 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.3 (Maipo) Components on hosts: libvirt-client-2.0.0-10.el7_3.5.x86_64 ovirt-hosted-engine-setup-2.1.0.5-1.el7ev.noarch ovirt-host-deploy-1.6.3-1.el7ev.noarch ovirt-imageio-common-1.0.0-0.el7ev.noarch sanlock-3.4.0-1.el7.x86_64 ovirt-vmconsole-host-1.0.4-1.el7ev.noarch mom-0.5.9-1.el7ev.noarch vdsm-4.19.10.1-1.el7ev.x86_64 ovirt-hosted-engine-ha-2.1.0.5-1.el7ev.noarch ovirt-setup-lib-1.1.0-1.el7ev.noarch ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch ovirt-vmconsole-1.0.4-1.el7ev.noarch qemu-kvm-rhev-2.6.0-28.el7_3.9.x86_64 ovirt-imageio-daemon-1.0.0-0.el7ev.noarch Linux version 3.10.0-514.16.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Fri Mar 10 13:12:32 EST 2017 Linux 3.10.0-514.16.1.el7.x86_64 #1 SMP Fri Mar 10 13:12:32 EST 2017 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.3 (Maipo) As this is a clean and fresh deployment on pair of fresh hosts, I expect from the system to work out of the box. Please consider moving this bug back to assigned. Created attachment 1268681 [details]
Screenshot from 2017-04-04 17-38-20.png
Created attachment 1268701 [details]
sosreport-nsednev-he-4.scl.lab.tlv.redhat.com-20170404173939.tar.xz
Created attachment 1268702 [details]
sosreport-puma18.scl.lab.tlv.redhat.com-20170404173912.tar.xz
Created attachment 1268703 [details]
sosreport-puma18.scl.lab.tlv.redhat.com-20170404173912.tar.xz
Created attachment 1268705 [details]
sosreport-puma19.scl.lab.tlv.redhat.com-20170404173929.tar.xz
ovirt-host-deploy-1.6.3-1.el7ev.noarch is also installed on both hosts and engine: puma18 ~]# rpm -qa | grep host-deploy ovirt-host-deploy-1.6.3-1.el7ev.noarch nsednev-he-4 ~]# rpm -qa | grep host-deploy ovirt-host-deploy-java-1.6.3-1.el7ev.noarch ovirt-host-deploy-1.6.3-1.el7ev.noarch Following Nikolai's comment 34 and comment 40, reopening this issue. Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release. Hi Nikolai, Can you please attach host-deploy/engine-setup/engine/imageio-proxy/imageio-daemon logs? Can you please address the comment #43? Please reopen if you can provide the needed logs. Hi All, I wasn't around the past week, so was unable to address the issue. Anyway, since the environment is not mine, I don't have these logs. Yaniv Levi, I don't really understand why closing this bug - this issue was fixed but not yet verified, IMHO it cannot be closed as insufficient data, it's best to move this to ON_QA. 1. On a clean RHEL7.3 pair of hosts, deployed hosted engine. 2. Configured certificates for both Chrome and Firefox, with all required (and possible) permissions. 3. Tried to upload an image via GUI (disks tab -> upload) * Tried both RAW and Qcow2 v3 formats. Result: ------- All attempts to upload passed. I did not tested this on NGN (RHVH hosts), so I can't provide any information on how this is working there. Engine: nsednev-he-1 ~]# rpm -qa | grep host-deploy ovirt-host-deploy-1.6.5-1.el7ev.noarch ovirt-host-deploy-java-1.6.5-1.el7ev.noarch rhev-guest-tools-iso-4.1-5.el7ev.noarch rhevm-dependencies-4.1.1-1.el7ev.noarch rhevm-4.1.3.1-0.1.el7.noarch rhevm-setup-plugins-4.1.2-1.el7ev.noarch rhevm-branding-rhev-4.1.0-1.el7ev.noarch rhevm-doc-4.1.3-1.el7ev.noarch Linux version 3.10.0-514.16.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Fri Mar 10 13:12:32 EST 2017 Linux 3.10.0-514.16.1.el7.x86_64 #1 SMP Fri Mar 10 13:12:32 EST 2017 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.3 (Maipo) Hosts: alma04 ~]# rpm -qa | grep host-deploy ovirt-host-deploy-1.6.5-1.el7ev.noarch ovirt-vmconsole-1.0.4-1.el7ev.noarch vdsm-4.19.17-1.el7ev.x86_64 ovirt-host-deploy-1.6.5-1.el7ev.noarch ovirt-imageio-common-1.0.0-0.el7ev.noarch sanlock-3.4.0-1.el7.x86_64 ovirt-vmconsole-host-1.0.4-1.el7ev.noarch mom-0.5.9-1.el7ev.noarch qemu-kvm-rhev-2.6.0-28.el7_3.10.x86_64 ovirt-imageio-daemon-1.0.0-0.el7ev.noarch ovirt-hosted-engine-setup-2.1.2-2.el7ev.noarch ovirt-setup-lib-1.1.1-1.el7ev.noarch ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch libvirt-client-2.0.0-10.el7_3.9.x86_64 ovirt-hosted-engine-ha-2.1.1-1.el7ev.noarch Linux version 3.10.0-514.21.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Sat Apr 22 02:41:35 EDT 2017 Linux 3.10.0-514.21.1.el7.x86_64 #1 SMP Sat Apr 22 02:41:35 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.3 (Maipo) Regarding HE deployment on RHVH4.1, there are some problems at this moment which prevents me from trying reproduction on NGN hosts. Please see https://bugzilla.redhat.com/show_bug.cgi?id=1457684#c12 for more details. Reproduced on RHVH with these components: sanlock-3.4.0-1.el7.x86_64 ovirt-node-ng-nodectl-4.1.3-0.20170608.1.el7.noarch ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch mom-0.5.9-1.el7ev.noarch qemu-kvm-rhev-2.6.0-28.el7_3.10.x86_64 vdsm-4.19.18-1.el7ev.x86_64 ovirt-host-deploy-1.6.6-1.el7ev.noarch ovirt-imageio-common-1.0.0-0.el7ev.noarch ovirt-vmconsole-1.0.4-1.el7ev.noarch libvirt-client-2.0.0-10.el7_3.9.x86_64 ovirt-imageio-daemon-1.0.0-0.el7ev.noarch ovirt-hosted-engine-ha-2.1.2-1.el7ev.noarch ovirt-hosted-engine-setup-2.1.3-1.el7ev.noarch ovirt-setup-lib-1.1.3-1.el7ev.noarch ovirt-vmconsole-host-1.0.4-1.el7ev.noarch Linux version 3.10.0-514.21.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Sat Apr 22 02:41:35 EDT 2017 Linux 3.10.0-514.21.1.el7.x86_64 #1 SMP Sat Apr 22 02:41:35 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux release 7.3 Engine: rhevm-dependencies-4.1.1-1.el7ev.noarch rhevm-doc-4.1.3-1.el7ev.noarch rhevm-4.1.3.4-0.1.el7.noarch rhev-guest-tools-iso-4.1-5.el7ev.noarch rhevm-setup-plugins-4.1.2-1.el7ev.noarch rhevm-branding-rhev-4.1.0-2.el7ev.noarch Linux version 3.10.0-514.21.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Sat Apr 22 02:41:35 EDT 2017 Linux 3.10.0-514.21.1.el7.x86_64 #1 SMP Sat Apr 22 02:41:35 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.4 Beta (Maipo) Logs from the engine and the host being attached. Please also see the screencast as being attached. Moving back to assigned. Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release. Created attachment 1289077 [details]
screencast
Created attachment 1289078 [details]
sosreport from the engine
Created attachment 1289080 [details]
sosreport from puma18 the host
(In reply to Daniel Erez from comment #43) > Hi Nikolai, > > Can you please attach > host-deploy/engine-setup/engine/imageio-proxy/imageio-daemon logs? Just attaching the required log here, it is also available in sosreport from puma18: /var/log/ovirt-imageio-daemon/daemon.log Created attachment 1289096 [details]
ovirt-imageio-daemon daemon.log
@Nikolai - can you please also attach engine-setup/imageio-proxy/imageio-daemon? Not sure it's the same issue, is restarting the the imageio-daemon solves the issue? (In reply to Daniel Erez from comment #56) > @Nikolai - can you please also attach > engine-setup/imageio-proxy/imageio-daemon? > Not sure it's the same issue, is restarting the the imageio-daemon solves > the issue? Yes, it solves, all logs were already attached within the sosreport. @Nikolai - Can you please also upload image-proxy.log It should be located at /var/log/ovirt-imageio-proxy Created attachment 1289119 [details]
imageproxy log from an engine
Created attachment 1289120 [details]
image-proxy.log.2017-06-18 from an engine
(In reply to Nikolai Sednev from comment #60) > Created attachment 1289120 [details] > image-proxy.log.2017-06-18 from an engine The proxy seems empty. Are you connection to a different proxy perhaps? Is uploading currently works? (In reply to Daniel Erez from comment #61) > (In reply to Nikolai Sednev from comment #60) > > Created attachment 1289120 [details] > > image-proxy.log.2017-06-18 from an engine > > The proxy seems empty. Are you connection to a different proxy perhaps? Is > uploading currently works? Uploading currently fails with the same problem as before. Log really looks strange on engine: nsednev-he-4 ~]# cat /var/log/ovirt-imageio-proxy/image-proxy.log (MainThread) INFO 2017-06-19 07:44:58,282 image_proxy:35:root:(main) Server shut down, exiting (MainThread) INFO 2017-06-19 07:46:46,650 image_proxy:26:root:(main) Server started, successfully notified systemd (MainThread) INFO 2017-06-19 11:02:39,873 image_proxy:26:root:(main) Server started, successfully notified systemd Log on host puma18 looks as follows: puma18 ~]# cat /var/log/ovirt-imageio-daemon/daemon.log 2017-06-19 13:38:36,021 INFO (MainThread) [server] Starting (version 1.0.0) 2017-06-19 13:38:36,026 INFO (MainThread) [server] Ready for requests 2017-06-19 15:02:35,941 INFO (MainThread) [server] Received signal 15, shutting down 2017-06-19 15:02:37,655 INFO (MainThread) [server] Stopped 2017-06-19 15:02:37,905 INFO (MainThread) [server] Starting (version 1.0.0) 2017-06-19 15:02:37,996 INFO (MainThread) [server] Ready for requests 2017-06-19 15:21:46,410 INFO (ticket.server) [tickets] Adding ticket {u'uuid': u'56fc6346-acf4-497b-a61c-7cbff186bd20', u'ops': [u'write'], u'url': ParseResult(scheme=u'file', netloc=u'', path=u'/rhev/data-center/00000001-0001-0001-0001-000000000311/0b4c1b5c-7421-447b-911a-735af600c453/images/08e5a6e7-ed12-4b38-b0ce-2f029a45e29f/171b400a-9a1e-4f26-a3e3-a88872f0793b', params='', query='', fragment=''), 'expires': 4301179, u'timeout': 300, u'size': 421527552} 2017-06-19 15:21:46,411 INFO (ticket.server) [web] / - PUT /56fc6346-acf4-497b-a61c-7cbff186bd20 200 0 (0.00s) 2017-06-19 15:22:01,342 INFO (ticket.server) [tickets] Removing ticket 56fc6346-acf4-497b-a61c-7cbff186bd20 2017-06-19 15:22:01,342 INFO (ticket.server) [web] / - DELETE /56fc6346-acf4-497b-a61c-7cbff186bd20 204 0 (0.00s) 2017-06-19 18:16:13,374 INFO (ticket.server) [tickets] Adding ticket {u'uuid': u'eecb87d1-582d-4b2a-a46e-da59049e06fd', u'ops': [u'write'], u'url': ParseResult(scheme=u'file', netloc=u'', path=u'/rhev/data-center/00000001-0001-0001-0001-000000000311/0b4c1b5c-7421-447b-911a-735af600c453/images/a119b118-0b7d-4b25-87ec-caeaa36323f2/1284bf0f-bd51-43e4-83ef-138dbb271962', params='', query='', fragment=''), 'expires': 4311646, u'timeout': 300, u'size': 421527552} 2017-06-19 18:16:13,374 INFO (ticket.server) [web] / - PUT /eecb87d1-582d-4b2a-a46e-da59049e06fd 200 0 (0.00s) 2017-06-19 18:16:27,581 INFO (ticket.server) [tickets] Removing ticket eecb87d1-582d-4b2a-a46e-da59049e06fd 2017-06-19 18:16:27,582 INFO (ticket.server) [web] / - DELETE /eecb87d1-582d-4b2a-a46e-da59049e06fd 204 0 (0.00s) Created attachment 1289138 [details]
screencast
I've tried to upload an image using you env with FireFox 53 and it seems to work fine. Which browser version are you using? Did you restart the browser after importing the certificate? Can you please try to upload from another machine (With latest FireFox) in order to understand whether it's a browser issue indeed. I was using: Firefox 52.1.0 (64-bit). I did restarted browser, tried to clean also cookies and history, as well as restarting my laptop. Nothing helped, upload being paused. Same thing happened also for Google Chrome. Version 59.0.3071.86 (Official Build) (64-bit)  Returning to ON_QA as I've discovered that in my case somehow the issue was caused by stale CA certificate. Although I've installed a new and fresh CA certificate for new environment, somehow an old CA certificate was used instead of new during image upload and was failing to get uploaded. Once I've manually removed both old and new CA certificates and then manually re-added CA certificate, then upload succeeded. Works for me on cleanly deployed HE environment over NFS, on pair of RHVH4.1(based on el7.3) aka NGN hosts, with two NFS data storage domains. Components on hosts: sanlock-3.4.0-1.el7.x86_64 ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch ovirt-hosted-engine-ha-2.1.0.6-1.el7ev.noarch ovirt-imageio-common-1.0.0-0.el7ev.noarch ovirt-setup-lib-1.1.0-1.el7ev.noarch ovirt-vmconsole-1.0.4-1.el7ev.noarch ovirt-node-ng-nodectl-4.1.1-0.20170404.0.el7.noarch ovirt-vmconsole-host-1.0.4-1.el7ev.noarch libvirt-client-2.0.0-10.el7_3.5.x86_64 mom-0.5.9-1.el7ev.noarch ovirt-imageio-daemon-1.0.0-0.el7ev.noarch ovirt-hosted-engine-setup-2.1.0.6-1.el7ev.noarch qemu-kvm-rhev-2.6.0-28.el7_3.9.x86_64 vdsm-4.19.15-1.el7ev.x86_64 ovirt-host-deploy-1.6.5-1.el7ev.noarch Linux version 3.10.0-514.21.2.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Sun May 28 17:08:21 EDT 2017 Linux 3.10.0-514.21.2.el7.x86_64 #1 SMP Sun May 28 17:08:21 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux release 7.3 Components on Engine: rhevm-dependencies-4.1.1-1.el7ev.noarch rhevm-doc-4.1.3-1.el7ev.noarch rhevm-4.1.3.4-0.1.el7.noarch rhev-guest-tools-iso-4.1-5.el7ev.noarch rhevm-setup-plugins-4.1.2-1.el7ev.noarch rhevm-branding-rhev-4.1.0-2.el7ev.noarch Linux version 3.10.0-514.21.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Sat Apr 22 02:41:35 EDT 2017 Linux 3.10.0-514.21.1.el7.x86_64 #1 SMP Sat Apr 22 02:41:35 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.4 Beta (Maipo) I've deleted an old CA certificate of the engine and then got a new after deployed a fresh HE environment. Uploaded ISO image successfully. Moving this bug to verified as it works for me on RHVH4.1(based on el7.3) and also on RHEL7.3. Attaching a successfully upload screencast here: https://drive.google.com/a/redhat.com/file/d/0B85BEaDBcF88cFpnc0N0d0NrVlk/view?usp=sharing Created attachment 1290023 [details]
Screenshot from 2017-06-21 14-24-42.png
|