Bug 1480902 - image_handler:186:root:(make_imaged_request) Failed communicating with vdsm-imaged: A Connection error occurred. ovirt-imageio-proxy
image_handler:186:root:(make_imaged_request) Failed communicating with vdsm-i...
Status: CLOSED CURRENTRELEASE
Product: cockpit-ovirt
Classification: oVirt
Component: Gdeploy (Show other bugs)
0.10.7-0.0.23
Unspecified Unspecified
high Severity urgent
: ovirt-4.1.6
: 0.11.0
Assigned To: Gobinda Das
RamaKasturi
:
Depends On:
Blocks: RHHI-1.1-In-Flight-BZs
  Show dependency treegraph
 
Reported: 2017-08-12 15:01 EDT by ldomb
Modified: 2017-09-19 06:04 EDT (History)
20 users (show)

See Also:
Fixed In Version: cockpit-ovirt-0.10.8-2.0.ovirt41.el7ev
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-09-19 06:04:00 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Gluster
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
dchaplyg: needinfo-
sabose: ovirt‑4.1?
sabose: planning_ack?
rule-engine: devel_ack+
knarra: testing_ack+


Attachments (Terms of Use)
host-deploy log (387.52 KB, text/plain)
2017-08-13 10:15 EDT, Daniel Erez
no flags Details
screenshot of the uploaded disk image (190.19 KB, image/png)
2017-09-12 06:47 EDT, RamaKasturi
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 81087 master MERGED Added port 54322 in cockpit conf file, so that gdeploy can open it 2017-08-30 07:16 EDT
oVirt gerrit 81090 ovirt-4.1 MERGED Added port 54322 in cockpit conf file, so that gdeploy can open it 2017-08-30 08:18 EDT

  None (edit)
Description ldomb 2017-08-12 15:01:45 EDT
Description of problem:
Not able to upload images through the image tab in an HCI deployment

Version-Release number of selected component (if applicable):
4.1.4.2-0.1.el7

How reproducible:
Install a 3 node HCI deployment. Don't change any settings and just use what we ship. Once the cluster is up and you followed all the instructions try to add an image via web UI to the vmstore on gluster. 

Steps to Reproduce:
1. Click on the Disk tab and click upload
2. choose the rhel74.qcow2 image from the redhat portal
3. hit Ok

Actual results:
Image does not upload, status gets to upload stalled


Expected results:
image should upload

Additional info:
journalctl -xeu ovirt-imageio-proxy
                                                               File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
                                                                 r = adapter.send(request, **kwargs)
                                                               File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
                                                                 raise ConnectionError(err, request=request)
                                                             ConnectionError: ('Connection aborted.', error(113, 'No route to host'))
Aug 12 14:51:17 engine1.local.jj.com ovirt-imageio-proxy[17186]: 172.31.1.72 - - [12/Aug/2017 14:51:17] "PUT /images/7a3a89ac-3a9d-44b9-b9a8-6be770854808 HTTP/1.1" 503 215
Aug 12 14:51:17 engine1.local.jj.com ovirt-imageio-p[17186]: ovirt-imageio-proxy web ERROR 172.31.1.72 - PUT  503 215 (0.00s)
                                                             Traceback (most recent call last):
                                                               File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/web.py", line 48, in __call__
                                                                 resp = self.dispatch(request)
                                                               File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/web.py", line 73, in dispatch
                                                                 return method(*match.groups())
                                                               File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/http_helper.py", line 88, in wrapper
                                                                 ret = func(self, *args)
                                                               File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/http_helper.py", line 59, in wrapper
                                                                 ret = func(self, *args)
                                                               File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_handler.py", line 75, in put
                                                                 return self.send_data(self.request)
                                                               File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_handler.py", line 116, in send_data
                                                                 request.method, imaged_url, headers, body, stream)
                                                               File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_handler.py", line 187, in make_imaged_request
                                                                 raise exc.HTTPServiceUnavailable(s)
                                                             HTTPServiceUnavailable: Failed communicating with vdsm-imaged: A Connection error occurred.
Aug 12 14:51:17 engine1.local.jj.com ovirt-imageio-p[17186]: ovirt-imageio-proxy root ERROR Failed communicating with vdsm-imaged: A Connection error occurred.
                                                             Traceback (most recent call last):
                                                               File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_handler.py", line 177, in make_imaged_request
                                                                 timeout=timeout, stream=stream)
                                                               File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
                                                                 r = adapter.send(request, **kwargs)
                                                               File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
                                                                 raise ConnectionError(err, request=request)
                                                             ConnectionError: ('Connection aborted.', error(113, 'No route to host'))
Aug 12 14:51:17 engine1.local.jj.com ovirt-imageio-p[17186]: ovirt-imageio-proxy web ERROR 172.31.1.72 - PUT  503 215 (0.00s)
                                                             Traceback (most recent call last):
                                                               File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/web.py", line 48, in __call__
                                                                 resp = self.dispatch(request)
                                                               File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/web.py", line 73, in dispatch
                                                                 return method(*match.groups())
                                                               File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/http_helper.py", line 88, in wrapper
                                                                 ret = func(self, *args)
                                                               File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/http_helper.py", line 59, in wrapper
                                                                 ret = func(self, *args)
                                                               File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_handler.py", line 75, in put
                                                                 return self.send_data(self.request)
                                                               File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_handler.py", line 116, in send_data
                                                                 request.method, imaged_url, headers, body, stream)
                                                               File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_handler.py", line 187, in make_imaged_request
                                                                 raise exc.HTTPServiceUnavailable(s)
                                                             HTTPServiceUnavailable: Failed communicating with vdsm-imaged: A Connection error occurred.




image-proxy.log:
(Thread-1  ) INFO 2017-08-12 14:51:07,239 web:89:web:(log_response) 172.31.1.72 - OPTIONS /7a3a89ac-3a9d-44b9-b9a8-6be770854808 204 0 (0.00s)
(Thread-2  ) INFO 2017-08-12 14:51:07,350 session:280:root:(_decode_proxy_ticket) Proxy ticket valid: {u'iat': 1502563861, u'imaged-uri': u'https://hosted-engine1.local.jj.com:54322', u'nbf': 1502563861, u'exp': 1502567461, u'transfer-ticket': u'7a3a89ac-3a9d-44b9-b9a8-6be770854808'}
(Thread-2  ) INFO 2017-08-12 14:51:07,351 session:212:root:(_create_update_session) Established session: expiration: '1502567461', imaged-host-uri: 'https://hosted-engine1.local.jj.com:54322', proxy-ticket: 'eyJzYWx0IjoiMDFjSTRVUVVJVlk9IiwiZGF0YSI6IntcbiAgXCJuYmZcIiA6...IjIwMTcwODEyMTg1MTAxIiwidmFsaWRUbyI6IjIwMTcwODEyMTk1MTAxIn0=', session-id: '4747c52b-fc8b-454b-84bd-0a8d1d38b275', transfer-ticket: '7a3a89ac-3a9d-44b9-b9a8-6be770854808'
(Thread-2  ) INFO 2017-08-12 14:51:07,362 connectionpool:735:requests.packages.urllib3.connectionpool:(_new_conn) Starting new HTTPS connection (1): hosted-engine1.local.jj.com
(Thread-2  ) ERROR 2017-08-12 14:51:07,367 image_handler:186:root:(make_imaged_request) Failed communicating with vdsm-imaged: A Connection error occurred.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_handler.py", line 177, in make_imaged_request
    timeout=timeout, stream=stream)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', error(113, 'No route to host'))
(Thread-2  ) ERROR 2017-08-12 14:51:07,367 web:89:web:(log_response) 172.31.1.72 - PUT  503 215 (0.02s)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/web.py", line 48, in __call__
    resp = self.dispatch(request)
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/web.py", line 73, in dispatch
    return method(*match.groups())
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/http_helper.py", line 88, in wrapper
    ret = func(self, *args)
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/http_helper.py", line 59, in wrapper
    ret = func(self, *args)
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_handler.py", line 75, in put
    return self.send_data(self.request)
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_handler.py", line 116, in send_data
    request.method, imaged_url, headers, body, stream)
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_handler.py", line 187, in make_imaged_request
    raise exc.HTTPServiceUnavailable(s)
HTTPServiceUnavailable: Failed communicating with vdsm-imaged: A Connection error occurred.
(Thread-3  ) INFO 2017-08-12 14:51:07,503 session:280:root:(_decode_proxy_ticket) Proxy ticket valid: {u'iat': 1502563861, u'imaged-uri': u'https://hosted-engine1.local.jj.com:54322', u'nbf': 1502563861, u'exp': 1502567461, u'transfer-ticket': u'7a3a89ac-3a9d-44b9-b9a8-6be770854808'}
(Thread-3  ) INFO 2017-08-12 14:51:07,503 session:212:root:(_create_update_session) Established session: expiration: '1502567461', imaged-host-uri: 'https://hosted-engine1.local.jj.com:54322', proxy-ticket: 'eyJzYWx0IjoiMDFjSTRVUVVJVlk9IiwiZGF0YSI6IntcbiAgXCJuYmZcIiA6...IjIwMTcwODEyMTg1MTAxIiwidmFsaWRUbyI6IjIwMTcwODEyMTk1MTAxIn0=', session-id: 'cdc709f1-bfc7-420a-95fc-b90797168f5d', transfer-ticket: '7a3a89ac-3a9d-44b9-b9a8-6be770854808'
(Thread-3  ) INFO 2017-08-12 14:51:07,505 connectionpool:735:requests.packages.urllib3.connectionpool:(_new_conn) Starting new HTTPS connection (1): hosted-engine1.local.jj.com
(Thread-3  ) ERROR 2017-08-12 14:51:07,508 image_handler:186:root:(make_imaged_request) Failed communicating with vdsm-imaged: A Connection error occurred.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_handler.py", line 177, in make_imaged_request
    timeout=timeout, stream=stream)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', error(113, 'No route to host'))
(Thread-3  ) ERROR 2017-08-12 14:51:07,508 web:89:web:(log_response) 172.31.1.72 - PUT  503 215 (0.00s)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/web.py", line 48, in __call__
    resp = self.dispatch(request)
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/web.py", line 73, in dispatch
    return method(*match.groups())
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/http_helper.py", line 88, in wrapper
    ret = func(self, *args)
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/http_helper.py", line 59, in wrapper
    ret = func(self, *args)
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_handler.py", line 75, in put
    return self.send_data(self.request)
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_handler.py", line 116, in send_data
    request.method, imaged_url, headers, body, stream)
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_handler.py", line 187, in make_imaged_request
    raise exc.HTTPServiceUnavailable(s)
HTTPServiceUnavailable: Failed communicating with vdsm-imaged: A Connection error occurred.
(Thread-4  ) INFO 2017-08-12 14:51:10,594 web:89:web:(log_response) 172.31.1.72 - OPTIONS /7a3a89ac-3a9d-44b9-b9a8-6be770854808 204 0 (0.00s)
(Thread-5  ) INFO 2017-08-12 14:51:10,722 session:280:root:(_decode_proxy_ticket) Proxy ticket valid: {u'iat': 1502563861, u'imaged-uri': u'https://hosted-engine1.local.jj.com:54322', u'nbf': 1502563861, u'exp': 1502567461, u'transfer-ticket': u'7a3a89ac-3a9d-44b9-b9a8-6be770854808'}
(Thread-5  ) INFO 2017-08-12 14:51:10,722 session:212:root:(_create_update_session) Established session: expiration: '1502567461', imaged-host-uri: 'https://hosted-engine1.local.jj.com:54322', proxy-ticket: 'eyJzYWx0IjoiMDFjSTRVUVVJVlk9IiwiZGF0YSI6IntcbiAgXCJuYmZcIiA6...IjIwMTcwODEyMTg1MTAxIiwidmFsaWRUbyI6IjIwMTcwODEyMTk1MTAxIn0=', session-id: '5231ca77-8cd5-4de0-a0c7-f2046ff08b47', transfer-ticket: '7a3a89ac-3a9d-44b9-b9a8-6be770854808'
(Thread-5  ) INFO 2017-08-12 14:51:10,723 connectionpool:735:requests.packages.urllib3.connectionpool:(_new_conn) Starting new HTTPS connection (1): hosted-engine1.local.jj.com
(Thread-5  ) ERROR 2017-08-12 14:51:10,726 image_handler:186:root:(make_imaged_request) Failed communicating with vdsm-imaged: A Connection error occurred.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_handler.py", line 177, in make_imaged_request
    timeout=timeout, stream=stream)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', error(113, 'No route to host'))
(Thread-5  ) ERROR 2017-08-12 14:51:10,727 web:89:web:(log_response) 172.31.1.72 - PUT  503 215 (0.01s)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/web.py", line 48, in __call__
    resp = self.dispatch(request)
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/web.py", line 73, in dispatch
    return method(*match.groups())
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/http_helper.py", line 88, in wrapper
    ret = func(self, *args)
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/http_helper.py", line 59, in wrapper
    ret = func(self, *args)
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_handler.py", line 75, in put
    return self.send_data(self.request)
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_handler.py", line 116, in send_data
    request.method, imaged_url, headers, body, stream)
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_handler.py", line 187, in make_imaged_request
    raise exc.HTTPServiceUnavailable(s)
HTTPServiceUnavailable: Failed communicating with vdsm-imaged: A Connection error occurred.
(Thread-6  ) INFO 2017-08-12 14:51:14,016 web:89:web:(log_response) 172.31.1.72 - OPTIONS /7a3a89ac-3a9d-44b9-b9a8-6be770854808 204 0 (0.00s)
(Thread-7  ) INFO 2017-08-12 14:51:14,159 session:280:root:(_decode_proxy_ticket) Proxy ticket valid: {u'iat': 1502563861, u'imaged-uri': u'https://hosted-engine1.local.jj.com:54322', u'nbf': 1502563861, u'exp': 1502567461, u'transfer-ticket': u'7a3a89ac-3a9d-44b9-b9a8-6be770854808'}
(Thread-7  ) INFO 2017-08-12 14:51:14,160 session:212:root:(_create_update_session) Established session: expiration: '1502567461', imaged-host-uri: 'https://hosted-engine1.local.jj.com:54322', proxy-ticket: 'eyJzYWx0IjoiMDFjSTRVUVVJVlk9IiwiZGF0YSI6IntcbiAgXCJuYmZcIiA6...IjIwMTcwODEyMTg1MTAxIiwidmFsaWRUbyI6IjIwMTcwODEyMTk1MTAxIn0=', session-id: '2463274e-491a-4605-bcfd-572e8ac5c266', transfer-ticket: '7a3a89ac-3a9d-44b9-b9a8-6be770854808'
(Thread-7  ) INFO 2017-08-12 14:51:14,161 connectionpool:735:requests.packages.urllib3.connectionpool:(_new_conn) Starting new HTTPS connection (1): hosted-engine1.local.jj.com
(Thread-7  ) ERROR 2017-08-12 14:51:14,164 image_handler:186:root:(make_imaged_request) Failed communicating with vdsm-imaged: A Connection error occurred.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_handler.py", line 177, in make_imaged_request
    timeout=timeout, stream=stream)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', error(113, 'No route to host'))
(Thread-7  ) ERROR 2017-08-12 14:51:14,164 web:89:web:(log_response) 172.31.1.72 - PUT  503 215 (0.01s)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/web.py", line 48, in __call__
    resp = self.dispatch(request)
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/web.py", line 73, in dispatch
    return method(*match.groups())
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/http_helper.py", line 88, in wrapper
    ret = func(self, *args)
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/http_helper.py", line 59, in wrapper
    ret = func(self, *args)
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_handler.py", line 75, in put
    return self.send_data(self.request)
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_handler.py", line 116, in send_data
    request.method, imaged_url, headers, body, stream)
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_handler.py", line 187, in make_imaged_request
    raise exc.HTTPServiceUnavailable(s)
HTTPServiceUnavailable: Failed communicating with vdsm-imaged: A Connection error occurred.
^C
Comment 1 ldomb 2017-08-12 17:30:04 EDT
Firewall port 54322 is not opened by default with the HCI installation therefor the upload will fail. Not sure what bugzilla I have to update here
Comment 2 Daniel Erez 2017-08-13 02:56:57 EDT
(In reply to ldomb from comment #1)
> Firewall port 54322 is not opened by default with the HCI installation
> therefor the upload will fail. Not sure what bugzilla I have to update here

We've recently added this requirement to 'Virtualization Host Firewall Requirements' section of the 'Planning And Prerequisites Guide' of 4.1.

Is it the only issue here? I.e. Is opening the port solves it?
Comment 4 Daniel Erez 2017-08-13 03:02:42 EDT
(In reply to Daniel Erez from comment #2)
> (In reply to ldomb from comment #1)
> > Firewall port 54322 is not opened by default with the HCI installation
> > therefor the upload will fail. Not sure what bugzilla I have to update here
> 
> We've recently added this requirement to 'Virtualization Host Firewall
> Requirements' section of the 'Planning And Prerequisites Guide' of 4.1.

@Tahlia - do we have a BZ for the 4.1 documentation update?

> 
> Is it the only issue here? I.e. Is opening the port solves it?
Comment 5 Daniel Erez 2017-08-13 09:42:13 EDT
(In reply to Daniel Erez from comment #2)
> (In reply to ldomb from comment #1)
> > Firewall port 54322 is not opened by default with the HCI installation
> > therefor the upload will fail. Not sure what bugzilla I have to update here
> 
> We've recently added this requirement to 'Virtualization Host Firewall
> Requirements' section of the 'Planning And Prerequisites Guide' of 4.1.
> 
> Is it the only issue here? I.e. Is opening the port solves it?

A few more questions:
* Have you previously performed update on that host? 
* If yes, from which version?
* Can you please attach the host-deploy log of the relevant host?
* Also, please attach the output of 'iptables -L'
Comment 6 ldomb 2017-08-13 09:58:22 EDT
* Have you previously performed update on that host? 
No
* If yes, from which version?
No
* Can you please attach the host-deploy log of the relevant host?
* Also, please attach the output of 'iptables -L'
iptables -L 
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            
INPUT_direct  all  --  anywhere             anywhere            
INPUT_ZONES_SOURCE  all  --  anywhere             anywhere            
INPUT_ZONES  all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere             ctstate INVALID
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            
FORWARD_direct  all  --  anywhere             anywhere            
FORWARD_IN_ZONES_SOURCE  all  --  anywhere             anywhere            
FORWARD_IN_ZONES  all  --  anywhere             anywhere            
FORWARD_OUT_ZONES_SOURCE  all  --  anywhere             anywhere            
FORWARD_OUT_ZONES  all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere             ctstate INVALID
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
OUTPUT_direct  all  --  anywhere             anywhere            

Chain FORWARD_IN_ZONES (1 references)
target     prot opt source               destination         
FWDI_public  all  --  anywhere             anywhere            [goto] 

Chain FORWARD_IN_ZONES_SOURCE (1 references)
target     prot opt source               destination         

Chain FORWARD_OUT_ZONES (1 references)
target     prot opt source               destination         
FWDO_public  all  --  anywhere             anywhere            [goto] 

Chain FORWARD_OUT_ZONES_SOURCE (1 references)
target     prot opt source               destination         

Chain FORWARD_direct (1 references)
target     prot opt source               destination         

Chain FWDI_public (1 references)
target     prot opt source               destination         
FWDI_public_log  all  --  anywhere             anywhere            
FWDI_public_deny  all  --  anywhere             anywhere            
FWDI_public_allow  all  --  anywhere             anywhere            
ACCEPT     icmp --  anywhere             anywhere            

Chain FWDI_public_allow (1 references)
target     prot opt source               destination         

Chain FWDI_public_deny (1 references)
target     prot opt source               destination         

Chain FWDI_public_log (1 references)
target     prot opt source               destination         

Chain FWDO_public (1 references)
target     prot opt source               destination         
FWDO_public_log  all  --  anywhere             anywhere            
FWDO_public_deny  all  --  anywhere             anywhere            
FWDO_public_allow  all  --  anywhere             anywhere            

Chain FWDO_public_allow (1 references)
target     prot opt source               destination         

Chain FWDO_public_deny (1 references)
target     prot opt source               destination         

Chain FWDO_public_log (1 references)
target     prot opt source               destination         

Chain INPUT_ZONES (1 references)
target     prot opt source               destination         
IN_public  all  --  anywhere             anywhere            [goto] 

Chain INPUT_ZONES_SOURCE (1 references)
target     prot opt source               destination         

Chain INPUT_direct (1 references)
target     prot opt source               destination         

Chain IN_public (1 references)
target     prot opt source               destination         
IN_public_log  all  --  anywhere             anywhere            
IN_public_deny  all  --  anywhere             anywhere            
IN_public_allow  all  --  anywhere             anywhere            
ACCEPT     icmp --  anywhere             anywhere            

Chain IN_public_allow (1 references)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:54321 ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpts:rfb:6923 ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpts:49152:49216 ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:sunrpc ctstate NEW
ACCEPT     udp  --  anywhere             anywhere             udp dpt:sunrpc ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:16509 ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:websm ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:24007 ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:24008 ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:24009 ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:38465 ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:38466 ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:38467 ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:38468 ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:38469 ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpts:49152:49664 ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:nfs ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:mountd ctstate NEW
ACCEPT     udp  --  anywhere             anywhere             udp dpt:mountd ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:5666 ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:16514 ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:54322 ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:rfb ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpts:rfb:6923 ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:54321 ctstate NEW
ACCEPT     udp  --  anywhere             anywhere             udp dpt:54322 ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:nfs ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:sunrpc ctstate NEW

Chain IN_public_deny (1 references)
target     prot opt source               destination         

Chain IN_public_log (1 references)
target     prot opt source               destination         

Chain OUTPUT_direct (1 references)
target     prot opt source               destination
Comment 7 Daniel Erez 2017-08-13 10:08:42 EDT
(In reply to ldomb from comment #6)
> * Have you previously performed update on that host? 
> No
> * If yes, from which version?
> No
> * Can you please attach the host-deploy log of the relevant host?
> * Also, please attach the output of 'iptables -L'
> iptables -L 
> Chain INPUT (policy ACCEPT)
> target     prot opt source               destination         
> ACCEPT     all  --  anywhere             anywhere             ctstate
> RELATED,ESTABLISHED
> ACCEPT     all  --  anywhere             anywhere            
> INPUT_direct  all  --  anywhere             anywhere            
> INPUT_ZONES_SOURCE  all  --  anywhere             anywhere            
> INPUT_ZONES  all  --  anywhere             anywhere            
> DROP       all  --  anywhere             anywhere             ctstate INVALID
> REJECT     all  --  anywhere             anywhere             reject-with
> icmp-host-prohibited
> 
> Chain FORWARD (policy ACCEPT)
> target     prot opt source               destination         
> ACCEPT     all  --  anywhere             anywhere             ctstate
> RELATED,ESTABLISHED
> ACCEPT     all  --  anywhere             anywhere            
> FORWARD_direct  all  --  anywhere             anywhere            
> FORWARD_IN_ZONES_SOURCE  all  --  anywhere             anywhere            
> FORWARD_IN_ZONES  all  --  anywhere             anywhere            
> FORWARD_OUT_ZONES_SOURCE  all  --  anywhere             anywhere            
> FORWARD_OUT_ZONES  all  --  anywhere             anywhere            
> DROP       all  --  anywhere             anywhere             ctstate INVALID
> REJECT     all  --  anywhere             anywhere             reject-with
> icmp-host-prohibited
> 
> Chain OUTPUT (policy ACCEPT)
> target     prot opt source               destination         
> OUTPUT_direct  all  --  anywhere             anywhere            
> 
> Chain FORWARD_IN_ZONES (1 references)
> target     prot opt source               destination         
> FWDI_public  all  --  anywhere             anywhere            [goto] 
> 
> Chain FORWARD_IN_ZONES_SOURCE (1 references)
> target     prot opt source               destination         
> 
> Chain FORWARD_OUT_ZONES (1 references)
> target     prot opt source               destination         
> FWDO_public  all  --  anywhere             anywhere            [goto] 
> 
> Chain FORWARD_OUT_ZONES_SOURCE (1 references)
> target     prot opt source               destination         
> 
> Chain FORWARD_direct (1 references)
> target     prot opt source               destination         
> 
> Chain FWDI_public (1 references)
> target     prot opt source               destination         
> FWDI_public_log  all  --  anywhere             anywhere            
> FWDI_public_deny  all  --  anywhere             anywhere            
> FWDI_public_allow  all  --  anywhere             anywhere            
> ACCEPT     icmp --  anywhere             anywhere            
> 
> Chain FWDI_public_allow (1 references)
> target     prot opt source               destination         
> 
> Chain FWDI_public_deny (1 references)
> target     prot opt source               destination         
> 
> Chain FWDI_public_log (1 references)
> target     prot opt source               destination         
> 
> Chain FWDO_public (1 references)
> target     prot opt source               destination         
> FWDO_public_log  all  --  anywhere             anywhere            
> FWDO_public_deny  all  --  anywhere             anywhere            
> FWDO_public_allow  all  --  anywhere             anywhere            
> 
> Chain FWDO_public_allow (1 references)
> target     prot opt source               destination         
> 
> Chain FWDO_public_deny (1 references)
> target     prot opt source               destination         
> 
> Chain FWDO_public_log (1 references)
> target     prot opt source               destination         
> 
> Chain INPUT_ZONES (1 references)
> target     prot opt source               destination         
> IN_public  all  --  anywhere             anywhere            [goto] 
> 
> Chain INPUT_ZONES_SOURCE (1 references)
> target     prot opt source               destination         
> 
> Chain INPUT_direct (1 references)
> target     prot opt source               destination         
> 
> Chain IN_public (1 references)
> target     prot opt source               destination         
> IN_public_log  all  --  anywhere             anywhere            
> IN_public_deny  all  --  anywhere             anywhere            
> IN_public_allow  all  --  anywhere             anywhere            
> ACCEPT     icmp --  anywhere             anywhere            
> 
> Chain IN_public_allow (1 references)
> target     prot opt source               destination         
> ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:54321
> ctstate NEW
> ACCEPT     tcp  --  anywhere             anywhere             tcp
> dpts:rfb:6923 ctstate NEW
> ACCEPT     tcp  --  anywhere             anywhere             tcp
> dpts:49152:49216 ctstate NEW
> ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:sunrpc
> ctstate NEW
> ACCEPT     udp  --  anywhere             anywhere             udp dpt:sunrpc
> ctstate NEW
> ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:16509
> ctstate NEW
> ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:websm
> ctstate NEW
> ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:24007
> ctstate NEW
> ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:24008
> ctstate NEW
> ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:24009
> ctstate NEW
> ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:38465
> ctstate NEW
> ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:38466
> ctstate NEW
> ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:38467
> ctstate NEW
> ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:38468
> ctstate NEW
> ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:38469
> ctstate NEW
> ACCEPT     tcp  --  anywhere             anywhere             tcp
> dpts:49152:49664 ctstate NEW
> ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:nfs
> ctstate NEW
> ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh
> ctstate NEW
> ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:mountd
> ctstate NEW
> ACCEPT     udp  --  anywhere             anywhere             udp dpt:mountd
> ctstate NEW
> ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:5666
> ctstate NEW
> ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:16514
> ctstate NEW
> ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:54322
> ctstate NEW

So this was added after deploying the host?
Do you have another host you can check whether the rule exist?
Can you please attach host-deploy log, as the rule should be added as part of the deployment.


> ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:rfb
> ctstate NEW
> ACCEPT     tcp  --  anywhere             anywhere             tcp
> dpts:rfb:6923 ctstate NEW
> ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:54321
> ctstate NEW
> ACCEPT     udp  --  anywhere             anywhere             udp dpt:54322
> ctstate NEW
> ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:nfs
> ctstate NEW
> ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:sunrpc
> ctstate NEW
> 
> Chain IN_public_deny (1 references)
> target     prot opt source               destination         
> 
> Chain IN_public_log (1 references)
> target     prot opt source               destination         
> 
> Chain OUTPUT_direct (1 references)
> target     prot opt source               destination
Comment 8 Daniel Erez 2017-08-13 10:15 EDT
Created attachment 1312703 [details]
host-deploy log
Comment 9 Daniel Erez 2017-08-13 10:44:42 EDT
According to the host-deploy log[1], iptables wasn't enabled during the deploy.
So it seems like an issue of the HCI installer host deployment automation.

@Denis - is it a known issue?

[1]
2017-08-11 13:19:29 DEBUG otopi.context context.dumpEnvironment:770 ENV NETWORK/iptablesEnable=bool:'False'
2017-08-11 13:19:29 DEBUG otopi.context context.dumpEnvironment:770 ENV NETWORK/iptablesRules=NoneType:'None'
Comment 10 Tahlia Richardson 2017-08-13 20:25:23 EDT
(In reply to Daniel Erez from comment #4)
> (In reply to Daniel Erez from comment #2)
> > (In reply to ldomb from comment #1)
> > > Firewall port 54322 is not opened by default with the HCI installation
> > > therefor the upload will fail. Not sure what bugzilla I have to update here
> > 
> > We've recently added this requirement to 'Virtualization Host Firewall
> > Requirements' section of the 'Planning And Prerequisites Guide' of 4.1.
> 
> @Tahlia - do we have a BZ for the 4.1 documentation update?

There's an individual BZ for the port: BZ#1467153
It's on release_pending, waiting for the completion of BZ#1450254, which is very close to done. The planned publication date is August 23. 

> 
> > 
> > Is it the only issue here? I.e. Is opening the port solves it?
Comment 11 Denis Chaplygin 2017-08-14 03:43:12 EDT
@Daniel - No, it is not yet a known issue. Could you please provide more details?
Comment 12 Daniel Erez 2017-08-14 04:10:40 EDT
(In reply to Denis Chaplygin from comment #11)
> @Daniel - No, it is not yet a known issue. Could you please provide more
> details?

Sure. imageio-daemon service on host requires an iptables rule to open port 54322 (which is opened by default on regular host deploy by engine). However, according to the host-deploy logs[1] of HCI env, I see that no rules are passed. The question is what's the difference in host deployment between HCI and regular envs.


[1]
2017-08-11 13:19:29 DEBUG otopi.context context.dumpEnvironment:770 ENV NETWORK/iptablesEnable=bool:'False'
2017-08-11 13:19:29 DEBUG otopi.context context.dumpEnvironment:770 ENV NETWORK/iptablesRules=NoneType:'None'
Comment 13 ldomb 2017-08-14 08:11:06 EDT
I also realized on a default HCI install the ovirt-imageio-daemon is not enabled as well. 

ovirt-imageio-daemon.service                  disabled
Comment 14 Daniel Erez 2017-08-14 08:29:15 EDT
(In reply to ldomb from comment #13)
> I also realized on a default HCI install the ovirt-imageio-daemon is not
> enabled as well. 
> 
> ovirt-imageio-daemon.service                  disabled

That's actually expected as host-deploy/vdsm starts the service.
Comment 15 Daniel Erez 2017-08-16 03:06:40 EDT
(In reply to Daniel Erez from comment #12)
> (In reply to Denis Chaplygin from comment #11)
> > @Daniel - No, it is not yet a known issue. Could you please provide more
> > details?
> 
> Sure. imageio-daemon service on host requires an iptables rule to open port
> 54322 (which is opened by default on regular host deploy by engine).
> However, according to the host-deploy logs[1] of HCI env, I see that no
> rules are passed. The question is what's the difference in host deployment
> between HCI and regular envs.
> 
> 
> [1]
> 2017-08-11 13:19:29 DEBUG otopi.context context.dumpEnvironment:770 ENV
> NETWORK/iptablesEnable=bool:'False'
> 2017-08-11 13:19:29 DEBUG otopi.context context.dumpEnvironment:770 ENV
> NETWORK/iptablesRules=NoneType:'None'

Moving the bug, as it seems the issue is relevant only for Grafton deploy process (the iptables rules are opened correctly on regular deployment).
Comment 17 RamaKasturi 2017-08-16 06:16:15 EDT
changing the product and component to ovirt-cockpit & gdeploy as the fix needs to be in ovirt-cockpit -> gdeploy.

gdeploy opens the ports what ever is present in the conf file, so conf file has to be updated with the correct ports so that gdeploy can open it.
Comment 18 Allon Mureinik 2017-08-31 08:17:18 EDT
Gobinda, both patches attached here are merged. Should this BZ be moved to MODIFIED, or are we pending anything else?
Comment 19 RamaKasturi 2017-09-12 06:34:15 EDT
Verified and works fine with build cockpit-ovirt-dashboard-0.10.8-2.0.ovirt41.el7ev.noarch.

I see that the port required to upload disk is being opened as part of cockpit-gdeploy.

gdeployConfig.conf file:
=======================================================
ports=111/tcp,2049/tcp,54321/tcp,5900/tcp,5900-6923/tcp,5666/tcp,16514/tcp,54322/tcp - port 54322 is added

iptables -L output from the host:
================================================
[root@rhsqa-grafton1 ~]# iptables -L | grep 54322
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:54322 ctstate NEW

Below are the steps followed to verify the bug:
===================================================
1) Install ovirt-imageio-proxy on the engine side if not already installed.
2) check for the value in DB by executing the commands below.
# su - postgres
Comment 20 RamaKasturi 2017-09-12 06:34:41 EDT
Please ignore comment 19
Comment 21 RamaKasturi 2017-09-12 06:46:09 EDT
Verified and works fine with build cockpit-ovirt-dashboard-0.10.8-2.0.ovirt41.el7ev.noarch.

I see that the port required to upload disk is being opened as part of cockpit-gdeploy.

gdeployConfig.conf file:
=======================================================
ports=111/tcp,2049/tcp,54321/tcp,5900/tcp,5900-6923/tcp,5666/tcp,16514/tcp,54322/tcp - port 54322 is added

iptables -L output from the host:
================================================
[root@rhsqa-grafton1 ~]# iptables -L | grep 54322
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:54322 ctstate NEW

Below are the steps followed to verify the bug:
===================================================

Engine side:
=================================
1) Install ovirt-imageio-proxy on the engine side if not already installed.
2) check for the value in DB by executing the commands below.
# su - postgres
# psql -d engine
# select * from vdc_options where option_name='ImageProxyAddress';
option_id |    option_name    |  option_value   | version 
-----------+-------------------+-----------------+---------
      1113 | ImageProxyAddress | hostedenginesm1.lab.eng.blr.redhat.com:54323 | general

Option_value should be FQDN of your rhevm instance.

3) If the option_value is shown as localhost then set it using the following command
"UPDATE vdc_options SET option_value='<FQDN_OF_YOUR_RHEVM_instance>:54323' WHERE option_name = 'ImageProxyAddress'; 

Host side:
=======================
4) restart ovirt-engine service by running the command 'service ovirt-engine restart'

5)On the host side make sure "ovirt-imageio-daemon" is installed and 
service ovirt-imageio-daemon is started by running the command 'systemctl status ovirt-imageio-daemon"

Browser side:
================================================
6) Download the certificate by browsing the url below in firefox "https://<FQDN_OF_RHEVM_INSTANCE/ovirt-engine/services/pki-resource?resource=ca-certificate&format=X509-PEM-CA"

7)click on all the check boxes which appears in the popup dialog and say "ok"

8)Download rhel7.4 qcow2 image from access.redhat.com

9) click on disks tab in UI and click 'Upload'

10) Input size and disk name

11) Verified that disk is uploaded successfully to glusterfs storage domain.


Attaching screenshot for the same.
Comment 22 RamaKasturi 2017-09-12 06:46:50 EDT
You can refer to the bug https://bugzilla.redhat.com/show_bug.cgi?id=1348993 on the procedure of how to upload disk image.
Comment 23 RamaKasturi 2017-09-12 06:47 EDT
Created attachment 1324825 [details]
screenshot of the uploaded disk image

Note You need to log in before you can comment on or make changes to this bug.