Bug 1011345

Summary: Met "exception : Operation not permitted - send(2)" when resolve DNS via rest api.
Product: OpenShift Online Reporter: Liang Xia <lxia>
Component: MasterAssignee: Lili Nader <lnader>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: low Docs Contact:
Priority: medium    
Version: 2.xCC: mfisher, tkramer
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-01-24 03:23:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Liang Xia 2013-09-24 06:48:34 UTC 
Description of problem:
Try to resolve an app's DNS via rest api, following message returned:
Could not resolve DNS phps-lxia.dev.rhcloud.com: Send failed to 204.13.250.23:53 from 0.0.0.0:63561, use_tcp=false, exception : Operation not permitted - send(2)
These apps can be access via browser. And can be resolved via nslookup.
# nslookup phps-lxia.dev.rhcloud.com
Server:        10.66.127.17
Address:    10.66.127.17#53
Non-authoritative answer:
phps-lxia.dev.rhcloud.com    canonical name = ec2-23-20-228-13.compute-1.amazonaws.com.
Name:    ec2-23-20-228-13.compute-1.amazonaws.com
Address: 23.20.228.13

Version-Release number of selected component (if applicable):
devenv_3816
INT(devenv_?)
STG(devenv_stage_?)

How reproducible:
always

Steps to Reproduce:
1.create an app of any type.
2.resolve the app's DNS via rest api.
3.

Actual results:
# curl -k -s -H "Accept: application/xml" -u user:password https://server/broker/rest/domains/lxia/applications/phps/dns_resolvable
<?xml version="1.0" encoding="UTF-8"?>
<response>
  <status>not_found</status>
  <type nil="true"></type>
  <data>
    <datum nil="true"></datum>
  </data>
  <messages>
    <message>
      <severity>error</severity>
      <text>Could not resolve DNS phps-lxia.dev.rhcloud.com: Send failed to 204.13.250.23:53 from 0.0.0.0:63561, use_tcp=false, exception : Operation not permitted - send(2)</text>
      <exit-code>170</exit-code>
      <field nil="true"></field>
    </message>
  </messages>
  <version>1.6</version>
  <api-version>1.6</api-version>
  <supported-api-versions>
    <supported-api-version>1.0</supported-api-version>
    <supported-api-version>1.1</supported-api-version>
    <supported-api-version>1.2</supported-api-version>
    <supported-api-version>1.3</supported-api-version>
    <supported-api-version>1.4</supported-api-version>
    <supported-api-version>1.5</supported-api-version>
    <supported-api-version>1.6</supported-api-version>
  </supported-api-versions>
</response>

Expected results:
No errors.

Additional info:
Affected apps in INT:
jenkins-lxiamigrate.int.rhcloud.com
wordpress-lxiamigrate.int.rhcloud.com
nscalejenkins-wjiang.int.rhcloud.com

Affected apps in STG:
jenkins-lxiamigrate.stg.rhcloud.com
jbossews2-lxiamigrate.stg.rhcloud.com

Not affected apps in INT:
jbossews2-lxiamigrate.int.rhcloud.com
jbossews2s-lxiamigrate.int.rhcloud.com

Note affected apps in STG:
jbossews2s-lxiamigrate.stg.rhcloud.com

# nslookup wordpress-lxiamigrate.int.rhcloud.com
Server:        10.66.127.17
Address:    10.66.127.17#53
Non-authoritative answer:
wordpress-lxiamigrate.int.rhcloud.com    canonical name = ex-std-node2.int.rhcloud.com.
ex-std-node2.int.rhcloud.com    canonical name = ec2-23-22-238-189.compute-1.amazonaws.com.
Name:    ec2-23-22-238-189.compute-1.amazonaws.com
Address: 23.22.238.189
Comment 1 Liang Xia 2013-09-24 06:56:04 UTC 
devenv_3816
INT(release 2.0.33)
STG(devenv_stage_472)
Comment 2 Lili Nader 2013-10-14 18:57:21 UTC 
Could you please try this on production?  I think the permissions might be different on devenv setup.
Comment 3 Liang Xia 2013-10-15 01:51:16 UTC 
Tried this on production, got following errors:

# curl -k -s -H 'Accept:application/xml' -u 'lxia:passwd' https://openshift.redhat.com/broker/rest/domains/lxia/applications/phps/dns_resolvable -X GET
<?xml version="1.0" encoding="UTF-8"?>
<response>
  <status>not_found</status>
  <type nil="true"></type>
  <data>
    <datum nil="true"></datum>
  </data>
  <messages>
    <message>
      <severity>error</severity>
      <text>Could not resolve DNS phps-lxia.rhcloud.com: dnsruby can't connect to 208.78.71.23:53 from 0.0.0.0:64173, use_tcp=false, exception = Errno::EACCES, Permission denied - bind(2)</text>
      <exit-code>170</exit-code>
      <field nil="true"></field>
    </message>
  </messages>
  <version>1.6</version>
  <api-version>1.6</api-version>
  <supported-api-versions>
    <supported-api-version>1.0</supported-api-version>
    <supported-api-version>1.1</supported-api-version>
    <supported-api-version>1.2</supported-api-version>
    <supported-api-version>1.3</supported-api-version>
    <supported-api-version>1.4</supported-api-version>
    <supported-api-version>1.5</supported-api-version>
    <supported-api-version>1.6</supported-api-version>
  </supported-api-versions>
</response>
Comment 6 Liang Xia 2013-11-06 07:08:11 UTC 
On devenv_3993,
  <status>ok</status>
  <type>boolean</type>
  <data>
    <datum>true</datum>
  </data>
  <messages>
    <message>
      <severity>info</severity>
      <text>Resolved DNS php-lxia.dev.rhcloud.com</text>
      <exit-code>0</exit-code>
      <field nil="true"></field>
      <index nil="true"></index>
    </message>
  </messages>
  <version>1.6</version>

On INT,
  <status>ok</status>
  <type>boolean</type>
  <data>
    <datum>true</datum>
  </data>
  <messages>
    <message>
      <severity>info</severity>
      <text>Resolved DNS jbossews20-lxiamigrate.int.rhcloud.com</text>
      <exit-code>0</exit-code>
      <field nil="true"></field>
      <index nil="true"></index>
    </message>
  </messages>
  <version>1.6</version>

On STG,
  <status>not_found</status>
  <type nil="true"></type>
  <data>
    <datum nil="true"></datum>
  </data>
  <messages>
    <message>
      <severity>error</severity>
      <text>Could not resolve DNS jbossews20-lxiamigrate.stg.rhcloud.com: recvfrom failed from ; Connection refused - recvfrom(2)</text>
      <exit-code>170</exit-code>
      <field nil="true"></field>
      <index nil="true"></index>
    </message>
  </messages>
  <version>1.6</version>

On PROD,
  <status>not_found</status>
  <type nil="true"></type>
  <data>
    <datum nil="true"></datum>
  </data>
  <messages>
    <message>
      <severity>error</severity>
      <text>Could not resolve DNS phps-lxia.rhcloud.com: recvfrom failed from ; Connection refused - recvfrom(2)</text>
      <exit-code>170</exit-code>
      <field nil="true"></field>
      <index nil="true"></index>
    </message>
  </messages>
  <version>1.6</version>
Comment 7 Liang Xia 2013-11-06 07:24:23 UTC 
Move to verified based on comment #6
Comment 10 Liang Xia 2013-11-07 04:37:43 UTC 
Verified on STG ( HotFix Release 2.0.35.1 ), DNS can be resolved via REST API.

  <messages>
    <message>
      <severity>info</severity>
      <text>Resolved DNS diy-lxiamigrate.stg.rhcloud.com</text>
      <exit-code>0</exit-code>
      <field nil="true"></field>
      <index nil="true"></index>
    </message>
  </messages>

  <messages>
    <message>
      <severity>info</severity>
      <text>Resolved DNS jbossews22s-lxiamigrate.stg.rhcloud.com</text>
      <exit-code>0</exit-code>
      <field nil="true"></field>
      <index nil="true"></index>
    </message>
  </messages>

Move to verified.
Comment 12 Liang Xia 2013-11-08 02:03:51 UTC 
Verified on Production.

# curl -k -s -H 'Accept:application/xml' -u 'lxia' https://openshift.redhat.com/broker/rest/domains/lxia/applications/phps/dns_resolvable -X GET
Enter host password for user 'lxia':
<?xml version="1.0" encoding="UTF-8"?>
<response>
  <status>ok</status>
  <type>boolean</type>
  <data>
    <datum>true</datum>
  </data>
  <messages>
    <message>
      <severity>info</severity>
      <text>Resolved DNS phps-lxia.rhcloud.com</text>
      <exit-code>0</exit-code>
      <field nil="true"></field>
      <index nil="true"></index>
    </message>
  </messages>
  <version>1.6</version>
  <api-version>1.6</api-version>
  <supported-api-versions>
    <supported-api-version>1.0</supported-api-version>
    <supported-api-version>1.1</supported-api-version>
    <supported-api-version>1.2</supported-api-version>
    <supported-api-version>1.3</supported-api-version>
    <supported-api-version>1.4</supported-api-version>
    <supported-api-version>1.5</supported-api-version>
    <supported-api-version>1.6</supported-api-version>
  </supported-api-versions>
</response>