Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1118396

Summary: When nsupdate fails, no useful diagnostic information is reported
Product: OpenShift Container Platform Reporter: Miciah Dashiel Butler Masters <mmasters>
Component: NodeAssignee: Miciah Dashiel Butler Masters <mmasters>
Status: CLOSED ERRATA QA Contact: libra bugs <libra-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 2.2.0CC: abhgupta, adellape, gpei, jokerman, libra-bugs, libra-onpremise-devel, lmeyer, mmccomas, tiwillia
Target Milestone: ---Keywords: Upstream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: rubygem-openshift-origin-dns-nsupdate-1.16.2.2-1.el6op Doc Type: Bug Fix
Doc Text:
Previously, the output was not logged when the nsupdate command was executed by the DNS plug-in, which complicated diagnosing errors when application DNS records failed to be created or deleted. This bug fix updates the plug-in to log the standard output and errors of the nsupdate command to the broker application log (the /var/log/openshift/broker/production.log file, by default), and errors from the command are now easier to diagnose.
 Story Points: ---
Clone Of: 1089697 Environment:
Last Closed: 2014-08-04 13:27:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1089697    
Bug Blocks: 1113825    

Description Miciah Dashiel Butler Masters 2014-07-10 15:19:44 UTC 
+++ This bug was initially created as a clone of Bug #1089697 +++

Description of problem:
When an app DNS record fails to be created/deleted for some reason, there is no useful output. Only an error 'error adding app record GEARUUID-domain.example.com`. This does not give any information on why the nsupdate command that the broker uses to create DNS entries failed.

Version-Release number of selected component (if applicable):
2.1

How reproducible:
Always

Steps to Reproduce:
1. Temporarily take down the DNS server for an OpenShift instance
2. Attempt to create a gear
3.

Actual results:
2014-03-22 18:07:58.290 [ERROR] Reference ID: 20101eec951f6f326a628e8aa3dfd1db - error adding app record 5i7z05icq-comp.prd-platform.ficoanalyticcloud.com

Expected results:
2014-03-22 18:07:58.290 [ERROR] Reference ID: 20101eec951f6f326a628e8aa3dfd1db - error adding app record 5i7z05icq-comp.prd-platform.ficoanalyticcloud.com - nsupdate output: example error output

Additional info:
Current code:
=======================
    def register_application(app_name, namespace, public_hostname)
      # create an A record for the application in the domain
      fqdn = "#{app_name}-#{namespace}.#{@domain_suffix}"

      cmd = add_cmd(fqdn, public_hostname)    <--- nsupdate command is created through add_cmd method and assigned to `cmd`

      success = system cmd     <--- `cmd` is run
      if not success        <--- if `cmd` returns false, raise error
        raise DNSException.new("error adding app record #{fqdn}")    <---- here is the error
      end
    end
=======================

Using `system` to run a command doesn't return the output, but just true or false, depending on the exit code of the command. It would be useful to use the backticks instead so that output from nsupdate is returned. Then, we can return the output in the error reported.

--- Additional comment from Timothy Williams on 2014-05-13 12:42:10 EDT ---

https://github.com/openshift/origin-server/pull/5408
Comment 1 Miciah Dashiel Butler Masters 2014-07-11 01:31:28 UTC 
PR: https://github.com/openshift/enterprise-server/pull/318
Comment 4 Gaoyun Pei 2014-07-15 05:49:55 UTC 
Verify this bug with rubygem-openshift-origin-dns-nsupdate-1.16.2.2-1.el6op.noarch.

When nsupdate fails during creating or deleting app, the error log in /var/log/openshift/broker/production.log will give the error output of nsupdate. 


1. Modify the nsupdate commands in add_cmd, change the server port to 100 to make exception. Restart openshift-broker and try to create an app.

[root@broker ~]# rhc app create test6 ruby-1.9
Application Options
-------------------
Domain:     111
Cartridges: ruby-1.9
Gear Size:  default
Scaling:    no

Creating application 'test6' ... Error adding DNS application record test6-111.ose21z-manual.com.cn rc=2

In production.log:
2014-07-14 20:30:58.888 [ERROR] Error adding DNS application record test6-111.ose21z-manual.com.cn: ; Communication with server failed: timed out (pid:8530)
2014-07-14 20:30:58.889 [ERROR] Error adding DNS application record test6-111.ose21z-manual.com.cn rc=2 (pid:8530)
2014-07-14 20:30:58.889 [ERROR] /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-dns-nsupdate-1.16.2.2/lib/openshift/nsupdate_plugin.rb:191:in `modify_dns'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-dns-nsupdate-1.16.2.2/lib/openshift/nsupdate_plugin.rb:212:in `register_application'
...


2. Remove #{keystring} in nsupdate commands of del_cmd. Restart openshift-broker and try to delete an app.

[root@broker ~]# rhc app delete test4 --confirm
Deleting application 'test4' ... Error removing DNS application record test4-111.ose21z-manual.com.cn rc=2

In production.log:
2014-07-14 20:33:59.168 [ERROR] Error removing DNS application record test4-111.ose21z-manual.com.cn: update failed: REFUSED (pid:8875)
2014-07-14 20:33:59.169 [ERROR] Error removing DNS application record test4-111.ose21z-manual.com.cn rc=2 (pid:8875)
2014-07-14 20:33:59.169 [ERROR] /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-dns-nsupdate-1.16.2.2/lib/openshift/nsupdate_plugin.rb:190:in `modify_dns'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-dns-nsupdate-1.16.2.2/lib/openshift/nsupdate_plugin.rb:228:in `deregister_application'
...
Comment 6 errata-xmlrpc 2014-08-04 13:27:48 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0999.html