Bug 1118396 - When nsupdate fails, no useful diagnostic information is reported
Summary: When nsupdate fails, no useful diagnostic information is reported
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 2.2.0
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
: ---
Assignee: Miciah Dashiel Butler Masters
QA Contact: libra bugs
URL:
Whiteboard:
Depends On: 1089697
Blocks: 1113825
TreeView+ depends on / blocked
 
Reported: 2014-07-10 15:19 UTC by Miciah Dashiel Butler Masters
Modified: 2018-12-05 18:59 UTC (History)
9 users (show)

Fixed In Version: rubygem-openshift-origin-dns-nsupdate-1.16.2.2-1.el6op
Doc Type: Bug Fix
Doc Text:
Previously, the output was not logged when the nsupdate command was executed by the DNS plug-in, which complicated diagnosing errors when application DNS records failed to be created or deleted. This bug fix updates the plug-in to log the standard output and errors of the nsupdate command to the broker application log (the /var/log/openshift/broker/production.log file, by default), and errors from the command are now easier to diagnose.
Clone Of: 1089697
Environment:
Last Closed: 2014-08-04 13:27:48 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2014:0999 normal SHIPPED_LIVE Red Hat OpenShift Enterprise 2.1.4 bug fix and enhancement update 2014-08-04 17:26:43 UTC

Description Miciah Dashiel Butler Masters 2014-07-10 15:19:44 UTC
+++ This bug was initially created as a clone of Bug #1089697 +++

Description of problem:
When an app DNS record fails to be created/deleted for some reason, there is no useful output. Only an error 'error adding app record GEARUUID-domain.example.com`. This does not give any information on why the nsupdate command that the broker uses to create DNS entries failed.

Version-Release number of selected component (if applicable):
2.1

How reproducible:
Always

Steps to Reproduce:
1. Temporarily take down the DNS server for an OpenShift instance
2. Attempt to create a gear
3.

Actual results:
2014-03-22 18:07:58.290 [ERROR] Reference ID: 20101eec951f6f326a628e8aa3dfd1db - error adding app record 5i7z05icq-comp.prd-platform.ficoanalyticcloud.com

Expected results:
2014-03-22 18:07:58.290 [ERROR] Reference ID: 20101eec951f6f326a628e8aa3dfd1db - error adding app record 5i7z05icq-comp.prd-platform.ficoanalyticcloud.com - nsupdate output: example error output

Additional info:
Current code:
=======================
    def register_application(app_name, namespace, public_hostname)
      # create an A record for the application in the domain
      fqdn = "#{app_name}-#{namespace}.#{@domain_suffix}"

      cmd = add_cmd(fqdn, public_hostname)    <--- nsupdate command is created through add_cmd method and assigned to `cmd`

      success = system cmd     <--- `cmd` is run
      if not success        <--- if `cmd` returns false, raise error
        raise DNSException.new("error adding app record #{fqdn}")    <---- here is the error
      end
    end
=======================

Using `system` to run a command doesn't return the output, but just true or false, depending on the exit code of the command. It would be useful to use the backticks instead so that output from nsupdate is returned. Then, we can return the output in the error reported.

--- Additional comment from Timothy Williams on 2014-05-13 12:42:10 EDT ---

https://github.com/openshift/origin-server/pull/5408

Comment 1 Miciah Dashiel Butler Masters 2014-07-11 01:31:28 UTC
PR: https://github.com/openshift/enterprise-server/pull/318

Comment 4 Gaoyun Pei 2014-07-15 05:49:55 UTC
Verify this bug with rubygem-openshift-origin-dns-nsupdate-1.16.2.2-1.el6op.noarch.

When nsupdate fails during creating or deleting app, the error log in /var/log/openshift/broker/production.log will give the error output of nsupdate. 


1. Modify the nsupdate commands in add_cmd, change the server port to 100 to make exception. Restart openshift-broker and try to create an app.

[root@broker ~]# rhc app create test6 ruby-1.9
Application Options
-------------------
Domain:     111
Cartridges: ruby-1.9
Gear Size:  default
Scaling:    no

Creating application 'test6' ... Error adding DNS application record test6-111.ose21z-manual.com.cn rc=2

In production.log:
2014-07-14 20:30:58.888 [ERROR] Error adding DNS application record test6-111.ose21z-manual.com.cn: ; Communication with server failed: timed out (pid:8530)
2014-07-14 20:30:58.889 [ERROR] Error adding DNS application record test6-111.ose21z-manual.com.cn rc=2 (pid:8530)
2014-07-14 20:30:58.889 [ERROR] /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-dns-nsupdate-1.16.2.2/lib/openshift/nsupdate_plugin.rb:191:in `modify_dns'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-dns-nsupdate-1.16.2.2/lib/openshift/nsupdate_plugin.rb:212:in `register_application'
...


2. Remove #{keystring} in nsupdate commands of del_cmd. Restart openshift-broker and try to delete an app.

[root@broker ~]# rhc app delete test4 --confirm
Deleting application 'test4' ... Error removing DNS application record test4-111.ose21z-manual.com.cn rc=2

In production.log:
2014-07-14 20:33:59.168 [ERROR] Error removing DNS application record test4-111.ose21z-manual.com.cn: update failed: REFUSED (pid:8875)
2014-07-14 20:33:59.169 [ERROR] Error removing DNS application record test4-111.ose21z-manual.com.cn rc=2 (pid:8875)
2014-07-14 20:33:59.169 [ERROR] /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-dns-nsupdate-1.16.2.2/lib/openshift/nsupdate_plugin.rb:190:in `modify_dns'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-dns-nsupdate-1.16.2.2/lib/openshift/nsupdate_plugin.rb:228:in `deregister_application'
...

Comment 6 errata-xmlrpc 2014-08-04 13:27:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0999.html


Note You need to log in before you can comment on or make changes to this bug.