Bug 1280497

Summary: [Supportability] Build OpenShift with DWARF
Product: OpenShift Container Platform Reporter: Eric Rich <erich>
Component: BuildAssignee: Troy Dawson <tdawson>
Status: CLOSED ERRATA QA Contact: Wenjing Zheng <wzheng>
Severity: medium Docs Contact:
Priority: high    
Version: 3.1.0CC: aos-bugs, atomic-bugs, bleanhar, ccoleman, jamills, jokerman, lfriedma, lsm5, mmccomas, pep, pruan, uobergfe
Target Milestone: ---Keywords: Extras
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1280068 Environment:
Last Closed: 2016-01-26 19:17:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1280068    
Bug Blocks: 1267746    

Description Eric Rich 2015-11-11 22:02:10 UTC
+++ This bug was initially created as a clone of Bug #1280068 +++

Description of problem:

debug symbols are currently not included in openshift binaries, and we are unable to strip debug symbols into *debug packages (https://bugzilla.redhat.com/show_bug.cgi?id=1236317).

Without DWARF debug info, GSS is unable to analyze docker core files.

Version-Release number of selected component (if applicable):

All versions of OpenShift

How reproducible:

100%

Steps to Reproduce:
1. generate a OpenShift corefile
2. open the core file up in gdb

Actual results:

No real usable information can be obtained

Expected results:

docker core can be analyzed as effectively as any other core file.

Additional info:

Comment 1 Clayton Coleman 2015-11-12 07:20:17 UTC
Interesting, Origin had Dwarf at one point and I don't recall anyone removing it.

Comment 5 Troy Dawson 2016-01-05 21:11:15 UTC
This turned out to be a rpmbuild problem, and if being fixed by this pull request.
https://github.com/openshift/origin/pull/6545
Basically we are just adding
%global __os_install_post %{_rpmconfigdir}/brp-compress

Before it put's everything in it's proper rpm, rpmbuild tries to make things as small as possible.  We were already skipping the part where it pulls out the debuginfo (which breaks go packages), but there is another step called brp-strip-static-archive.
According to rpmbuild, go binaries are static-archive's, and so it was stripping the debug stuff out of them.

Why did origin have dwarf debugging and then it didn't?
Origin has always been compiled with the DWARF debugging on.
We originally were not packaging origin into rpms.  They were either in tarballs, images, or compiled locally.
When we started putting it into rpm's was when we started looking the debuginfo.

NOTE: How to tell if the binary has dwarf debug in it or not.
1 - Run something that will take some time
  /usr/bin/oc status  # On a non-setup machine, this takes a while to timeout

2 - Find out the process id
  ps ax | grep status | grep oc

3 - gdb attach to that process and look if it was able to load debuginfo for that process.

3a - No DWARF debugging
  gdb attach 4237
  <snip>
  Attaching to process 4237
  Reading symbols from /usr/bin/oc...(no debugging symbols found)...done.
  <snip>

3b - DWARF debugging in binary
  gdb attach 8138
  <snip>
  Attaching to process 8138
  Reading symbols from /usr/bin/oc...done.
  <snip>

Comment 7 Wenjing Zheng 2016-01-12 06:11:22 UTC
Verified with atomic-openshift-3.1.1.1-1.git.0.dba03a7.el7aosAttaching to Attaching to process 30758
Reading symbols from /usr/bin/oc...done.

Comment 8 Wenjing Zheng 2016-01-15 02:47:24 UTC
I can reproduce this issue with AEP container install RH7.2 env with below version:
openshift v3.1.1.3
kubernetes v1.1.0-origin-1107-g4c8e6f4
etcd 2.1.2

Here is the result, so will re-open this bug now:
Attaching to process 6456
Reading symbols from /usr/bin/bash...Reading symbols from /usr/bin/bash...(no debugging symbols found)...done.

Comment 9 Troy Dawson 2016-01-15 04:40:27 UTC
This ticket isn't about /usr/bin/bash, it is about /usr/bin/oc, or any of the binaries that come from openshift.

Please re-test against /usr/bin/oc, or /usr/bin/openshift.

Comment 10 Wenjing Zheng 2016-01-15 06:29:59 UTC
Thanks, it works now:
For container env, oc/openshift binary should be copied out from atomic-openshift-master container:
1. docker cp atomic-openshift-master:/usr/bin/oc ./
2. ./oc get pods --watch
3. ps-ef | grep oc
4. gdb attach <oc_process_id>
Attaching to process 12699
Reading symbols from /root/oc...done.

Comment 12 errata-xmlrpc 2016-01-26 19:17:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2016:0070