Bug 763331 (GLUSTER-1599)

Summary: can we build glusterfs with -O0 -g by default
Product: [Community] GlusterFS Reporter: Krishna Srinivas <krishna>
Component: buildAssignee: Amar Tumballi <amarts>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: low Docs Contact:
Priority: low    
Version: 3.0.5CC: csaba, fharshav, gluster-bugs, joe, shehjart, vraman
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Krishna Srinivas 2010-09-14 20:31:45 UTC
gdb debugging is difficult because by default glusterfs is built with -O2 because of op_errno=<value optimized out>:

(gdb)
dht_readdirp_cbk (frame=<value optimized out>, cookie=0x2454dd0, this=0x2438cf0, op_ret=<value optimized out>, op_errno=<value optimized out>,
    orig_entries=<value optimized out>) at dht-common.c:2179
2179    dht-common.c: No such file or directory.
        in dht-common.c
(gdb)


Can we build by default with "-O0 -g" ?

Thanks

Comment 1 Amar Tumballi 2010-09-17 03:49:00 UTC
By default we should have -O2 only as debug symbols may not be desirable in lot environment.. Instead of having info based on gdb, we should increase the logging behavior to have all these traced. 

If its RPM based build, there will be debuginfo rpm, which will have required debuginfo, withwhich, you should be able to get the backtrace properly.

Comment 2 Krishna Srinivas 2010-09-17 04:05:51 UTC
Would like to reopen this please.

(In reply to comment #1)
> By default we should have -O2 only as debug symbols may not be desirable in lot
> environment.. Instead of having info based on gdb, we should increase the
> logging behavior to have all these traced. 
> 

In what environment are debug symbols not desirable? Definitely in embedded systems with a few megs of RAM and low disk space it is not desirable. Our storage servers are huge when it comes to RAM.

Logging behavior should be extensive. But until we are confident that we have enough logging to debug any issue it is safer to run with -O0. Besides even with extensive logging it would be harmless to run with -O0

> If its RPM based build, there will be debuginfo rpm, which will have required
> debuginfo, withwhich, you should be able to get the backtrace properly.

debuginfo is fine, the issue is about -O0

Comment 3 Amar Tumballi 2010-09-18 03:46:04 UTC
you want it in 3.0.x branch or doing it in mainline for all new releases is fine?

Comment 4 Csaba Henk 2010-09-18 04:00:04 UTC
(In reply to comment #2)
> debuginfo is fine, the issue is about -O0

Would that not hurt performance?

Comment 5 Krishna Srinivas 2010-09-18 13:44:43 UTC
(In reply to comment #4)
> (In reply to comment #2)
> > debuginfo is fine, the issue is about -O0
> 
> Would that not hurt performance?

I will check with Anush if he can do some basic comparison tests. I doubt if it will practically cause any hit to performance. 

(In reply to comment #3)
> you want it in 3.0.x branch or doing it in mainline for all new releases is
> fine?

Both...

Comment 6 Shehjar Tikoo 2010-09-18 17:38:13 UTC
(In reply to comment #5)
> (In reply to comment #4)
> > (In reply to comment #2)
> > > debuginfo is fine, the issue is about -O0
> > 
> > Would that not hurt performance?
> 
> I will check with Anush if he can do some basic comparison tests. I doubt if it
> will practically cause any hit to performance. 
> 

Even if it does, i'd prefer to go with them simply because it reduces the effort required for fixing things when they go wrong. Such small items constitute what we can call manageability of the file system.
Moreover, we should depend on our design and algos for performance, not how the binary is shaped.


> (In reply to comment #3)
> > you want it in 3.0.x branch or doing it in mainline for all new releases is
> > fine?
> 
> Both...

Comment 7 Krishna Srinivas 2010-09-18 23:46:30 UTC
> > > Would that not hurt performance?
> > 
> > I will check with Anush if he can do some basic comparison tests. I doubt if it
> > will practically cause any hit to performance. 
> > 
> 
> Even if it does, i'd prefer to go with them simply because it reduces the
> effort required for fixing things when they go wrong. Such small items
> constitute what we can call manageability of the file system.
> Moreover, we should depend on our design and algos for performance, not how the
> binary is shaped.
> 

I agree because the gdb o/p pasted in this bug was actually from a customer setup which I was debugging and I wished glusterfs was built with -O0. In future when we are confident about glusterfs' logging to debug any issue we can go back to -O2.

Comment 8 Csaba Henk 2010-09-19 04:40:35 UTC
(In reply to comment #7)
> I agree because the gdb o/p pasted in this bug was actually from a customer
> setup which I was debugging and I wished glusterfs was built with -O0. In
> future when we are confident about glusterfs' logging to debug any issue we can

That day will come when glusterfs gets rewritten in a memory managed language.

> go back to -O2.

(although then we'd were lost for -O2 as such.)

OCaml would be a promising plan! I wonder though how much headache would it cause to Deepti to fill up prospective positions with proficient OCaml hackers... but I'm sure that after all that wouldn't be too much a big deal in B'lore ;),  of F#  folks should there be many, given the corporate support of that language. I can also imagine how it would affect sales & marketing, and inspiring VC folks...

RELEASE NOTES FOR 5.0 RELEASE:

 Glusterfs 5.0
 * Rewritten in OCaml

<<EOF.

"Ummm... hey dude now it's written in a supercool memory managed language... you know like Java just even better... can you imagine any other product in the market like this? Err, how to express... now runs with the speed of Mr. Beckham but dances with the elegance of Mrs. Beckham and handles complex situation like a Houdini! That camel language, yes, oh no, no Perl, for heaven sake!"

Comment 9 Vijay Bellur 2010-09-20 12:13:21 UTC
PATCH: http://patches.gluster.com/patch/4867 in master (build glusterfs with '-O0' flag by default)

Comment 10 Vijay Bellur 2010-09-22 03:22:36 UTC
PATCH: http://patches.gluster.com/patch/4868 in release-3.0 (build glusterfs with '-O0' flag by default)

Comment 11 Joe Julian 2010-11-09 16:51:01 UTC
Ugh, this is a horrible performance hit. Go back to -O2 for releases and build your own rpm if you need -O0 please.

Comment 12 Harshavardhana 2010-11-09 19:23:45 UTC
(In reply to comment #11)
> Ugh, this is a horrible performance hit. Go back to -O2 for releases and build
> your own rpm if you need -O0 please.

There are always two things for this bug first of all 

-O0 is an optimization flag

-g is debugging symbols flag

Now optimization is all in necessary hood done to optimize code paths and relatively less execution times. Many applications if not all have benefited from having -O2 or higher due to floating point operations (large cpu cycles). 

-g in all is a different beast which is required for debugging symbols which are only used or loaded when "gdb <program>" is executed. Size of the binary size increases which is of no concern with -g, there are no additional memory foot prints and it doesn't even come into picture during normal execution of the binary. 

Without augmentative understanding of the effects of such a change in case of glusterfs performance it would be unwise to take it as a default. 

Also it will be really painful for package maintainers for various projects, since packaging glusterfs for a distribution leads to adding -O2 flag. You would see spurious maintainers maintaining their own package specification file to fix this problem. 

Currently such an example is Fedora distribution which is stuck @ 2.0.9 release. When we asked them to move to 3.0.x they argued over backward compatibility and now they have their own RPM specification file. 

ArchLinux where the maintainers have constantly argued over our Arrays with NULL  ending which leads to Segfaults for Glusterfs with -D_FORTIFY_SOURCE.

Comment 13 Joe Julian 2010-11-09 20:06:18 UTC
I run mysqld with the files hosted on glusterfs. A query that typically took less than 2 seconds on 3.0.5 took over 30 seconds using the 3.0.6 version (redhat rpm).

I applied a patch in the spec file reverting http://patches.gluster.com/patch/4868, rebuilt the rpms and installed on the servers and mysqld client. The queries returned to <2 seconds.

Comment 14 Krishna Srinivas 2010-11-10 03:34:48 UTC
(In reply to comment #13)
> I run mysqld with the files hosted on glusterfs. A query that typically took
> less than 2 seconds on 3.0.5 took over 30 seconds using the 3.0.6 version
> (redhat rpm).
> 
> I applied a patch in the spec file reverting
> http://patches.gluster.com/patch/4868, rebuilt the rpms and installed on the
> servers and mysqld client. The queries returned to <2 seconds.

Ah, we should do -O2 then.