Bug 557266

Summary: Compilation of static markers is broken in rawhide
Product: [Fedora] Fedora Reporter: Tom Lane <tgl>
Component: postgresqlAssignee: Tom Lane <tgl>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: high    
Version: rawhideCC: devrim, dsmith, fche, hhorak, jistone, mjw, mjw, roland, scox, tgl, wcohen
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 557297 (view as bug list) Environment:
Last Closed: 2010-01-20 17:50:16 EST Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Description Tom Lane 2010-01-20 16:08:19 EST
Description of problem:
I just tried to rebuild postgresql in rawhide, and it fails with an "undefined reference" linker error for each systemtap static marker.  This worked last week, and I have not changed the postgresql code, so something went pear-shaped on the systemtap side.

Version-Release number of selected component (if applicable):
systemtap-sdt-devel-1.1-1.fc13.x86_64

How reproducible:
Probably 100%, but I didn't retry

Steps to Reproduce:
1. Try to build postgresql
  
Actual results:
See tail of build log here:
http://koji.fedoraproject.org/koji/getfile?taskID=1934768&name=build.log

Expected results:
Should build

Additional info:
This is a bit urgent since it blocks rebuilds of postgresql, and presumably every other package with static markers.  If I have to, I'll disable static marker support, but I'd rather not ...
Comment 2 Mark Wielaard 2010-01-20 16:36:43 EST
Stan, maybe this rings a bell for you? This is because references to the semaphores are missing:

[...]
make[3]: Entering directory `/builddir/build/BUILD/postgresql-8.4.2/src/backend/utils'
dtrace -C -h -s probes.d -o probes.h.tmp
sed -e 's/POSTGRESQL_/TRACE_POSTGRESQL_/g' probes.h.tmp >probes.h
rm probes.h.tmp
make[3]: Leaving directory `/builddir/build/BUILD/postgresql-8.4.2/src/backend/utils'
cd ../../src/include/utils/ && rm -f probes.h && \
	    ln -s ../../../src/backend/utils/probes.h .
make[3]: Leaving directory `/builddir/build/BUILD/postgresql-8.4.2/src/backend/utils'
[...]
gcc -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -DLINUX_OOM_ADJ=0 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -fno-strict-aliasing -fwrapv -L../../src/port -Wl,--as-needed -Wl,-E access/common/heaptuple.o access/common/indextuple.o access/common/printtup.o access/common/reloptions.o access/common/scankey.o access/common/tupdesc.o access/gist/gist.o access/gist/gistutil.o access/gist/gistxlog.o access/gist/gistvacuum.o access/gist/gistget.o access/gist/gistscan.o access/gist/gistproc.o access/gist/gistsplit.o access/hash/hash.o access/hash/hashfunc.o access/hash/hashinsert.o access/hash/hashovfl.o access/hash/hashpage.o access/hash/hashscan.o access/hash/hashsearch.o access/hash/hashsort.o access/hash/hashutil.o access/heap/heapam.o access/heap/hio.o access/heap/pruneheap.o access/heap/rewriteheap.o access/heap/syncscan.o access/heap/tuptoaster.o access/heap/visibilitymap.o access/index/genam.o access/index/indexam.o access/nbtree/nbtcompare.o access/nbtree/nbtinsert.o access/nbtree/nbtpage.o access/nbtree/nbtree.o access/nbtree/nbtsearch.o access/nbtree/nbtutils.o access/nbtree/nbtsort.o access/nbtree/nbtxlog.o access/transam/clog.o access/transam/transam.o access/transam/varsup.o access/transam/xact.o access/transam/xlog.o access/transam/xlogutils.o access/transam/rmgr.o access/transam/slru.o access/transam/subtrans.o access/transam/multixact.o access/transam/twophase.o access/transam/twophase_rmgr.o access/gin/ginutil.o access/gin/gininsert.o access/gin/ginxlog.o access/gin/ginentrypage.o access/gin/gindatapage.o access/gin/ginbtree.o access/gin/ginscan.o access/gin/ginget.o access/gin/ginvacuum.o access/gin/ginarrayproc.o access/gin/ginbulk.o access/gin/ginfast.o bootstrap/bootparse.o bootstrap/bootstrap.o catalog/catalog.o catalog/dependency.o catalog/heap.o catalog/index.o catalog/indexing.o catalog/namespace.o catalog/aclchk.o catalog/pg_aggregate.o catalog/pg_constraint.o catalog/pg_conversion.o catalog/pg_depend.o catalog/pg_enum.o catalog/pg_inherits.o catalog/pg_largeobject.o catalog/pg_namespace.o catalog/pg_operator.o catalog/pg_proc.o catalog/pg_shdepend.o catalog/pg_type.o catalog/storage.o catalog/toasting.o parser/analyze.o parser/gram.o parser/keywords.o parser/parser.o parser/parse_agg.o parser/parse_cte.o parser/parse_clause.o parser/parse_expr.o parser/parse_func.o parser/parse_node.o parser/parse_oper.o parser/parse_relation.o parser/parse_type.o parser/parse_coerce.o parser/parse_target.o parser/parse_utilcmd.o parser/scansup.o parser/kwlookup.o commands/aggregatecmds.o commands/alter.o commands/analyze.o commands/async.o commands/cluster.o commands/comment.o commands/conversioncmds.o commands/copy.o commands/dbcommands.o commands/define.o commands/discard.o commands/explain.o commands/foreigncmds.o commands/functioncmds.o commands/indexcmds.o commands/lockcmds.o commands/operatorcmds.o commands/opclasscmds.o commands/portalcmds.o commands/prepare.o commands/proclang.o commands/schemacmds.o commands/sequence.o commands/tablecmds.o commands/tablespace.o commands/trigger.o commands/tsearchcmds.o commands/typecmds.o commands/user.o commands/vacuum.o commands/vacuumlazy.o commands/variable.o commands/view.o executor/execAmi.o executor/execCurrent.o executor/execGrouping.o executor/execJunk.o executor/execMain.o executor/execProcnode.o executor/execQual.o executor/execScan.o executor/execTuples.o executor/execUtils.o executor/functions.o executor/instrument.o executor/nodeAppend.o executor/nodeAgg.o executor/nodeBitmapAnd.o executor/nodeBitmapOr.o executor/nodeBitmapHeapscan.o executor/nodeBitmapIndexscan.o executor/nodeHash.o executor/nodeHashjoin.o executor/nodeIndexscan.o executor/nodeMaterial.o executor/nodeMergejoin.o executor/nodeNestloop.o executor/nodeFunctionscan.o executor/nodeRecursiveunion.o executor/nodeResult.o executor/nodeSeqscan.o executor/nodeSetOp.o executor/nodeSort.o executor/nodeUnique.o executor/nodeValuesscan.o executor/nodeCtescan.o executor/nodeWorktablescan.o executor/nodeLimit.o executor/nodeGroup.o executor/nodeSubplan.o executor/nodeSubqueryscan.o executor/nodeTidscan.o executor/nodeWindowAgg.o executor/tstoreReceiver.o executor/spi.o foreign/foreign.o lib/dllist.o lib/stringinfo.o libpq/be-fsstubs.o libpq/be-secure.o libpq/auth.o libpq/crypt.o libpq/hba.o libpq/ip.o libpq/md5.o libpq/pqcomm.o libpq/pqformat.o libpq/pqsignal.o main/main.o nodes/nodeFuncs.o nodes/nodes.o nodes/list.o nodes/bitmapset.o nodes/tidbitmap.o nodes/copyfuncs.o nodes/equalfuncs.o nodes/makefuncs.o nodes/outfuncs.o nodes/readfuncs.o nodes/print.o nodes/read.o nodes/params.o nodes/value.o optimizer/geqo/geqo_copy.o optimizer/geqo/geqo_eval.o optimizer/geqo/geqo_main.o optimizer/geqo/geqo_misc.o optimizer/geqo/geqo_mutation.o optimizer/geqo/geqo_pool.o optimizer/geqo/geqo_recombination.o optimizer/geqo/geqo_selection.o optimizer/geqo/geqo_erx.o optimizer/geqo/geqo_pmx.o optimizer/geqo/geqo_cx.o optimizer/geqo/geqo_px.o optimizer/geqo/geqo_ox1.o optimizer/geqo/geqo_ox2.o optimizer/path/allpaths.o optimizer/path/clausesel.o optimizer/path/costsize.o optimizer/path/equivclass.o optimizer/path/indxpath.o optimizer/path/joinpath.o optimizer/path/joinrels.o optimizer/path/orindxpath.o optimizer/path/pathkeys.o optimizer/path/tidpath.o optimizer/plan/createplan.o optimizer/plan/initsplan.o optimizer/plan/planagg.o optimizer/plan/planmain.o optimizer/plan/planner.o optimizer/plan/setrefs.o optimizer/plan/subselect.o optimizer/prep/prepjointree.o optimizer/prep/prepqual.o optimizer/prep/preptlist.o optimizer/prep/prepunion.o optimizer/util/clauses.o optimizer/util/joininfo.o optimizer/util/pathnode.o optimizer/util/placeholder.o optimizer/util/plancat.o optimizer/util/predtest.o optimizer/util/relnode.o optimizer/util/restrictinfo.o optimizer/util/tlist.o optimizer/util/var.o port/dynloader.o port/pg_sema.o port/pg_shmem.o postmaster/autovacuum.o postmaster/bgwriter.o postmaster/fork_process.o postmaster/pgarch.o postmaster/pgstat.o postmaster/postmaster.o postmaster/syslogger.o postmaster/walwriter.o regex/regcomp.o regex/regerror.o regex/regexec.o regex/regfree.o rewrite/rewriteRemove.o rewrite/rewriteDefine.o rewrite/rewriteHandler.o rewrite/rewriteManip.o rewrite/rewriteSupport.o storage/buffer/buf_table.o storage/buffer/buf_init.o storage/buffer/bufmgr.o storage/buffer/freelist.o storage/buffer/localbuf.o storage/file/fd.o storage/file/buffile.o storage/freespace/freespace.o storage/freespace/fsmpage.o storage/freespace/indexfsm.o storage/ipc/ipc.o storage/ipc/ipci.o storage/ipc/pmsignal.o storage/ipc/procarray.o storage/ipc/shmem.o storage/ipc/shmqueue.o storage/ipc/sinval.o storage/ipc/sinvaladt.o storage/large_object/inv_api.o storage/lmgr/lmgr.o storage/lmgr/lock.o storage/lmgr/proc.o storage/lmgr/deadlock.o storage/lmgr/lwlock.o storage/lmgr/spin.o storage/lmgr/s_lock.o storage/page/bufpage.o storage/page/itemptr.o storage/smgr/md.o storage/smgr/smgr.o storage/smgr/smgrtype.o tcop/dest.o tcop/fastpath.o tcop/postgres.o tcop/pquery.o tcop/utility.o tsearch/ts_locale.o tsearch/ts_parse.o tsearch/wparser.o tsearch/wparser_def.o tsearch/dict.o tsearch/dict_simple.o tsearch/dict_synonym.o tsearch/dict_thesaurus.o tsearch/dict_ispell.o tsearch/regis.o tsearch/spell.o tsearch/to_tsany.o tsearch/ts_selfuncs.o tsearch/ts_typanalyze.o tsearch/ts_utils.o utils/adt/acl.o utils/adt/arrayfuncs.o utils/adt/array_userfuncs.o utils/adt/arrayutils.o utils/adt/bool.o utils/adt/cash.o utils/adt/char.o utils/adt/date.o utils/adt/datetime.o utils/adt/datum.o utils/adt/domains.o utils/adt/enum.o utils/adt/float.o utils/adt/format_type.o utils/adt/geo_ops.o utils/adt/geo_selfuncs.o utils/adt/int.o utils/adt/int8.o utils/adt/like.o utils/adt/lockfuncs.o utils/adt/misc.o utils/adt/nabstime.o utils/adt/name.o utils/adt/numeric.o utils/adt/numutils.o utils/adt/oid.o utils/adt/oracle_compat.o utils/adt/pseudotypes.o utils/adt/rowtypes.o utils/adt/regexp.o utils/adt/regproc.o utils/adt/ruleutils.o utils/adt/selfuncs.o utils/adt/tid.o utils/adt/timestamp.o utils/adt/varbit.o utils/adt/varchar.o utils/adt/varlena.o utils/adt/version.o utils/adt/xid.o utils/adt/network.o utils/adt/mac.o utils/adt/inet_net_ntop.o utils/adt/inet_net_pton.o utils/adt/ri_triggers.o utils/adt/pg_lzcompress.o utils/adt/pg_locale.o utils/adt/formatting.o utils/adt/ascii.o utils/adt/quote.o utils/adt/pgstatfuncs.o utils/adt/encode.o utils/adt/dbsize.o utils/adt/genfile.o utils/adt/trigfuncs.o utils/adt/tsginidx.o utils/adt/tsgistidx.o utils/adt/tsquery.o utils/adt/tsquery_cleanup.o utils/adt/tsquery_gist.o utils/adt/tsquery_op.o utils/adt/tsquery_rewrite.o utils/adt/tsquery_util.o utils/adt/tsrank.o utils/adt/tsvector.o utils/adt/tsvector_op.o utils/adt/tsvector_parser.o utils/adt/txid.o utils/adt/uuid.o utils/adt/windowfuncs.o utils/adt/xml.o utils/cache/catcache.o utils/cache/inval.o utils/cache/plancache.o utils/cache/relcache.o utils/cache/syscache.o utils/cache/lsyscache.o utils/cache/typcache.o utils/cache/ts_cache.o utils/error/assert.o utils/error/elog.o utils/fmgr/dfmgr.o utils/fmgr/fmgr.o utils/fmgr/funcapi.o utils/hash/dynahash.o utils/hash/hashfn.o utils/hash/pg_crc.o utils/init/flatfiles.o utils/init/globals.o utils/init/miscinit.o utils/init/postinit.o utils/mb/encnames.o utils/mb/conv.o utils/mb/mbutils.o utils/mb/wchar.o utils/mb/wstrcmp.o utils/mb/wstrncmp.o utils/misc/guc.o utils/misc/help_config.o utils/misc/pg_rusage.o utils/misc/ps_status.o utils/misc/superuser.o utils/misc/tzparser.o utils/mmgr/aset.o utils/mmgr/mcxt.o utils/mmgr/portalmem.o utils/resowner/resowner.o utils/sort/logtape.o utils/sort/tuplesort.o utils/sort/tuplestore.o utils/time/combocid.o utils/time/tqual.o utils/time/snapmgr.o utils/fmgrtab.o ../../src/timezone/localtime.o ../../src/timezone/strftime.o ../../src/timezone/pgtz.o ../../src/port/libpgport_srv.a -lxslt -lxml2 -lpam -lssl -lcrypto -lgssapi_krb5 -lcrypt -ldl -lm -lldap -o postgres
access/transam/clog.o: In function `CheckPointCLOG':
/builddir/build/BUILD/postgresql-8.4.2/src/backend/access/transam/clog.c:545: undefined reference to `clog__checkpoint__start_semaphore'
/builddir/build/BUILD/postgresql-8.4.2/src/backend/access/transam/clog.c:547: undefined reference to `clog__checkpoint__done_semaphore'
access/transam/clog.o: In function `ShutdownCLOG':
/builddir/build/BUILD/postgresql-8.4.2/src/backend/access/transam/clog.c:533: undefined reference to `clog__checkpoint__start_semaphore'
/builddir/build/BUILD/postgresql-8.4.2/src/backend/access/transam/clog.c:535: undefined reference to `clog__checkpoint__done_semaphore'
[... etc ...]
Comment 3 Frank Ch. Eigler 2010-01-20 16:37:42 EST
Tom, was this with the postgres makefile hack that turns off the probe.o 
generation/linking (dtrace -G)?
Comment 4 Frank Ch. Eigler 2010-01-20 17:00:02 EST
Specifically, this bit in src/backend/Makefile et al.

# As of 9/2009:
# * The probes.o file is necessary for dtrace support on Solaris.
# * OS X's dtrace doesn't use it and doesn't even recognize the -G option.
# * Systemtap's dtrace will take -G, but it produces a useless empty file.
# So, build probes.o only on Solaris.
# This will likely need adjustment as other platforms add dtrace support.
ifeq ($(PORTNAME), solaris)
ifeq ($(enable_dtrace), yes)
LOCALOBJS += utils/probes.o
endif
endif


appears to be counterproductive on current systemtap.
Comment 5 Tom Lane 2010-01-20 17:08:08 EST
So we should now do the -G part?  I had gathered from the last conversation that that would be useless for the foreseeable future, but ...
Comment 6 Stan Cox 2010-01-20 17:29:49 EST
The dtrace -G step now creates probe semaphore definitions.  So for example:
provider sdt_misc {
        probe test_probe_0 ();
};
 
Will create with dtrace -G probes.d
a definition 'test_probe_0_semaphore' in probes.o.  This value is used to avoid some of the overhead of setting up the probe when systemtap is not being used.  The value is set by systemtap itself whenever stap is run.  So the use of the dtrace script now more or less parallels the solaris usage,  -G builds probes.o and -h builds probes.h.
Comment 7 Tom Lane 2010-01-20 17:32:41 EST
OK, that makes this a postgres bug not a systemtap bug.  Thanks for your advice.

Oh, one other question: is this version of systemtap going to appear in Fedora 11 and/or 12?
Comment 8 Frank Ch. Eigler 2010-01-20 17:41:41 EST
It's already in F11 and F12.

(Later versions will probably not require -G to make this stuff work,
though should still support it.)
Comment 9 Tom Lane 2010-01-20 17:50:16 EST
Well, given that Solaris has to have the -G file, there's not a lot of advantage to having systemtap not need it.  In any case, please don't break this again ;-)
Comment 10 Mark Wielaard 2010-01-21 03:08:55 EST
Sorry about that. The problem really comes from macos, they don't support the dtrace -G option. We tried to be compatible and support it (the Sun variant doesn't work without it), but indeed in the past it wasn't required (and it would be nice if we didn't require it again in the future). Python for example has the following configure check to detect how to build:

# Check for dtrace support
AC_MSG_CHECKING(for --with-dtrace)
AC_ARG_WITH(dtrace,
            AC_HELP_STRING(--with(out)-dtrace, disable/enable dtrace support))

if test ! -z "$with_dtrace"
then
    if dtrace -G -o /dev/null -s $srcdir/Include/pydtrace.d 2>/dev/null
    then
	AC_DEFINE(WITH_DTRACE, 1, 
	 [Define if you want to compile in Dtrace support])
	with_dtrace="Sun"
	DTRACEOBJS="Python/phelper.o Python/dtrace.o"
	DTRADEHDRS=""
    elif dtrace -h -o /dev/null -s $srcdir/Include/pydtrace.d
    then
	AC_DEFINE(WITH_DTRACE, 1, 
	 [Define if you want to compile in Dtrace support])
	with_dtrace="Apple"
	DTRACEOBJS=""
	DTRADEHDRS="phelper.h pydtrace.h"
    else
	with_dtrace="no"
    fi
else
    with_dtrace="no"
fi

AC_MSG_RESULT($with_dtrace)
AC_SUBST(DTRACEOBJS)
AC_SUBST(DTRACEHDRS)

SystemTap is then detected as the "Sun" variant.
Comment 11 Stan Cox 2010-01-21 09:49:45 EST
The stap testsuite builds postgres with --enable-dtrace and runs the postgres
tests.  I grab it from upstream though because it is so easy to automate:
 wget http://wwwmaster.postgresql.org/...
 tar -x -f postgresql-${postgresrelease}.tar.bz2