Bug 557266

Stan, maybe this rings a bell for you? This is because references to the semaphores are missing:

[...]
make[3]: Entering directory `/builddir/build/BUILD/postgresql-8.4.2/src/backend/utils'
dtrace -C -h -s probes.d -o probes.h.tmp
sed -e 's/POSTGRESQL_/TRACE_POSTGRESQL_/g' probes.h.tmp >probes.h
rm probes.h.tmp
make[3]: Leaving directory `/builddir/build/BUILD/postgresql-8.4.2/src/backend/utils'
cd ../../src/include/utils/ && rm -f probes.h && \
	    ln -s ../../../src/backend/utils/probes.h .
make[3]: Leaving directory `/builddir/build/BUILD/postgresql-8.4.2/src/backend/utils'
[...]
gcc -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -DLINUX_OOM_ADJ=0 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -fno-strict-aliasing -fwrapv -L../../src/port -Wl,--as-needed -Wl,-E access/common/heaptuple.o access/common/indextuple.o access/common/printtup.o access/common/reloptions.o access/common/scankey.o access/common/tupdesc.o access/gist/gist.o access/gist/gistutil.o access/gist/gistxlog.o access/gist/gistvacuum.o access/gist/gistget.o access/gist/gistscan.o access/gist/gistproc.o access/gist/gistsplit.o access/hash/hash.o access/hash/hashfunc.o access/hash/hashinsert.o access/hash/hashovfl.o access/hash/hashpage.o access/hash/hashscan.o access/hash/hashsearch.o access/hash/hashsort.o access/hash/hashutil.o access/heap/heapam.o access/heap/hio.o access/heap/pruneheap.o access/heap/rewriteheap.o access/heap/syncscan.o access/heap/tuptoaster.o access/heap/visibilitymap.o access/index/genam.o access/index/indexam.o access/nbtree/nbtcompare.o access/nbtree/nbtinsert.o access/nbtree/nbtpage.o access/nbtree/nbtree.o access/nbtree/nbtsearch.o access/nbtree/nbtutils.o access/nbtree/nbtsort.o access/nbtree/nbtxlog.o access/transam/clog.o access/transam/transam.o access/transam/varsup.o access/transam/xact.o access/transam/xlog.o access/transam/xlogutils.o access/transam/rmgr.o access/transam/slru.o access/transam/subtrans.o access/transam/multixact.o access/transam/twophase.o access/transam/twophase_rmgr.o access/gin/ginutil.o access/gin/gininsert.o access/gin/ginxlog.o access/gin/ginentrypage.o access/gin/gindatapage.o access/gin/ginbtree.o access/gin/ginscan.o access/gin/ginget.o access/gin/ginvacuum.o access/gin/ginarrayproc.o access/gin/ginbulk.o access/gin/ginfast.o bootstrap/bootparse.o bootstrap/bootstrap.o catalog/catalog.o catalog/dependency.o catalog/heap.o catalog/index.o catalog/indexing.o catalog/namespace.o catalog/aclchk.o catalog/pg_aggregate.o catalog/pg_constraint.o catalog/pg_conversion.o catalog/pg_depend.o catalog/pg_enum.o catalog/pg_inherits.o catalog/pg_largeobject.o catalog/pg_namespace.o catalog/pg_operator.o catalog/pg_proc.o catalog/pg_shdepend.o catalog/pg_type.o catalog/storage.o catalog/toasting.o parser/analyze.o parser/gram.o parser/keywords.o parser/parser.o parser/parse_agg.o parser/parse_cte.o parser/parse_clause.o parser/parse_expr.o parser/parse_func.o parser/parse_node.o parser/parse_oper.o parser/parse_relation.o parser/parse_type.o parser/parse_coerce.o parser/parse_target.o parser/parse_utilcmd.o parser/scansup.o parser/kwlookup.o commands/aggregatecmds.o commands/alter.o commands/analyze.o commands/async.o commands/cluster.o commands/comment.o commands/conversioncmds.o commands/copy.o commands/dbcommands.o commands/define.o commands/discard.o commands/explain.o commands/foreigncmds.o commands/functioncmds.o commands/indexcmds.o commands/lockcmds.o commands/operatorcmds.o commands/opclasscmds.o commands/portalcmds.o commands/prepare.o commands/proclang.o commands/schemacmds.o commands/sequence.o commands/tablecmds.o commands/tablespace.o commands/trigger.o commands/tsearchcmds.o commands/typecmds.o commands/user.o commands/vacuum.o commands/vacuumlazy.o commands/variable.o commands/view.o executor/execAmi.o executor/execCurrent.o executor/execGrouping.o executor/execJunk.o executor/execMain.o executor/execProcnode.o executor/execQual.o executor/execScan.o executor/execTuples.o executor/execUtils.o executor/functions.o executor/instrument.o executor/nodeAppend.o executor/nodeAgg.o executor/nodeBitmapAnd.o executor/nodeBitmapOr.o executor/nodeBitmapHeapscan.o executor/nodeBitmapIndexscan.o executor/nodeHash.o executor/nodeHashjoin.o executor/nodeIndexscan.o executor/nodeMaterial.o executor/nodeMergejoin.o executor/nodeNestloop.o executor/nodeFunctionscan.o executor/nodeRecursiveunion.o executor/nodeResult.o executor/nodeSeqscan.o executor/nodeSetOp.o executor/nodeSort.o executor/nodeUnique.o executor/nodeValuesscan.o executor/nodeCtescan.o executor/nodeWorktablescan.o executor/nodeLimit.o executor/nodeGroup.o executor/nodeSubplan.o executor/nodeSubqueryscan.o executor/nodeTidscan.o executor/nodeWindowAgg.o executor/tstoreReceiver.o executor/spi.o foreign/foreign.o lib/dllist.o lib/stringinfo.o libpq/be-fsstubs.o libpq/be-secure.o libpq/auth.o libpq/crypt.o libpq/hba.o libpq/ip.o libpq/md5.o libpq/pqcomm.o libpq/pqformat.o libpq/pqsignal.o main/main.o nodes/nodeFuncs.o nodes/nodes.o nodes/list.o nodes/bitmapset.o nodes/tidbitmap.o nodes/copyfuncs.o nodes/equalfuncs.o nodes/makefuncs.o nodes/outfuncs.o nodes/readfuncs.o nodes/print.o nodes/read.o nodes/params.o nodes/value.o optimizer/geqo/geqo_copy.o optimizer/geqo/geqo_eval.o optimizer/geqo/geqo_main.o optimizer/geqo/geqo_misc.o optimizer/geqo/geqo_mutation.o optimizer/geqo/geqo_pool.o optimizer/geqo/geqo_recombination.o optimizer/geqo/geqo_selection.o optimizer/geqo/geqo_erx.o optimizer/geqo/geqo_pmx.o optimizer/geqo/geqo_cx.o optimizer/geqo/geqo_px.o optimizer/geqo/geqo_ox1.o optimizer/geqo/geqo_ox2.o optimizer/path/allpaths.o optimizer/path/clausesel.o optimizer/path/costsize.o optimizer/path/equivclass.o optimizer/path/indxpath.o optimizer/path/joinpath.o optimizer/path/joinrels.o optimizer/path/orindxpath.o optimizer/path/pathkeys.o optimizer/path/tidpath.o optimizer/plan/createplan.o optimizer/plan/initsplan.o optimizer/plan/planagg.o optimizer/plan/planmain.o optimizer/plan/planner.o optimizer/plan/setrefs.o optimizer/plan/subselect.o optimizer/prep/prepjointree.o optimizer/prep/prepqual.o optimizer/prep/preptlist.o optimizer/prep/prepunion.o optimizer/util/clauses.o optimizer/util/joininfo.o optimizer/util/pathnode.o optimizer/util/placeholder.o optimizer/util/plancat.o optimizer/util/predtest.o optimizer/util/relnode.o optimizer/util/restrictinfo.o optimizer/util/tlist.o optimizer/util/var.o port/dynloader.o port/pg_sema.o port/pg_shmem.o postmaster/autovacuum.o postmaster/bgwriter.o postmaster/fork_process.o postmaster/pgarch.o postmaster/pgstat.o postmaster/postmaster.o postmaster/syslogger.o postmaster/walwriter.o regex/regcomp.o regex/regerror.o regex/regexec.o regex/regfree.o rewrite/rewriteRemove.o rewrite/rewriteDefine.o rewrite/rewriteHandler.o rewrite/rewriteManip.o rewrite/rewriteSupport.o storage/buffer/buf_table.o storage/buffer/buf_init.o storage/buffer/bufmgr.o storage/buffer/freelist.o storage/buffer/localbuf.o storage/file/fd.o storage/file/buffile.o storage/freespace/freespace.o storage/freespace/fsmpage.o storage/freespace/indexfsm.o storage/ipc/ipc.o storage/ipc/ipci.o storage/ipc/pmsignal.o storage/ipc/procarray.o storage/ipc/shmem.o storage/ipc/shmqueue.o storage/ipc/sinval.o storage/ipc/sinvaladt.o storage/large_object/inv_api.o storage/lmgr/lmgr.o storage/lmgr/lock.o storage/lmgr/proc.o storage/lmgr/deadlock.o storage/lmgr/lwlock.o storage/lmgr/spin.o storage/lmgr/s_lock.o storage/page/bufpage.o storage/page/itemptr.o storage/smgr/md.o storage/smgr/smgr.o storage/smgr/smgrtype.o tcop/dest.o tcop/fastpath.o tcop/postgres.o tcop/pquery.o tcop/utility.o tsearch/ts_locale.o tsearch/ts_parse.o tsearch/wparser.o tsearch/wparser_def.o tsearch/dict.o tsearch/dict_simple.o tsearch/dict_synonym.o tsearch/dict_thesaurus.o tsearch/dict_ispell.o tsearch/regis.o tsearch/spell.o tsearch/to_tsany.o tsearch/ts_selfuncs.o tsearch/ts_typanalyze.o tsearch/ts_utils.o utils/adt/acl.o utils/adt/arrayfuncs.o utils/adt/array_userfuncs.o utils/adt/arrayutils.o utils/adt/bool.o utils/adt/cash.o utils/adt/char.o utils/adt/date.o utils/adt/datetime.o utils/adt/datum.o utils/adt/domains.o utils/adt/enum.o utils/adt/float.o utils/adt/format_type.o utils/adt/geo_ops.o utils/adt/geo_selfuncs.o utils/adt/int.o utils/adt/int8.o utils/adt/like.o utils/adt/lockfuncs.o utils/adt/misc.o utils/adt/nabstime.o utils/adt/name.o utils/adt/numeric.o utils/adt/numutils.o utils/adt/oid.o utils/adt/oracle_compat.o utils/adt/pseudotypes.o utils/adt/rowtypes.o utils/adt/regexp.o utils/adt/regproc.o utils/adt/ruleutils.o utils/adt/selfuncs.o utils/adt/tid.o utils/adt/timestamp.o utils/adt/varbit.o utils/adt/varchar.o utils/adt/varlena.o utils/adt/version.o utils/adt/xid.o utils/adt/network.o utils/adt/mac.o utils/adt/inet_net_ntop.o utils/adt/inet_net_pton.o utils/adt/ri_triggers.o utils/adt/pg_lzcompress.o utils/adt/pg_locale.o utils/adt/formatting.o utils/adt/ascii.o utils/adt/quote.o utils/adt/pgstatfuncs.o utils/adt/encode.o utils/adt/dbsize.o utils/adt/genfile.o utils/adt/trigfuncs.o utils/adt/tsginidx.o utils/adt/tsgistidx.o utils/adt/tsquery.o utils/adt/tsquery_cleanup.o utils/adt/tsquery_gist.o utils/adt/tsquery_op.o utils/adt/tsquery_rewrite.o utils/adt/tsquery_util.o utils/adt/tsrank.o utils/adt/tsvector.o utils/adt/tsvector_op.o utils/adt/tsvector_parser.o utils/adt/txid.o utils/adt/uuid.o utils/adt/windowfuncs.o utils/adt/xml.o utils/cache/catcache.o utils/cache/inval.o utils/cache/plancache.o utils/cache/relcache.o utils/cache/syscache.o utils/cache/lsyscache.o utils/cache/typcache.o utils/cache/ts_cache.o utils/error/assert.o utils/error/elog.o utils/fmgr/dfmgr.o utils/fmgr/fmgr.o utils/fmgr/funcapi.o utils/hash/dynahash.o utils/hash/hashfn.o utils/hash/pg_crc.o utils/init/flatfiles.o utils/init/globals.o utils/init/miscinit.o utils/init/postinit.o utils/mb/encnames.o utils/mb/conv.o utils/mb/mbutils.o utils/mb/wchar.o utils/mb/wstrcmp.o utils/mb/wstrncmp.o utils/misc/guc.o utils/misc/help_config.o utils/misc/pg_rusage.o utils/misc/ps_status.o utils/misc/superuser.o utils/misc/tzparser.o utils/mmgr/aset.o utils/mmgr/mcxt.o utils/mmgr/portalmem.o utils/resowner/resowner.o utils/sort/logtape.o utils/sort/tuplesort.o utils/sort/tuplestore.o utils/time/combocid.o utils/time/tqual.o utils/time/snapmgr.o utils/fmgrtab.o ../../src/timezone/localtime.o ../../src/timezone/strftime.o ../../src/timezone/pgtz.o ../../src/port/libpgport_srv.a -lxslt -lxml2 -lpam -lssl -lcrypto -lgssapi_krb5 -lcrypt -ldl -lm -lldap -o postgres
access/transam/clog.o: In function `CheckPointCLOG':
/builddir/build/BUILD/postgresql-8.4.2/src/backend/access/transam/clog.c:545: undefined reference to `clog__checkpoint__start_semaphore'
/builddir/build/BUILD/postgresql-8.4.2/src/backend/access/transam/clog.c:547: undefined reference to `clog__checkpoint__done_semaphore'
access/transam/clog.o: In function `ShutdownCLOG':
/builddir/build/BUILD/postgresql-8.4.2/src/backend/access/transam/clog.c:533: undefined reference to `clog__checkpoint__start_semaphore'
/builddir/build/BUILD/postgresql-8.4.2/src/backend/access/transam/clog.c:535: undefined reference to `clog__checkpoint__done_semaphore'
[... etc ...]

Tom, was this with the postgres makefile hack that turns off the probe.o 
generation/linking (dtrace -G)?

Specifically, this bit in src/backend/Makefile et al.

# As of 9/2009:
# * The probes.o file is necessary for dtrace support on Solaris.
# * OS X's dtrace doesn't use it and doesn't even recognize the -G option.
# * Systemtap's dtrace will take -G, but it produces a useless empty file.
# So, build probes.o only on Solaris.
# This will likely need adjustment as other platforms add dtrace support.
ifeq ($(PORTNAME), solaris)
ifeq ($(enable_dtrace), yes)
LOCALOBJS += utils/probes.o
endif
endif


appears to be counterproductive on current systemtap.

So we should now do the -G part?  I had gathered from the last conversation that that would be useless for the foreseeable future, but ...

The dtrace -G step now creates probe semaphore definitions.  So for example:
provider sdt_misc {
        probe test_probe_0 ();
};
 
Will create with dtrace -G probes.d
a definition 'test_probe_0_semaphore' in probes.o.  This value is used to avoid some of the overhead of setting up the probe when systemtap is not being used.  The value is set by systemtap itself whenever stap is run.  So the use of the dtrace script now more or less parallels the solaris usage,  -G builds probes.o and -h builds probes.h.

OK, that makes this a postgres bug not a systemtap bug.  Thanks for your advice.

Oh, one other question: is this version of systemtap going to appear in Fedora 11 and/or 12?

It's already in F11 and F12.

(Later versions will probably not require -G to make this stuff work,
though should still support it.)

Well, given that Solaris has to have the -G file, there's not a lot of advantage to having systemtap not need it.  In any case, please don't break this again ;-)

Sorry about that. The problem really comes from macos, they don't support the dtrace -G option. We tried to be compatible and support it (the Sun variant doesn't work without it), but indeed in the past it wasn't required (and it would be nice if we didn't require it again in the future). Python for example has the following configure check to detect how to build:

# Check for dtrace support
AC_MSG_CHECKING(for --with-dtrace)
AC_ARG_WITH(dtrace,
            AC_HELP_STRING(--with(out)-dtrace, disable/enable dtrace support))

if test ! -z "$with_dtrace"
then
    if dtrace -G -o /dev/null -s $srcdir/Include/pydtrace.d 2>/dev/null
    then
	AC_DEFINE(WITH_DTRACE, 1, 
	 [Define if you want to compile in Dtrace support])
	with_dtrace="Sun"
	DTRACEOBJS="Python/phelper.o Python/dtrace.o"
	DTRADEHDRS=""
    elif dtrace -h -o /dev/null -s $srcdir/Include/pydtrace.d
    then
	AC_DEFINE(WITH_DTRACE, 1, 
	 [Define if you want to compile in Dtrace support])
	with_dtrace="Apple"
	DTRACEOBJS=""
	DTRADEHDRS="phelper.h pydtrace.h"
    else
	with_dtrace="no"
    fi
else
    with_dtrace="no"
fi

AC_MSG_RESULT($with_dtrace)
AC_SUBST(DTRACEOBJS)
AC_SUBST(DTRACEHDRS)

SystemTap is then detected as the "Sun" variant.

The stap testsuite builds postgres with --enable-dtrace and runs the postgres
tests.  I grab it from upstream though because it is so easy to automate:
 wget http://wwwmaster.postgresql.org/...
 tar -x -f postgresql-${postgresrelease}.tar.bz2