Bug 1975438

Created attachment 1793547 [details]
build.log

file build.log too big, will only attach last 32768 bytes

Created attachment 1793548 [details]
root.log

file root.log too big, will only attach last 32768 bytes

Created attachment 1793549 [details]
state.log

This is really weird, the same srpm fails in stream koji: https://kojihub.stream.rdu2.redhat.com/koji/taskinfo?taskID=467646
but succeeds in brew: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=38429894

This is the difference in the buildroot based on root.log:

$ diff -up /tmp/brew /tmp/koji
--- /tmp/brew	2021-07-27 13:23:21.131801655 +0200
+++ /tmp/koji	2021-07-27 13:23:12.363931388 +0200
@@ -10,13 +10,14 @@ bison-3.7.4-4.el9.x86_64
 bzip2-1.0.8-7.el9.x86_64
 bzip2-libs-1.0.8-7.el9.x86_64
 ca-certificates-2020.2.50-92.el9.noarch
+centos-stream-release-9.0-1.0.7.el9.noarch
+centpkg-minimal-2.1.0-2.cs9.noarch
 compat-openssl11-1:1.1.1k-1.el9.x86_64
 coreutils-8.32-30.el9.x86_64
 coreutils-common-8.32-30.el9.x86_64
 cpio-2.13-10.el9.x86_64
 cpp-11.1.1-6.1.el9.x86_64
 cracklib-2.9.6-26.el9.x86_64
-cracklib-dicts-2.9.6-26.el9.x86_64
 crypto-policies-20210707-1.git29f6c0b.el9.noarch
 cryptsetup-libs-2.3.6-2.el9.x86_64
 curl-7.76.1-6.el9.x86_64
@@ -51,6 +52,7 @@ gdbm-libs-1:1.19-3.el9.x86_64
 gettext-0.21-6.el9.x86_64
 gettext-libs-0.21-6.el9.x86_64
 ghc-srpm-macros-1.5.0-5.el9.noarch
+git-core-2.31.1-2.el9.1.x86_64
 glibc-2.33.9000-42.el9.x86_64
 glibc-common-2.33.9000-42.el9.x86_64
 glibc-devel-2.33.9000-42.el9.x86_64
@@ -70,6 +72,7 @@ keyutils-libs-devel-1.6.1-3.el9.x86_64
 kmod-libs-28-4.el9.x86_64
 krb5-devel-1.19.1-10.el9.x86_64
 krb5-libs-1.19.1-10.el9.x86_64
+less-575-3.el9.x86_64
 libacl-2.3.1-2.el9.x86_64
 libarchive-3.5.1-6.el9.x86_64
 libattr-2.5.1-2.el9.x86_64
@@ -77,6 +80,7 @@ libblkid-2.37.1-1.el9.x86_64
 libbrotli-1.0.9-5.el9.x86_64
 libcap-2.48-5.el9.x86_64
 libcap-ng-0.8.2-5.el9.x86_64
+libcbor-0.7.0-4.el9.x86_64
 libcom_err-1.45.6-6.el9.x86_64
 libcom_err-devel-1.45.6-6.el9.x86_64
 libcurl-7.76.1-6.el9.x86_64
@@ -85,6 +89,7 @@ libeconf-0.4.1-1.el9.x86_64
 libedit-3.1-36.20210216cvs.el9.x86_64
 libfdisk-2.37.1-1.el9.x86_64
 libffi-3.1-29.el9.x86_64
+libfido2-1.6.0-6.el9.x86_64
 libgcc-11.1.1-6.1.el9.x86_64
 libgcrypt-1.9.3-3.el9.x86_64
 libgomp-11.1.1-6.1.el9.x86_64
@@ -98,7 +103,7 @@ libpkgconf-1.7.3-8.el9.x86_64
 libpq-13.2-3.el9.x86_64
 libpq-devel-13.2-3.el9.x86_64
 libpsl-0.21.1-4.el9.x86_64
-libpwquality-1.4.4-4.el9.x86_64
+libpwquality-1.4.4-3.el9.x86_64
 libseccomp-2.5.0-5.el9.x86_64
 libselinux-3.2-4.el9.x86_64
 libselinux-devel-3.2-4.el9.x86_64
@@ -134,6 +139,8 @@ ocaml-srpm-macros-6-5.el9.noarch
 openblas-srpm-macros-2-10.el9.noarch
 openldap-2.4.57-7.el9.x86_64
 openldap-devel-2.4.57-7.el9.x86_64
+openssh-8.6p1-5.el9.1.x86_64
+openssh-clients-8.6p1-5.el9.1.x86_64
 openssl-1:3.0.0-0.beta1.4.el9.x86_64
 openssl-devel-1:3.0.0-0.beta1.4.el9.x86_64
 openssl-libs-1:3.0.0-0.beta1.4.el9.x86_64
@@ -156,8 +163,6 @@ publicsuffix-list-dafsa-20210518-1.el9.n
 python-srpm-macros-3.9-41.el9.noarch
 qt5-srpm-macros-5.15.2-8.el9.noarch
 readline-8.1-3.el9.x86_64
-redhat-ca-certificate-1.1.0-2.el8eng.noarch
-redhat-release-9.0-1.8.el9.x86_64
 redhat-rpm-config-185-3.el9.noarch
 rpm-4.16.1.3-3.el9.x86_64
 rpm-build-4.16.1.3-3.el9.x86_64

now scratch build succeed for x86_64, other architectures fail, really weird behavior
https://kojihub.stream.rdu2.redhat.com/koji/taskinfo?taskID=467670

I'm pretty sure this is a dependency issue in the makefile.

One failure has this:

gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64 -march=z14 -mtune=z15 -fasynchronous-unwind-tables -fstack-clash-protection -I../../src/port -DFRONTEND -I../../src/include  -D_GNU_SOURCE   -c -o pg_bitutils.o pg_bitutils.c
make[2]: Leaving directory '/builddir/build/BUILD/postgresql-13.1/src/port'
path.c: In function 'get_lib_path':
path.c:751:38: error: 'LIBDIR' undeclared (first use in this function); did you mean 'EISDIR'?
  751 |         make_relative_path(ret_path, LIBDIR, PGBINDIR, my_exec_path);
      |                                      ^~~~~~
      |                                      EISDIR
path.c:751:38: note: each undeclared identifier is reported only once for each function it appears in
path.c: In function 'get_pkglib_path':
path.c:760:38: error: 'PKGLIBDIR' undeclared (first use in this function); did you mean 'PGBINDIR'?
  760 |         make_relative_path(ret_path, PKGLIBDIR, PGBINDIR, my_exec_path);
      |                                      ^~~~~~~~~
      |                                      PGBINDIR
path.c: In function 'get_locale_path':
path.c:769:38: error: 'LOCALEDIR' undeclared (first use in this function)
  769 |         make_relative_path(ret_path, LOCALEDIR, PGBINDIR, my_exec_path);
      |                                      ^~~~~~~~~
path.c: In function 'get_doc_path':
path.c:778:38: error: 'DOCDIR' undeclared (first use in this function)
  778 |         make_relative_path(ret_path, DOCDIR, PGBINDIR, my_exec_path);
      |                                      ^~~~~~
path.c: In function 'get_html_path':
path.c:787:38: error: 'HTMLDIR' undeclared (first use in this function)
  787 |         make_relative_path(ret_path, HTMLDIR, PGBINDIR, my_exec_path);
      |                                      ^~~~~~~
path.c: In function 'get_man_path':
path.c:796:38: error: 'MANDIR' undeclared (first use in this function); did you mean 'MAXDIM'?
  796 |         make_relative_path(ret_path, MANDIR, PGBINDIR, my_exec_path);
      |                                      ^~~~~~
      |                                      MAXDIM
make[2]: *** [<builtin>: path.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [../../../../src/Makefile.global:569: submake-libpgport] Error 2
make[1]: *** Waiting for unfinished jobs....
make[2]: Entering directory '/builddir/build/BUILD/postgresql-13.1/src/port'

And followed by:

make[2]: Entering directory '/builddir/build/BUILD/postgresql-13.1/src/port'
echo "#define PGBINDIR \"/usr/bin\"" >pg_config_paths.h
echo "#define PGSHAREDIR \"/usr/share/pgsql\"" >>pg_config_paths.h
echo "#define SYSCONFDIR \"/etc\"" >>pg_config_paths.h
echo "#define INCLUDEDIR \"/usr/include\"" >>pg_config_paths.h
echo "#define PKGINCLUDEDIR \"/usr/include/pgsql\"" >>pg_config_paths.h
echo "#define INCLUDEDIRSERVER \"/usr/include/pgsql/server\"" >>pg_config_paths.h
echo "#define LIBDIR \"/usr/lib64\"" >>pg_config_paths.h
echo "#define PKGLIBDIR \"/usr/lib64/pgsql\"" >>pg_config_paths.h
echo "#define LOCALEDIR \"/usr/share/locale\"" >>pg_config_paths.h
echo "#define DOCDIR \"/usr/share/doc//pgsql\"" >>pg_config_paths.h
echo "#define HTMLDIR \"/usr/share/doc//pgsql\"" >>pg_config_paths.h
echo "#define MANDIR \"/usr/share/man\"" >>pg_config_paths.h
make[2]: Leaving directory '/builddir/build/BUILD/postgresql-13.1/src/port'

This means that the writing of the pg_config_paths.h header file finished *after* it was used to compile code.

There is actually a dependency for path.o, but it may be ineffective because the file is written incrementally. This patch should fix it:

--- src/port/Makefile.orig	2021-07-29 16:39:00.918166001 +0200
+++ src/port/Makefile	2021-07-29 16:39:16.381992823 +0200
@@ -144,18 +144,19 @@
 # because many of these values come from makefiles and are not
 # available to configure.
 pg_config_paths.h: $(top_builddir)/src/Makefile.global
-	echo "#define PGBINDIR \"$(bindir)\"" >$@
-	echo "#define PGSHAREDIR \"$(datadir)\"" >>$@
-	echo "#define SYSCONFDIR \"$(sysconfdir)\"" >>$@
-	echo "#define INCLUDEDIR \"$(includedir)\"" >>$@
-	echo "#define PKGINCLUDEDIR \"$(pkgincludedir)\"" >>$@
-	echo "#define INCLUDEDIRSERVER \"$(includedir_server)\"" >>$@
-	echo "#define LIBDIR \"$(libdir)\"" >>$@
-	echo "#define PKGLIBDIR \"$(pkglibdir)\"" >>$@
-	echo "#define LOCALEDIR \"$(localedir)\"" >>$@
-	echo "#define DOCDIR \"$(docdir)\"" >>$@
-	echo "#define HTMLDIR \"$(htmldir)\"" >>$@
-	echo "#define MANDIR \"$(mandir)\"" >>$@
+	echo "#define PGBINDIR \"$(bindir)\"" >$@T
+	echo "#define PGSHAREDIR \"$(datadir)\"" >>$@T
+	echo "#define SYSCONFDIR \"$(sysconfdir)\"" >>$@T
+	echo "#define INCLUDEDIR \"$(includedir)\"" >>$@T
+	echo "#define PKGINCLUDEDIR \"$(pkgincludedir)\"" >>$@T
+	echo "#define INCLUDEDIRSERVER \"$(includedir_server)\"" >>$@T
+	echo "#define LIBDIR \"$(libdir)\"" >>$@T
+	echo "#define PKGLIBDIR \"$(pkglibdir)\"" >>$@T
+	echo "#define LOCALEDIR \"$(localedir)\"" >>$@T
+	echo "#define DOCDIR \"$(docdir)\"" >>$@T
+	echo "#define HTMLDIR \"$(htmldir)\"" >>$@T
+	echo "#define MANDIR \"$(mandir)\"" >>$@T
+	mv $@T $@
 
 clean distclean maintainer-clean:
 	rm -f libpgport.a libpgport_shlib.a libpgport_srv.a

I saw another build failure that suggests a completely different parallel build failure:

make[2]: Leaving directory '/builddir/build/BUILD/postgresql-13.1/src/common'
make[2]: Entering directory '/builddir/build/BUILD/postgresql-13.1/src/common'
rm -f libpgcommon_shlib.a
ar crs libpgcommon_shlib.a archive_shlib.o base64_shlib.o checksum_helper_shlib.o config_info_shlib.o controldata_utils_shlib.o d2s_shlib.o encnames_shlib.o exec_shlib.o f2s_shlib.o file_perm_shlib.o hashfn_shlib.o ip_shlib.o jsonapi_shlib.o keywords_shlib.o kwlookup_shlib.o link-canary_shlib.o md5_shlib.o pg_lzcompress_shlib.o pgfnames_shlib.o psprintf_shlib.o relpath_shlib.o rmtree_shlib.o saslprep_shlib.o scram-common_shlib.o string_shlib.o stringinfo_shlib.o unicode_norm_shlib.o username_shlib.o wait_error_shlib.o wchar_shlib.o protocol_openssl_shlib.o sha2_openssl_shlib.o fe_memutils_shlib.o file_utils_shlib.o logging_shlib.o restricted_token_shlib.o
make[2]: Leaving directory '/builddir/build/BUILD/postgresql-13.1/src/common'
make[1]: Entering directory '/builddir/build/BUILD/postgresql-13.1/src/interfaces/ecpg/pgtypeslib'
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -fasynchronous-unwind-tables -fstack-clash-protection -pthread -D_REENTRANT -D_THREAD_SAFE -D_POSIX_PTHREAD_SEMANTICS -fPIC -shared -Wl,-soname,libpgtypes.so.3 -Wl,--version-script=exports.list -o libpgtypes.so.3.13  common.o datetime.o dt_common.o interval.o numeric.o timestamp.o -L../../../../src/port -L../../../../src/common -lpgcommon_shlib -lpgport_shlib -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1   -Wl,--as-needed  -lm
make[1]: Leaving directory '/builddir/build/BUILD/postgresql-13.1/src/interfaces/ecpg/pgtypeslib'
/usr/bin/ld: ../../../../src/common/libpgcommon_shlib.a: error adding symbols: no more archived files
collect2: error: ld returned 1 exit status

This obscure ld error suggests that the archive file was modified while being read, I think.

So adding -j1 is probably the right way to deal with this.

In the compose, there is an actual build, so it shouldn't block RHEL9-beta.

(In reply to Filip Januš from comment #10)
> In the compose, there is an actual build, so it shouldn't block RHEL9-beta.

I'm sorry, this build is outdated and does not use the correct compiler flag and build environment. We need a new build, I'm afraid.

As a stop-gap measure, you could consider using make -j1.

Yes, we should have done it after the first mass rebuild.

Prepared PR: https://gitlab.com/redhat/centos-stream/rpms/libecpg/-/merge_requests/3/diffs

I think -j1 is needed for the %build stage as well.

Definitely one more PR: https://gitlab.com/redhat/centos-stream/rpms/libecpg/-/merge_requests/5

*** Bug 1988968 has been marked as a duplicate of this bug. ***

Thanks! Build seems to be stable now.