Description of problem: It looks to me like something similar to bug 1467299 is reproducible with mock-3.5-1.fc37 when building in koji (for rawhide). Version-Release number of selected component (if applicable): mock-3.5-1.fc37 How reproducible: 100% Steps to Reproduce: 1. Try to rebuild ghc8.10-8.10.7-10.fc38 in mock ie against itself. Actual results: root.log contains: : DEBUG util.py:445: Running transaction DEBUG util.py:445: /usr/lib64/ghc-8.10.7/bin/ghc-pkg: error while loading shared libraries: libHSterminfo-0.4.1.4-ghc8.10.7.so: cannot open shared object file: No such file or directory DEBUG util.py:445: Installed: : Expected results: No error Additional info: Note unlike ghc8.10-8.10.7-6.fc38 and ghc-8.10.7-123.fc38, ghc8.10-8.10.7-10.fc38 uses RPATH's for it's shared libraries: $ chrpath /usr/lib64/ghc-8.10.7/bin/ghc-pkg /usr/lib64/ghc-8.10.7/bin/ghc-pkg: RUNPATH=$ORIGIN/../terminfo-0.4.1.4:$ORIGIN/../ghc-boot-8.10.7:$ORIGIN/../Cabal-3.2.1.0:$ORIGIN/../process-1.6.13.2:$ORIGIN/../parsec-3.1.14.0:$ORIGIN/../text-1.2.4.1:$ORIGIN/../template-haskell-2.16.0.0:$ORIGIN/../pretty-1.1.3.6:$ORIGIN/../ghc-boot-th-8.10.7:$ORIGIN/../mtl-2.2.2:$ORIGIN/../transformers-0.5.6.2:$ORIGIN/../directory-1.3.6.0:$ORIGIN/../unix-2.7.2.2:$ORIGIN/../time-1.9.3:$ORIGIN/../filepath-1.4.2.1:$ORIGIN/../binary-0.8.8.0:$ORIGIN/../containers-0.6.5.1:$ORIGIN/../bytestring-0.10.12.0:$ORIGIN/../deepseq-1.4.4.0:$ORIGIN/../array-0.5.4.0:$ORIGIN/../base-4.14.3.0:$ORIGIN/../integer-gmp-1.0.3.0:$ORIGIN/../ghc-prim-0.6.1:$ORIGIN/../rts which seems to be causing problems under systemd-nspawn mock again. ghc8.10-8.10.7-10.fc38 works fine for me locally.
(Just noting that currently ghc8.10-8.10.7-10.fc38 is only available in a koji sidetag) Actually same happens when building with ghc9.0 which does not use RPATHs. Maybe the filetriggers for ghc*-devel are not getting run properly? I also remember that in eg https://copr.fedorainfracloud.org/coprs/petersen/haskell-language-server/ I had to use simple chroot instead of systemd-nspawn in order to use ghcX.Y, like here.
Here is another example: https://koji.fedoraproject.org/koji/taskinfo?taskID=96927426 The first error is in root.log: DEBUG util.py:445: /usr/lib64/ghc-9.2.5/bin/ghc-pkg-9.2.5: error while loading shared libraries: libHSterminfo-0.4.1.5-ghc9.2.5.so: cannot open shared object file: No such file or directory
Unfortunately this is blocking my work on https://fedoraproject.org/wiki/Changes/Haskell_GHC_9.2_and_Stackage_20 : Though I realised though this problem has existed for some time: likely since well before https://fedoraproject.org/wiki/Changes/GHC_parallel_version_installs (F36)
This seems to be a problem with installation order. Locally, I get the following: $ koji download-build -a x86_64 -a noarch ghc8.10-8.10.7-10.fc38 $ mock --clean && mock -i ghc8.10-* ... Running transaction Preparing : 1/1 Installing : ghc8.10-base-4.14.3.0-10.fc38.x86_64 1/151 Installing : ghc8.10-array-0.5.4.0-10.fc38.x86_64 2/151 Installing : ghc8.10-deepseq-1.4.4.0-10.fc38.x86_64 3/151 Installing : ghc8.10-filesystem-8.10.7-10.fc38.noarch 4/151 Installing : ghc8.10-bytestring-0.10.12.0-10.fc38.x86_64 5/151 Installing : ghc8.10-containers-0.6.5.1-10.fc38.x86_64 6/151 Installing : ghc8.10-time-1.9.3-10.fc38.x86_64 7/151 Installing : ghc8.10-ghc-boot-th-8.10.7-10.fc38.x86_64 8/151 Installing : ghc8.10-unix-2.7.2.2-10.fc38.x86_64 9/151 Installing : ghc8.10-pretty-1.1.3.6-10.fc38.x86_64 10/151 Installing : ghc8.10-filepath-1.4.2.1-10.fc38.x86_64 11/151 Installing : ghc8.10-directory-1.3.6.0-10.fc38.x86_64 12/151 Installing : ghc8.10-template-haskell-2.16.0.0-10.fc38.x86_64 13/151 Installing : ghc8.10-transformers-0.5.6.2-10.fc38.x86_64 14/151 Installing : ghc8.10-binary-0.8.8.0-10.fc38.x86_64 15/151 Installing : ghc8.10-mtl-2.2.2-10.fc38.x86_64 16/151 Installing : ghc8.10-ghc-boot-8.10.7-10.fc38.x86_64 17/151 Installing : ghc8.10-process-1.6.13.2-10.fc38.x86_64 18/151 Installing : ghc8.10-ghc-heap-8.10.7-10.fc38.x86_64 19/151 Installing : ghc8.10-ghci-8.10.7-10.fc38.x86_64 20/151 Installing : ghc8.10-text-1.2.4.1-10.fc38.x86_64 21/151 Installing : ghc8.10-stm-2.5.0.1-10.fc38.x86_64 22/151 Installing : ghc8.10-terminfo-0.4.1.4-10.fc38.x86_64 23/151 Installing : ghc8.10-exceptions-0.10.4-10.fc38.x86_64 24/151 Installing : ghc8.10-parsec-3.1.14.0-10.fc38.x86_64 25/151 Installing : ghc8.10-hpc-0.6.1.0-10.fc38.x86_64 26/151 Installing : ghc8.10-ghc-8.10.7-10.fc38.x86_64 27/151 Installing : ghc8.10-Cabal-3.2.1.0-10.fc38.x86_64 28/151 Installing : ghc8.10-haskeline-0.8.2-10.fc38.x86_64 29/151 Installing : ghc8.10-libiserv-8.10.7-10.fc38.x86_64 30/151 Installing : ghc8.10-array-doc-0.5.4.0-10.fc38.noarch 31/151 Installing : ghc8.10-base-doc-4.14.3.0-10.fc38.noarch 32/151 Installing : ghc8.10-binary-doc-0.8.8.0-10.fc38.noarch 33/151 Installing : ghc8.10-bytestring-doc-0.10.12.0-10.fc38.noarch 34/151 Installing : ghc8.10-Cabal-doc-3.2.1.0-10.fc38.noarch 35/151 Installing : ghc8.10-containers-doc-0.6.5.1-10.fc38.noarch 36/151 Installing : ghc8.10-deepseq-doc-1.4.4.0-10.fc38.noarch 37/151 Installing : ghc8.10-directory-doc-1.3.6.0-10.fc38.noarch 38/151 Installing : ghc8.10-exceptions-doc-0.10.4-10.fc38.noarch 39/151 Installing : ghc8.10-filepath-doc-1.4.2.1-10.fc38.noarch 40/151 Installing : ghc8.10-ghc-boot-doc-8.10.7-10.fc38.noarch 41/151 Installing : ghc8.10-ghc-boot-th-doc-8.10.7-10.fc38.noarch 42/151 Installing : ghc8.10-ghc-compact-doc-0.1.0.0-10.fc38.noarch 43/151 Installing : ghc8.10-ghc-doc-8.10.7-10.fc38.noarch 44/151 Installing : ghc8.10-ghc-heap-doc-8.10.7-10.fc38.noarch 45/151 Installing : ghc8.10-ghci-doc-8.10.7-10.fc38.noarch 46/151 Installing : ghc8.10-haskeline-doc-0.8.2-10.fc38.noarch 47/151 Installing : ghc8.10-hpc-doc-0.6.1.0-10.fc38.noarch 48/151 Installing : ghc8.10-libiserv-doc-8.10.7-10.fc38.noarch 49/151 Installing : ghc8.10-mtl-doc-2.2.2-10.fc38.noarch 50/151 Installing : ghc8.10-parsec-doc-3.1.14.0-10.fc38.noarch 51/151 Installing : ghc8.10-pretty-doc-1.1.3.6-10.fc38.noarch 52/151 Installing : ghc8.10-process-doc-1.6.13.2-10.fc38.noarch 53/151 Installing : ghc8.10-stm-doc-2.5.0.1-10.fc38.noarch 54/151 Installing : ghc8.10-template-haskell-doc-2.16.0.0-10.fc38.noarch 55/151 Installing : ghc8.10-terminfo-doc-0.4.1.4-10.fc38.noarch 56/151 Installing : ghc8.10-text-doc-1.2.4.1-10.fc38.noarch 57/151 Installing : ghc8.10-time-doc-1.9.3-10.fc38.noarch 58/151 Installing : ghc8.10-transformers-doc-0.5.6.2-10.fc38.noarch 59/151 Installing : ghc8.10-unix-doc-2.7.2.2-10.fc38.noarch 60/151 Installing : ghc8.10-xhtml-doc-3000.2.2.1-10.fc38.noarch 61/151 Installing : ghc8.10-xhtml-3000.2.2.1-10.fc38.x86_64 62/151 Installing : libmpc-1.2.1-5.fc37.x86_64 63/151 Installing : cpp-12.2.1-4.fc37.x86_64 64/151 Installing : ghc8.10-doc-8.10.7-10.fc38.x86_64 65/151 Installing : ghc8.10-ghc-compact-0.1.0.0-10.fc38.x86_64 66/151 Installing : ghc8.10-manual-8.10.7-10.fc38.noarch 67/151 Installing : ncurses-c++-libs-6.3-4.20220501.fc37.x86_64 68/151 Installing : ncurses-devel-6.3-4.20220501.fc37.x86_64 69/151 Installing : kernel-headers-6.1.5-200.fc37.x86_64 70/151 Installing : glibc-headers-x86-2.36-9.fc37.noarch 71/151 Installing : libxcrypt-devel-4.4.33-4.fc37.x86_64 72/151 Installing : glibc-devel-2.36-9.fc37.x86_64 73/151 Installing : annobin-docs-10.92-1.fc37.noarch 74/151 Installing : libtool-ltdl-2.4.7-2.fc37.x86_64 75/151 Installing : gmp-c++-1:6.2.1-3.fc37.x86_64 76/151 Installing : gmp-devel-1:6.2.1-3.fc37.x86_64 77/151 Installing : gc-8.0.6-4.fc37.x86_64 78/151 Installing : guile22-2.2.7-6.fc37.x86_64 79/151 Installing : make-1:4.3-11.fc37.x86_64 80/151 Installing : gcc-12.2.1-4.fc37.x86_64 81/151 Running scriptlet: gcc-12.2.1-4.fc37.x86_64 81/151 Installing : ghc8.10-base-devel-4.14.3.0-10.fc38.x86_64 82/151 Installing : ghc8.10-compiler-8.10.7-10.fc38.x86_64 83/151 Installing : ghc8.10-base-prof-4.14.3.0-10.fc38.x86_64 84/151 Installing : ghc8.10-array-devel-0.5.4.0-10.fc38.x86_64 85/151 Installing : ghc8.10-deepseq-devel-1.4.4.0-10.fc38.x86_64 86/151 Installing : ghc8.10-bytestring-devel-0.10.12.0-10.fc38.x86_64 87/151 Installing : ghc8.10-containers-devel-0.6.5.1-10.fc38.x86_64 88/151 Installing : ghc8.10-filepath-devel-1.4.2.1-10.fc38.x86_64 89/151 Installing : ghc8.10-filepath-prof-1.4.2.1-10.fc38.x86_64 90/151 Installing : ghc8.10-array-prof-0.5.4.0-10.fc38.x86_64 91/151 Installing : ghc8.10-deepseq-prof-1.4.4.0-10.fc38.x86_64 92/151 Installing : ghc8.10-bytestring-prof-0.10.12.0-10.fc38.x86_64 93/151 Installing : ghc8.10-containers-prof-0.6.5.1-10.fc38.x86_64 94/151 Installing : ghc8.10-binary-devel-0.8.8.0-10.fc38.x86_64 95/151 Installing : ghc8.10-transformers-devel-0.5.6.2-10.fc38.x86_64 96/151 Installing : ghc8.10-transformers-prof-0.5.6.2-10.fc38.x86_64 97/151 Installing : ghc8.10-binary-prof-0.8.8.0-10.fc38.x86_64 98/151 Installing : ghc8.10-time-devel-1.9.3-10.fc38.x86_64 99/151 Installing : ghc8.10-unix-devel-2.7.2.2-10.fc38.x86_64 100/151 Installing : ghc8.10-directory-devel-1.3.6.0-10.fc38.x86_64 101/151 Installing : ghc8.10-time-prof-1.9.3-10.fc38.x86_64 102/151 Installing : ghc8.10-unix-prof-2.7.2.2-10.fc38.x86_64 103/151 Installing : ghc8.10-directory-prof-1.3.6.0-10.fc38.x86_64 104/151 Installing : ghc8.10-ghc-boot-th-devel-8.10.7-10.fc38.x86_64 105/151 Installing : ghc8.10-ghc-boot-th-prof-8.10.7-10.fc38.x86_64 106/151 Installing : ghc8.10-process-devel-1.6.13.2-10.fc38.x86_64 107/151 Installing : ghc8.10-mtl-devel-2.2.2-10.fc38.x86_64 108/151 Installing : ghc8.10-mtl-prof-2.2.2-10.fc38.x86_64 109/151 Installing : ghc8.10-process-prof-1.6.13.2-10.fc38.x86_64 110/151 Installing : ghc8.10-pretty-devel-1.1.3.6-10.fc38.x86_64 111/151 Installing : ghc8.10-template-haskell-devel-2.16.0.0-10.fc38.x86_64 112/151 Installing : ghc8.10-text-devel-1.2.4.1-10.fc38.x86_64 113/151 Installing : ghc8.10-stm-devel-2.5.0.1-10.fc38.x86_64 114/151 Installing : ghc8.10-ghc-heap-devel-8.10.7-10.fc38.x86_64 115/151 Installing : ghc8.10-terminfo-devel-0.4.1.4-10.fc38.x86_64 116/151 Installing : ghc8.10-terminfo-prof-0.4.1.4-10.fc38.x86_64 117/151 Installing : ghc8.10-ghc-heap-prof-8.10.7-10.fc38.x86_64 118/151 Installing : ghc8.10-exceptions-devel-0.10.4-10.fc38.x86_64 119/151 Installing : ghc8.10-stm-prof-2.5.0.1-10.fc38.x86_64 120/151 Installing : ghc8.10-parsec-devel-3.1.14.0-10.fc38.x86_64 121/151 Installing : ghc8.10-pretty-prof-1.1.3.6-10.fc38.x86_64 122/151 Installing : ghc8.10-template-haskell-prof-2.16.0.0-10.fc38.x86_64 123/151 Installing : ghc8.10-text-prof-1.2.4.1-10.fc38.x86_64 124/151 Installing : ghc8.10-ghc-boot-devel-8.10.7-10.fc38.x86_64 125/151 Installing : ghc8.10-ghc-boot-prof-8.10.7-10.fc38.x86_64 126/151 Installing : ghc8.10-ghci-devel-8.10.7-10.fc38.x86_64 127/151 Installing : ghc8.10-ghci-prof-8.10.7-10.fc38.x86_64 128/151 Installing : ghc8.10-hpc-devel-0.6.1.0-10.fc38.x86_64 129/151 Installing : ghc8.10-ghc-devel-8.10.7-10.fc38.x86_64 130/151 Installing : ghc8.10-hpc-prof-0.6.1.0-10.fc38.x86_64 131/151 Installing : ghc8.10-libiserv-devel-8.10.7-10.fc38.x86_64 132/151 Installing : ghc8.10-parsec-prof-3.1.14.0-10.fc38.x86_64 133/151 Installing : ghc8.10-exceptions-prof-0.10.4-10.fc38.x86_64 134/151 Installing : ghc8.10-Cabal-devel-3.2.1.0-10.fc38.x86_64 135/151 Installing : ghc8.10-haskeline-devel-0.8.2-10.fc38.x86_64 136/151 Installing : ghc8.10-ghc-compact-devel-0.1.0.0-10.fc38.x86_64 137/151 Installing : ghc8.10-xhtml-devel-3000.2.2.1-10.fc38.x86_64 138/151 Installing : ghc8.10-devel-8.10.7-10.fc38.x86_64 139/151 Installing : ghc8.10-xhtml-prof-3000.2.2.1-10.fc38.x86_64 140/151 Installing : ghc8.10-ghc-compact-prof-0.1.0.0-10.fc38.x86_64 141/151 Installing : ghc8.10-haskeline-prof-0.8.2-10.fc38.x86_64 142/151 Installing : ghc8.10-Cabal-prof-3.2.1.0-10.fc38.x86_64 143/151 Installing : ghc8.10-libiserv-prof-8.10.7-10.fc38.x86_64 144/151 Installing : ghc8.10-ghc-prof-8.10.7-10.fc38.x86_64 145/151 Installing : ghc8.10-prof-8.10.7-10.fc38.x86_64 146/151 Installing : ghc8.10-compiler-default-8.10.7-10.fc38.x86_64 147/151 Installing : ghc8.10-doc-index-8.10.7-10.fc38.noarch 148/151 Installing : ghc8.10-8.10.7-10.fc38.x86_64 149/151 Installing : annobin-plugin-gcc-10.92-1.fc37.x86_64 150/151 Running scriptlet: annobin-plugin-gcc-10.92-1.fc37.x86_64 150/151 Installing : gcc-plugin-annobin-12.2.1-4.fc37.x86_64 151/151 Running scriptlet: gcc-plugin-annobin-12.2.1-4.fc37.x86_64 151/151 /usr/lib64/ghc-8.10.7/bin/ghc-pkg: error while loading shared libraries: libHSterminfo-0.4.1.4-ghc8.10.7.so: cannot open shared object file: No such file or directory /usr/lib64/ghc-8.10.7/bin/haddock: error while loading shared libraries: libHSxhtml-3000.2.2.1-ghc8.10.7.so: cannot open shared object file: No such file or directory warning: %transfiletriggerin(ghc8.10-doc-index-8.10.7-10.fc38.noarch) scriptlet failed, exit status 127 Error in <unknown> scriptlet in rpm package gcc-plugin-annobin Verifying : gc-8.0.6-4.fc37.x86_64 1/151 ... If I do 'mock --shell' afterwards, /usr/lib64/ghc-8.10.7/bin/ghc-pkg works fine. So it seems some scriptlet does something which makes ghc-pgk work again. execsnoop shows: ... ghc-pkg-8.10.7 3790103 3790102 0 /usr/bin/ghc-pkg-8.10.7 recache --no-user-package-db ghc-pkg 3790103 3790102 0 /usr/lib64/ghc-8.10.7/bin/ghc-pkg --global-package-db /usr/lib64/ghc-8.10.7/package.conf.d recache --no-user-package-db sh 3790104 3790000 0 /bin/sh /var/tmp/rpm-tmp.t4raco 0 env 3790105 3790104 0 /bin/env -C /usr/share/doc/ghc8.10/html/libraries ./gen_contents_index gen_contents_in 3790105 3790104 0 ./gen_contents_index ls 3790106 3790105 0 /bin/ls Cabal-3.2.1.0/Cabal.haddock array-0.5.4.0/array.haddock base-4.14.3.0/base.haddock binary-0.8.8.0/binary.haddock bytestring-0.10.12.0/bytestring.haddock containers-0.6.5.1/containers.haddock deepseq-1.4.4.0/deepseq.haddock directory-1.3.6.0/directory.haddock exceptions-0.10.4/exceptions.haddock filepath-1.4.2.1/filepath.haddock ghc-8.10.7/ghc.haddock ghc-boot-8.10.7/ghc-boot.haddock ghc-boot-th-8.10.7/ghc-boot-th.haddock ghc-compact-0.1.0.0/ghc-compact.haddock ghc-heap-8.10.7/ghc-heap.haddock ghc-prim-0.6.1/ghc-prim.haddock ghci-8.10.7/ghci.haddock haskeline-0.8.2/haskeline.haddock hpc-0.6.1.0/hpc.haddock grep 3790109 3790107 0 /bin/grep -v /ghc\.haddock sort 3790110 3790107 0 /bin/sort ls 3790108 3790107 0 /bin/ls -1 Cabal-3.2.1.0/Cabal.haddock array-0.5.4.0/array.haddock base-4.14.3.0/base.haddock binary-0.8.8.0/binary.haddock bytestring-0.10.12.0/bytestring.haddock containers-0.6.5.1/containers.haddock deepseq-1.4.4.0/deepseq.haddock directory-1.3.6.0/directory.haddock exceptions-0.10.4/exceptions.haddock filepath-1.4.2.1/filepath.haddock ghc-8.10.7/ghc.haddock ghc-boot-8.10.7/ghc-boot.haddock ghc-boot-th-8.10.7/ghc-boot-th.haddock ghc-compact-0.1.0.0/ghc-compact.haddock ghc-heap-8.10.7/ghc-heap.haddock ghc-prim-0.6.1/ghc-prim.haddock ghci-8.10.7/ghci.haddock haskeline-0.8.2/haskeline.haddock sed 3790113 3790111 0 /bin/sed s#/.*## sed 3790116 3790114 0 /bin/sed s#/.*## sed 3790119 3790117 0 /bin/sed s#/.*## sed 3790122 3790120 0 /bin/sed s#/.*## sed 3790125 3790123 0 /bin/sed s#/.*## sed 3790128 3790126 0 /bin/sed s#/.*## sed 3790131 3790129 0 /bin/sed s#/.*## sed 3790134 3790132 0 /bin/sed s#/.*## sed 3790137 3790135 0 /bin/sed s#/.*## sed 3790140 3790138 0 /bin/sed s#/.*## sed 3790143 3790141 0 /bin/sed s#/.*## sed 3790146 3790144 0 /bin/sed s#/.*## sed 3790149 3790147 0 /bin/sed s#/.*## sed 3790152 3790150 0 /bin/sed s#/.*## sed 3790155 3790153 0 /bin/sed s#/.*## sed 3790158 3790156 0 /bin/sed s#/.*## sed 3790161 3790159 0 /bin/sed s#/.*## sed 3790164 3790162 0 /bin/sed s#/.*## sed 3790167 3790165 0 /bin/sed s#/.*## sed 3790170 3790168 0 /bin/sed s#/.*## sed 3790173 3790171 0 /bin/sed s#/.*## sed 3790176 3790174 0 /bin/sed s#/.*## sed 3790179 3790177 0 /bin/sed s#/.*## sed 3790182 3790180 0 /bin/sed s#/.*## sed 3790185 3790183 0 /bin/sed s#/.*## sed 3790188 3790186 0 /bin/sed s#/.*## sed 3790191 3790189 0 /bin/sed s#/.*## sed 3790194 3790192 0 /bin/sed s#/.*## sed 3790197 3790195 0 /bin/sed s#/.*## sed 3790200 3790198 0 /bin/sed s#/.*## sed 3790203 3790201 0 /bin/sed s#/.*## sed 3790206 3790204 0 /bin/sed s#/.*## haddock 3790207 3790105 0 /usr/bin/haddock --gen-index --gen-contents -o . -t Haskell Hierarchical Libraries -p prologue.txt --read-interface=Cabal-3.2.1.0,Cabal-3.2.1.0/Cabal.haddock --read-interface=array-0.5.4.0,array-0.5.4.0/array.haddock --read-interface=base-4.14.3.0,base-4.14.3.0/base.haddock --read-interface=binary-0.8.8.0,binary-0.8.8.0/binary.haddock --read-interface=bytestring-0.10.12.0,bytestring-0.10.12.0/bytestring.haddock --read-interface=containers-0.6.5.1,containers-0.6.5.1/containers.haddock --read-interface=deepseq-1.4.4.0,deepseq-1.4.4.0/deepseq.haddock --read-interface=directory-1.3.6.0,directory-1.3.6.0/directory.haddock --read-interface=exceptions-0.10.4,exceptions-0.10.4/exceptions.haddock --read-interface=filepath-1.4.2.1,filepath-1.4.2.1/filepath.haddock --read-interface=ghc-boot-8.10.7,ghc-boot-8.10.7/ghc-boot.haddock haddock 3790207 3790105 0 /usr/lib64/ghc-8.10.7/bin/haddock -B/usr/lib64/ghc-8.10.7 -l/usr/lib64/ghc-8.10.7 --gen-index --gen-contents -o . -t Haskell Hierarchical Libraries -p prologue.txt --read-interface=Cabal-3.2.1.0,Cabal-3.2.1.0/Cabal.haddock --read-interface=array-0.5.4.0,array-0.5.4.0/array.haddock --read-interface=base-4.14.3.0,base-4.14.3.0/base.haddock --read-interface=binary-0.8.8.0,binary-0.8.8.0/binary.haddock --read-interface=bytestring-0.10.12.0,bytestring-0.10.12.0/bytestring.haddock --read-interface=containers-0.6.5.1,containers-0.6.5.1/containers.haddock --read-interface=deepseq-1.4.4.0,deepseq-1.4.4.0/deepseq.haddock --read-interface=directory-1.3.6.0,directory-1.3.6.0/directory.haddock --read-interface=exceptions-0.10.4,exceptions-0.10.4/exceptions.haddock ...
Hmm, nvm. You are right, this seems to be a repeat of #1467299. 'sudo strace -f -o/tmp/log -p …' on the dnf process shows: 3801559 execve("/usr/lib64/ghc-8.10.7/bin/ghc-pkg", ["/usr/lib64/ghc-8.10.7/bin/ghc-pk"..., "--global-package-db", "/usr/lib64/ghc-8.10.7/package.co"..., "recache", "--no-user-package-db"], 0x563473a64da0 /* 18 vars */) = 0 3801559 brk(NULL) = 0x1b2e000 3801559 arch_prctl(0x3001 /* ARCH_??? */, 0x7ffca6c8aa50) = -1 EINVAL (Invalid argument) 3801559 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1664d1c000 3801559 readlink("/proc/self/exe", 0x7ffca6c89680, 4096) = -1 ENOENT (No such file or directory) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 3801559 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) 3801559 openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 3801559 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=7627, ...}, AT_EMPTY_PATH) = 0 3801559 mmap(NULL, 7627, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f1664d1a000 3801559 close(3) = 0 3801559 openat(AT_FDCWD, "/lib64/glibc-hwcaps/x86-64-v4/libHSterminfo-0.4.1.4-ghc8.10.7.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 3801559 newfstatat(AT_FDCWD, "/lib64/glibc-hwcaps/x86-64-v4", 0x7ffca6c89c80, 0) = -1 ENOENT (No such file or directory) 3801559 openat(AT_FDCWD, "/lib64/glibc-hwcaps/x86-64-v3/libHSterminfo-0.4.1.4-ghc8.10.7.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 3801559 newfstatat(AT_FDCWD, "/lib64/glibc-hwcaps/x86-64-v3", 0x7ffca6c89c80, 0) = -1 ENOENT (No such file or directory) 3801559 openat(AT_FDCWD, "/lib64/glibc-hwcaps/x86-64-v2/libHSterminfo-0.4.1.4-ghc8.10.7.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 3801559 newfstatat(AT_FDCWD, "/lib64/glibc-hwcaps/x86-64-v2", 0x7ffca6c89c80, 0) = -1 ENOENT (No such file or directory) 3801559 openat(AT_FDCWD, "/lib64/tls/haswell/avx512_1/x86_64/libHSterminfo-0.4.1.4-ghc8.10.7.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 3801559 newfstatat(AT_FDCWD, "/lib64/tls/haswell/avx512_1/x86_64", 0x7ffca6c89c80, 0) = -1 ENOENT (No such file or directory) 3801559 openat(AT_FDCWD, "/lib64/tls/haswell/avx512_1/libHSterminfo-0.4.1.4-ghc8.10.7.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 3801559 newfstatat(AT_FDCWD, "/lib64/tls/haswell/avx512_1", 0x7ffca6c89c80, 0) = -1 ENOENT (No such file or directory) 3801559 openat(AT_FDCWD, "/lib64/tls/haswell/x86_64/libHSterminfo-0.4.1.4-ghc8.10.7.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 3801559 newfstatat(AT_FDCWD, "/lib64/tls/haswell/x86_64", 0x7ffca6c89c80, 0) = -1 ENOENT (No such file or directory) 3801559 openat(AT_FDCWD, "/lib64/tls/haswell/libHSterminfo-0.4.1.4-ghc8.10.7.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 3801559 newfstatat(AT_FDCWD, "/lib64/tls/haswell", 0x7ffca6c89c80, 0) = -1 ENOENT (No such file or directory) 3801559 openat(AT_FDCWD, "/lib64/tls/avx512_1/x86_64/libHSterminfo-0.4.1.4-ghc8.10.7.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 3801559 newfstatat(AT_FDCWD, "/lib64/tls/avx512_1/x86_64", 0x7ffca6c89c80, 0) = -1 ENOENT (No such file or directory) 3801559 openat(AT_FDCWD, "/lib64/tls/avx512_1/libHSterminfo-0.4.1.4-ghc8.10.7.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 3801559 newfstatat(AT_FDCWD, "/lib64/tls/avx512_1", 0x7ffca6c89c80, 0) = -1 ENOENT (No such file or directory) 3801559 openat(AT_FDCWD, "/lib64/tls/x86_64/libHSterminfo-0.4.1.4-ghc8.10.7.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 3801559 newfstatat(AT_FDCWD, "/lib64/tls/x86_64", 0x7ffca6c89c80, 0) = -1 ENOENT (No such file or directory) 3801559 openat(AT_FDCWD, "/lib64/tls/libHSterminfo-0.4.1.4-ghc8.10.7.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) ... 3801559 openat(AT_FDCWD, "/usr/lib64/libHSterminfo-0.4.1.4-ghc8.10.7.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 3801559 newfstatat(AT_FDCWD, "/usr/lib64", {st_mode=S_IFDIR|0555, st_size=9536, ...}, 0) = 0 3801559 writev(2, [{iov_base="/usr/lib64/ghc-8.10.7/bin/ghc-pk"..., iov_len=33}, {iov_base=": ", iov_len=2}, {iov_base="error while loading shared libra"..., iov_len=36}, {iov_base=": ", iov_len=2}, {iov_base="libHSterminfo-0.4.1.4-ghc8.10.7."..., iov_len=34}, {iov_base=": ", iov_len=2}, {iov_base="cannot open shared object file", iov_len=30}, {iov_base=": ", iov_len=2}, {iov_base="No such file or directory", iov_len=25}, {iov_base="\n", iov_len=1}], 10) = 167 Oh, and this dnf is invoked in the following way: /usr/bin/systemd-nspawn -q -M 661de85990a24862b7e886e0caba252c -D /var/lib/mock/fedora-37-x86_64-bootstrap/root -a --capability=cap_ipc_lock --bind=/tmp/mock-resolv.x7l587j0:/etc/resolv.conf --console=pipe --setenv=TERM=vt100 --setenv=SHELL=/bin/bash --setenv=HOME=/var/lib/mock/fedora-37-x86_64/root/installation-homedir --setenv=HOSTNAME=mock --setenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin --setenv=PROMPT_COMMAND=printf "\033]0;<mock-chroot>\007" --setenv=PS1=<mock-chroot> \s-\v\$ --setenv=LANG=C.UTF-8 --setenv=LC_MESSAGES=C.UTF-8 --resolv-conf=off /usr/bin/dnf --installroot /var/lib/mock/fedora-37-x86_64/root/ --releasever 37 --setopt=deltarpm=False --allowerasing --disableplugin=local --disableplugin=spacewalk --disableplugin=versionlock install /builddir/ghc8.10-8.10.7-10.fc38.x86_64.rpm ... /usr/bin/python3 /usr/bin/dnf --installroot /var/lib/mock/fedora-37-x86_64/root/ --releasever 37 --setopt=deltarpm=False --allowerasing --disableplugin=local --disableplugin=spacewalk --disableplugin=versionlock install /builddir/ghc8.10-8.10.7-10.fc38.x86_64.rpm ... And dnf apparently does rpm installation by itself, it doesn't invoke rpm, and does not set up /var/lib/mock/fedora-37-x86_64/root/proc/. I would reassign this to dnf, but I don't know who should set up the chroot — dnf or mock.
In case this wasn't clear: systemd-nspawn sets up /var/lib/mock/fedora-37-x86_64-bootstrap/root, but dnf is called with --installroot=/var/lib/mock/fedora-37-x86_64/root/.
Thank you for the detailed analysis. This is weird, we have https://github.com/rpm-software-management/mock/blob/bf9b051b7dc74ad70c60165660da7db6a834ee9e/mock/py/mockbuild/package_manager.py#L186 and that should make the /proc mounted. I'll have a look.
On host there are four dirs: a) /proc b) /var/lib/mock/fedora-37-x86_64/root/proc c) /var/lib/mock/fedora-37-x86_64-bootstrap/root/proc d) /var/lib/mock/fedora-37-x86_64-bootstrap/root/var/lib/mock/fedora-37-x86_64/root/proc We seem to bind-mount "a -> b" and "a -> c". Then we switch using chroot() to "c" and call DNF there, we expect that "a" -> "d" mountpoint exists. It doesn't. Note that from within the "c" directory, "d" appears to be "/var/lib/mock/fedora-37-x86_64/root". Good catch Zbyszek. The fix from the previous experience won't be trivial. If you are in hurry, I'd propose disabling the bootstrap chroot in Koji for the time being (or at least for the Haskell stack).
Thank you, Pavel (In reply to Pavel Raiskup from comment #8) > We seem to bind-mount "a -> b" and "a -> c". Then we switch > using chroot() to "c" and call DNF there, we expect that "a" -> "d" > mountpoint exists. It doesn't. Note that from within the "c" directory, > "d" appears to be "/var/lib/mock/fedora-37-x86_64/root". I see > If you are in hurry, I'd propose disabling the bootstrap chroot in Koji > for the time being (or at least for the Haskell stack). How to do that? By patching the koji's mock? I was in a hurry but I doubt I will be able to start building before branching, let alone finishing.
I might even suggest that "simple chroot" should actually to removed from copr now to match koji. The reason I didn't report this issue earlier is really because I could use simple chroot as a workaround in copr. Of course this may not be practical for all repos I dunno.
(In reply to Jens Petersen from comment #10) > I might even suggest that "simple chroot" should actually to removed from > copr now to match koji. Copr is not intended to match 100% Koji. Copr is intended for easy builds, often experimental builds. Quite often with non-standard setup, which can be easily configured with few clicks. And not by admin setup upon ticket request. And this issue shows that "simple chroot" is still sometimes needed.
> How to do that? By patching the koji's mock? I believe Koji has some config options. I that would be a Fedora Infra/Releng task. I'm quite surprised --chroot=simple works-around this problem. The mountpoint patterns shouldn't differ, I'd guess it works by some weird accident. But then temporarily using --chroot=simple in Koji might be a less intrusive work-around.
(In reply to Miroslav Suchý from comment #11) > (In reply to Jens Petersen from comment #10) > > I might even suggest that "simple chroot" should actually to removed from > > copr now to match koji. > > Copr is not intended to match 100% Koji. Sure I get it - I simply meant that by forcing wider usage of systemd-nspawn more problems like this one could be sorted out perhaps for good. (That may not be realistic as I said, but quite some time passed now since mock/copr started defaulting to systemd-nspawn.) (In reply to Pavel Raiskup from comment #12) > I believe Koji has some config options. > I that would be a Fedora Infra/Releng task. Okay I may ask releng for advice. > I'm quite surprised --chroot=simple works-around this problem. > The mountpoint patterns shouldn't differ, I'd guess it works > by some weird accident. mock simple chroot uses a bootstrap chroot?
Correct, `--chroot=simple` doesn't imply --no-bootstrap-chroot.
I added ld.so.conf.d conf in ghc-9.2.5-126.fc38 last week which help a little, but nevertheless other package failed to build apparently due to ghc-pkg not setting up the ghc package package db correctly: <https://koji.fedoraproject.org/koji/taskinfo?taskID=97121910>. However I couldn't reproduce this locally with mock, so this new failure is still mysterious to me.
(In reply to Jens Petersen from comment #15) > I added ld.so.conf.d conf in ghc-9.2.5-126.fc38 last week which help a > little, > but nevertheless other package failed to build apparently due to ghc-pkg > not setting up the ghc package package db correctly: > <https://koji.fedoraproject.org/koji/taskinfo?taskID=97121910>. Erm oops, actually this new failure seems due to packaging dependency issue actually...
(In reply to Jens Petersen from comment #16) > (In reply to Jens Petersen from comment #15) > > I added ld.so.conf.d conf in ghc-9.2.5-126.fc38 last week which help a > > little, > > but nevertheless other package failed to build apparently due to ghc-pkg > > not setting up the ghc package package db correctly: > > <https://koji.fedoraproject.org/koji/taskinfo?taskID=97121910>. > > Erm oops, actually this new failure seems due to packaging dependency issue > actually... Well it may be somewhat related since the dependency generation uses ghc-pkg, but I hope restoring the RUNPATH in ghc may help fix/workaround it.
Draft PR: https://github.com/rpm-software-management/mock/pull/1040
FEDORA-2023-90a03d8d62 has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-90a03d8d62
FEDORA-2023-5c8d906435 has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2023-5c8d906435
FEDORA-EPEL-2023-1652f46b68 has been submitted as an update to Fedora EPEL 8. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2023-1652f46b68
FEDORA-EPEL-2023-74629814f7 has been submitted as an update to Fedora EPEL 9. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2023-74629814f7
FEDORA-EPEL-2023-74629814f7 has been pushed to the Fedora EPEL 9 testing repository. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2023-74629814f7 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2023-5c8d906435 has been pushed to the Fedora 37 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-5c8d906435` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-5c8d906435 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-EPEL-2023-1652f46b68 has been pushed to the Fedora EPEL 8 testing repository. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2023-1652f46b68 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2023-90a03d8d62 has been pushed to the Fedora 38 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-90a03d8d62` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-90a03d8d62 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2023-90a03d8d62 has been pushed to the Fedora 38 stable repository. If problem still persists, please make note of it in this bug report.
Thank you! At least I am able to build current ghc8.10 with mock-4.0 in my Rawhide toolbox locally now. (I think that wasn't possible earlier, though I didn't write it explicitly above.) So once Fedora Koji moves to mock-4.0, I should be able to build it again (and also in copr).
FEDORA-EPEL-2023-74629814f7 has been pushed to the Fedora EPEL 9 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-EPEL-2023-1652f46b68 has been pushed to the Fedora EPEL 8 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2023-5c8d906435 has been pushed to the Fedora 37 stable repository. If problem still persists, please make note of it in this bug report.