openQA tests are failing for sqlite 3.42.0. Several tests failed, all of which do package installation, including tests of building live and installer images, tests of enrolling as a FreeIPA domain client, etc. The problem seems to be a crash in sqlite. I'm including a backtrace from the live image build failure, where the installer (which is used to create live images) crashed during package install, with a traceback running through sqlite. Reproducible: Always Steps to Reproduce: 1. Update to sqlite-3.42.0 2. Try building a network install image, live image, or installing a significant package set Actual Results: The process doing package installation crashes Expected Results: It should install successfully These seem to be the most relevant frames of the backtrace, I will attach the entire thing: #9 rpmlog (code=4, fmt=0x7f3dc28fe798 "%s: %s: %s\n") at /usr/src/debug/rpm-4.18.91-1.fc39.x86_64/rpmio/rpmlog.c:442 saved_errno = 2 pri = 4 mask = 16 saverec = 1 ap = {{gp_offset = 24, fp_offset = 48, overflow_arg_area = 0x7f3db2afe510, reg_save_area = 0x7f3db2afe420}} n = <optimized out> #10 0x00007f3dc25da1b1 in renderLogMsg (iErrCode=28, zFormat=<optimized out>, ap=ap@entry=0x7f3db2afe640) at /usr/src/debug/sqlite-3.42.0-1.fc39.x86_64/sqlite3.c:31531 acc = {db = 0x0, zText = 0x7f3db2afe530 "file renamed while open: /mnt/sysroot/var/lib/dnf/history.sqlite", nAlloc = 210, mxAlloc = 0, nChar = 64, accError = 0 '\000', printfFlags = 0 '\000'} zMsg = "file renamed while open: /mnt/sysroot/var/lib/dnf/history.sqlite\000寲=\177\000\000\037\246\332\323=\177\000\0000u\301\340\340U\000\000\364寲=\177\000\000\000\000\000\000\000\000\000\000\210%\325\304=\177\000\000 \034\373\323=\177\000\000j\335\020\324=\177\000\000x8\017\324=\177\000\000\320%\325\304=\177\000\000\200H\237\302=\177\000\000\000\000\000\000\000\000\000\000`M\237\302=\177\000\000h\273\233\302=\177\000\0000u\301\340\340U\000\0000\001\226\302=\177\000\000\000\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000"... #11 0x00007f3dc25da294 in sqlite3_log (iErrCode=iErrCode@entry=28, zFormat=zFormat@entry=0x7f3dc26e20c6 "file renamed while open: %s") at /usr/src/debug/sqlite-3.42.0-1.fc39.x86_64/sqlite3.c:31542 ap = {{gp_offset = 24, fp_offset = 48, overflow_arg_area = 0x7f3db2afe720, reg_save_area = 0x7f3db2afe660}} #12 0x00007f3dc25da9cc in verifyDbFile (pFile=pFile@entry=0x7f3d962490d0) at /usr/src/debug/sqlite-3.42.0-1.fc39.x86_64/sqlite3.c:38712 buf = {st_dev = 64545, st_ino = 944182, st_nlink = 1, st_mode = 33188, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 118784, st_blksize = 4096, st_blocks = 232, st_atim = {tv_sec = 1687418596, tv_nsec = 989208330}, st_mtim = {tv_sec = 1687418596, tv_nsec = 988208330}, st_ctim = {tv_sec = 1687418596, tv_nsec = 988208330}, __glibc_reserved = {0, 0, 0}} rc = <optimized out> #13 0x00007f3dc25dfee7 in unixClose (id=0x7f3d962490d0) at /usr/src/debug/sqlite-3.42.0-1.fc39.x86_64/sqlite3.c:39348 rc = 0 pFile = 0x7f3d962490d0 pInode = 0x7f3d94196728 #14 0x00007f3dc26d9105 in sqlite3OsClose (pId=0x7f3d962490d0) at /usr/src/debug/sqlite-3.42.0-1.fc39.x86_64/sqlite3.c:25119 No locals. #15 sqlite3PagerClose.isra.0 (pPager=0x7f3d96248f48, db=0x7f3d97243348) at /usr/src/debug/sqlite-3.42.0-1.fc39.x86_64/sqlite3.c:59904 pTmp = 0x7f3d9621ae38 "" #16 0x00007f3dc26d9607 in sqlite3BtreeClose.isra.0 (p=0x7f3d9417f378) at /usr/src/debug/sqlite-3.42.0-1.fc39.x86_64/sqlite3.c:71641 pBt = 0x7f3d96103528 #17 0x00007f3dc267f378 in sqlite3LeaveMutexAndCloseZombie (db=0x7f3d97243348) at /usr/src/debug/sqlite-3.42.0-1.fc39.x86_64/sqlite3.c:176242 pDb = 0x7f3d972435e0 i = <optimized out> j = 0 i = <optimized out> j = <optimized out> pDb = <optimized out> pNext = <optimized out> p = <optimized out> pColl = <optimized out> pMod = <optimized out> #18 sqlite3LeaveMutexAndCloseZombie (db=0x7f3d97243348) at /usr/src/debug/sqlite-3.42.0-1.fc39.x86_64/sqlite3.c:45138 i = <optimized out> j = <optimized out> pDb = <optimized out> pNext = <optimized out> p = <optimized out> pColl = <optimized out> pMod = <optimized out> #19 0x00007f3dc267fabd in sqlite3Close (db=0x7f3d97243348, forceZombie=0) at /usr/src/debug/sqlite-3.42.0-1.fc39.x86_64/sqlite3.c:176155 No locals. #20 0x00007f3dc156d313 in SQLite3::close (this=0x7f3daff63fd0) at /usr/src/debug/libdnf-0.70.1-3.fc39.x86_64/libdnf/utils/sqlite3/Sqlite3.cpp:56 result = <optimized out> #21 0x00007f3dc06b3780 in _wrap_Swdb_closeDatabase (self=<optimized out>, args=<optimized out>) at /usr/src/debug/libdnf-0.70.1-3.fc39.x86_64/build-py3/bindings/python/CMakeFiles/_transaction.dir/transactionPYTHON_wrap.cxx:14540 fail = <optimized out> resultobj = 0x0 arg1 = <optimized out> argp1 = 0x7f3dafd5a620 res1 = <optimized out> swig_obj = <optimized out>
This is an automatic F39 Beta blocker per "Bugs which entirely prevent the composition of one or more of the release-blocking images required to be built for a currently-pending (pre-)release" - https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process#Automatic_blockers
Created attachment 1972047 [details] Full backtrace
Untag request: https://pagure.io/releng/issue/11497
There are several commits on the upstream 3.42 branch since 3.42.0 came out, but it's not immediately obvious to me whether any of them fixes this. It has now been untagged from Rawhide so, once the buildroot regenerates, should no longer cause builds to fail.
Also affects Koji builds: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/AXSVIFAX3XK4CWLN3ZAGE5YJED2FEJXZ/
As noted above, we already untagged it half an hour ago so it will no longer do so.
Yes, build completed now, thanks!
Hello, sorry for the mess and thank you for handling the situation. I will communicate with sqlite upstream about this behavior, revert the commit with the rebase causing this and add some downstream integration tests (that dnf works) to avoid such a situation in the future. Rebase will be added when this problem is solved. Sorry again.
Bodhi update reverting the rebase is here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-629236ba8b - currently in testing. I have tried to reproduce the issue locally and was successful when using dnf-3. Dnf-5 seems to work with sqlite 3.42.0.
Created attachment 1972231 [details] dnf-3 with sqlite 3.42.0 seqfault trace from Valgrind Attaching log from Valgrind for reference.
Thanks for the revert, but it wasn't necessary: what "untag" means is that we dropped the f39 tag from the 3.42.0 build, meaning it was taken out of the Rawhide buildroot and would never reach a Rawhide compose. Effectively it made 3.41.2-2.fc39 the 'current' Rawhide build again. Sorry for not making that clearer!
Yes, I am aware, but I wanted to make sure that the git is synced for planned rebuilds (19th July?) as I am currently not sure when exactly the fixed 3.42.0 will be delivered to rawhide.
Putzed around a little bisecting here and this one seems to be the culprit https://sqlite.org/src/info/e1702eb48d13c7c9
This appears to be a long-standing rpm bug, dragged into light by that related change in sqlite: https://github.com/rpm-software-management/rpm/pull/2553 / https://github.com/rpm-software-management/rpm/commit/ea3187cfcf9cac87e5bc5e7db79b0338da9e355e This should be fixed now in rpm-4.18.91-5 but only built in the f39-python tag.
For something like this, we might want to not wait for the 3.12 merge. I think it is OK to bump and rebuild on the rawhide tag, then bump and rebuild *again* on the python tag, so both branches have the fix and the python tag build is higher versioned. That's what they told me to do with another package, anyhow... otherwise we can't put the new sqlite back into Rawhide till Python 3.12 is merged.
Yeah. I was hoping they'd merge the side-tag soonish, I saw their initial estimate somewhere and thought that looks close enough, but seeing a progress report today at ~66% with blockers to fix, I dunno. I guess I can find some other bug to patch so we don't need to rebuild the same content three times. There are already two "rebuild for python 3.12" changelog entries and it's getting a bit old.
Okay, pulled a couple of unrelated regression fixes for good luck and built 4.18.91-6 for rawhide, -7 for python 3.12.
Awesome, so we can try landing the newer sqlite3 again. Zuzana, should we just try again with the existing build, or do you want to do a new one? Thanks!
(In reply to Adam Williamson from comment #18) > Awesome, so we can try landing the newer sqlite3 again. Zuzana, should we > just try again with the existing build, or do you want to do a new one? > Thanks! Hi, I personally do not see a problem in using the already built rpms. However, if it is needed I can of course build a new release. I have manually tested the sqlite-3.42.0-1 build with rpm-4.18.91-6.fc39 and the commands failing before were passing now.
https://pagure.io/releng/issue/11497#comment-863395 - retag request.