Bug 2098880 - F37FailsToInstall: dcrpm
Summary: F37FailsToInstall: dcrpm
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: python-dcrpm
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Davide Cavalca
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On: 2098802
Blocks: PYTHON3.11 F37FTBFS F37FailsToInstall
TreeView+ depends on / blocked
 
Reported: 2022-06-20 09:35 UTC by Miro Hrončok
Modified: 2022-07-01 16:30 UTC (History)
3 users (show)

Fixed In Version: python-dcrpm-0.6.3-3.fc37
Clone Of:
Environment:
Last Closed: 2022-07-01 16:30:50 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)

Description Miro Hrončok 2022-06-20 09:35:32 UTC
Hello,

Please note that this comment was generated automatically. If you feel that this output has mistakes, please contact me via email (mhroncok).

Your package (python-dcrpm) Fails To Install in Fedora 37:

can't install dcrpm:
  - nothing provides python3.10dist(psutil) needed by dcrpm-0.6.3-1.fc37.noarch
  - nothing provides python(abi) = 3.10 needed by dcrpm-0.6.3-1.fc37.noarch
  
If you know about this problem and are planning on fixing it, please acknowledge so by setting the bug status to ASSIGNED. If you don't have time to maintain this package, consider orphaning it, so maintainers of dependent packages realize the problem.


If you don't react accordingly to the policy for FTBFS/FTI bugs (https://docs.fedoraproject.org/en-US/fesco/Fails_to_build_from_source_Fails_to_install/), your package may be orphaned in 8+ weeks.


P.S. The data was generated solely from koji buildroot, so it might be newer than the latest compose or the content on mirrors. To reproduce, use the koji/local repo only, e.g. in mock:

    $ mock -r fedora-37-x86_64 --disablerepo='*' --enablerepo=local install dcrpm


P.P.S. If this bug has been reported in the middle of upgrading multiple dependent packages, please consider using side tags: https://docs.fedoraproject.org/en-US/fesco/Updates_Policy/#updating-inter-dependent-packages

Thanks!

Comment 1 Miro Hrončok 2022-06-20 10:07:09 UTC
This bugzilla is likely a fallout from the Python 3.11 rebuild.

If your package (or some of the dependencies it has) failed to rebuild during the Python 3.11 rebuild, they now fail to install. To fix this, packages need to be rebuilt in Rawhide.

We will slowly triage the bugzillas, but we'd appreciate your help.

If you know this is blocked by an existing reported build failure or another package not yet rebuilt with Python 3.11, please mark it as such by using the "Depends On"/"Blocks" bugzilla fields. That will help us determine what failures to prioritize.

If this is not Python 3.11 related, please remove the PYTHON3.11 blocking tracker.

Thank you and sorry for the inconvenience. Let me know if you need any help.

Comment 2 Miro Hrončok 2022-06-27 14:57:30 UTC
Hello,

This is the first reminder (step 3 from https://docs.fedoraproject.org/en-US/fesco/Fails_to_build_from_source_Fails_to_install/#_package_removal_for_long_standing_ftbfs_and_fti_bugs).

If you know about this problem and are planning on fixing it, please acknowledge so by setting the bug status to ASSIGNED. If you don't have time to maintain this package, consider orphaning it, so maintainers of dependent packages realize the problem.

Comment 3 Davide Cavalca 2022-06-27 15:36:24 UTC
Depends on testslide which currently fails to build.

Comment 4 Miro Hrončok 2022-06-30 20:05:33 UTC
testslide built, but this fails with:

____________ DcrpmIntegrationTest.test_rpmdb_centos7_missing_index _____________
self = <dcrpm.rpmutil.RPMUtil object at 0x7f76c9a12650>
    def check_rpmdb_indexes(self):
        # type: () -> None
        """
        For each rpmdb file we define a rpm command that blows up on inconsistencies,
        or returns incorrect results. Structure:
        'name_of_file': {
            'cmd': 'str', # rpm command
            'checks': [], # list of conditions to be met
        }
        """
        if sys.platform == "darwin":
            self.logger.debug("check_rpmdb_indexes is not implemented for darwin")
            return
    
        rpmdb_indexes = {
            "Basenames": {
                "cmd": [self.rpm_path, "-qf", self.rpm_path, "--dbpath", self.dbpath],
                "checks": [
                    lambda proc: proc.returncode != StatusCode.SEGFAULT,
                    lambda proc: len(proc.stdout.splitlines()) == 1,
                    lambda proc: proc.stdout.splitlines()[0].startswith("rpm-"),
                ],
            },
            "Conflictname": {
                "cmd": [
                    self.rpm_path,
                    "-q",
                    "--conflicts",
                    "setup",
                    "--dbpath",
                    self.dbpath,
                ],
                "checks": [
                    lambda proc: proc.returncode != StatusCode.SEGFAULT,
                    lambda proc: len(proc.stdout.splitlines()) == 3,
                ],
            },
            "Obsoletename": {
                "cmd": [
                    self.rpm_path,
                    "-q",
                    "--obsoletes",
                    "coreutils",
                    "--dbpath",
                    self.dbpath,
                ],
                "checks": [
                    lambda proc: proc.returncode != StatusCode.SEGFAULT,
                    lambda proc: len(proc.stdout.splitlines()) >= 1,
                ],
            },
            "Providename": {
                "cmd": [
                    self.rpm_path,
                    "-q",
                    "--whatprovides",
                    "rpm",
                    "--dbpath",
                    self.dbpath,
                ],
                "checks": [
                    lambda proc: proc.returncode != StatusCode.SEGFAULT,
                    lambda proc: len(proc.stdout.splitlines()) == 1,
                    lambda proc: proc.stdout.splitlines()[0].startswith("rpm-"),
                ],
            },
            "Requirename": {
                "cmd": [
                    self.rpm_path,
                    "-q",
                    "--whatrequires",
                    "rpm",
                    "--dbpath",
                    self.dbpath,
                ],
                "checks": [
                    lambda proc: proc.returncode != StatusCode.SEGFAULT,
                    lambda proc: len(proc.stdout.splitlines()) >= 1,
                    lambda proc: any(
                        line.startswith("rpm-") or line.startswith("yum-")
                        for line in proc.stdout.splitlines()
                    ),
                ],
            },
            "Recommendname": None,
            "Dirnames": None,
            "Group": None,
            "Name": None,
            "Installtid": None,
            "Enhancename": None,  # rarely used
            "Filetriggername": None,  # rarely used
            "Suggestname": None,  # rarely used
            "Supplementname": None,  # rarely used
            "Transfiletriggername": None,  # rarely used
            "Triggername": None,  # rarely used
        }  # type: t.Dict[str, t.Optional[t.Dict[str, t.Union[t.Sequence[str], t.Sequence[t.Callable[[CompletedProcess], bool]]]]]]
    
        # Checks for Packages db corruption
        post_checks = [
            lambda proc: not any(
                "cannot open Packages database" in line
                for line in proc.stderr.splitlines()
            ),
            lambda proc: any(
                "missing index" in line for line in proc.stderr.splitlines()
            ),
        ]
    
        for index, config in rpmdb_indexes.items():
            # Skip over indexes with no defined checks / conditions
            if not config:
                continue
    
            try:
                # Skip over non existing indexes
                if not os.path.join(self.dbpath, index):
                    self.logger.info("{} does not exist".format(index))
                    continue
    
                self.logger.info("Attempting to selectively poke at %s index", index)
                if t:
>                   self._poke_index(
                        t.cast(t.List[str], config["cmd"]),
                        t.cast(
                            t.List[t.Callable[[CompletedProcess], bool]],
                            config["checks"],
                        ),
                    )
dcrpm/rpmutil.py:246: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <dcrpm.rpmutil.RPMUtil object at 0x7f76c9a12650>
cmd = ['/usr/bin/rpm', '-q', '--whatprovides', 'rpm', '--dbpath', '/tmp/tmpaindzqtc/rpm']
checks = [<function RPMUtil.check_rpmdb_indexes.<locals>.<lambda> at 0x7f76c8f3f600>, <function RPMUtil.check_rpmdb_indexes.<locals>.<lambda> at 0x7f76c8f3f4c0>, <function RPMUtil.check_rpmdb_indexes.<locals>.<lambda> at 0x7f76c8f3f6a0>]
    def _poke_index(self, cmd, checks):
        # type: (t.Sequence[str], t.Iterable[t.Callable[[CompletedProcess], bool]]) -> CompletedProcess
        """
        Run cmd, and ensure all checks are True. Raise DBIndexNeedsRebuild otherwise
        """
        proc = run_with_timeout(cmd, RPM_CHECK_TIMEOUT_SEC, raise_on_nonzero=False)
        for check in checks:
            if not check(proc):
>               raise DBIndexNeedsRebuild
E               dcrpm.util.DBIndexNeedsRebuild
dcrpm/rpmutil.py:121: DBIndexNeedsRebuild
During handling of the above exception, another exception occurred:
self = <dcrpm.dcrpm.DcRPM object at 0x7f76c9a10dd0>
    def run(self):
        # type: () -> bool
        if not self.has_free_disk_space():
            self.status_logger.error("not_enough_disk")
            self.logger.error("Need at least %sB free to continue" % self.args.minspace)
            return False
    
        # Check old yum transactions.
        if self.args.clean_yum_transactions and self.stale_yum_transactions_exist():
            self.logger.info("Cleaning old yum transaction")
            self.rpmutil.clean_yum_transactions()
    
        # Check stuck yum.
        if self.args.check_stuck_yum:
            result = Yum().check_stuck(dry_run=self.args.dry_run)
            if not result:
                self.logger.error("Failed to unstuck yum processes")
    
        # Detect database backend
        backend = self.rpmutil.get_db_backend()
    
        # Start main checks.
        for i in range(self.args.max_passes):
            self.logger.debug("Running pass: %d", i)
    
            try:
                # Optional forensic data collection
                if self.args.forensic:
                    if backend == "bdb":
                        self.logger.info(
                            "Running forensic data collection (db_stat -CA)"
                        )
                        self.rpmutil.db_stat()
                    else:
                        self.logger.warning(
                            "Forensics data collection is not supported on %s" % backend
                        )
    
                # Kill any straggler rpm query processes
                self.logger.info("Searching for spinning rpm query processes")
                self.rpmutil.kill_spinning_rpm_query_processes()
    
                # Exercise single indexes
                self.logger.info("Sanity checking rpmdb indexes")
>               self.rpmutil.check_rpmdb_indexes()
dcrpm/dcrpm.py:89: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <dcrpm.rpmutil.RPMUtil object at 0x7f76c9a12650>
    def check_rpmdb_indexes(self):
        # type: () -> None
        """
        For each rpmdb file we define a rpm command that blows up on inconsistencies,
        or returns incorrect results. Structure:
        'name_of_file': {
            'cmd': 'str', # rpm command
            'checks': [], # list of conditions to be met
        }
        """
        if sys.platform == "darwin":
            self.logger.debug("check_rpmdb_indexes is not implemented for darwin")
            return
    
        rpmdb_indexes = {
            "Basenames": {
                "cmd": [self.rpm_path, "-qf", self.rpm_path, "--dbpath", self.dbpath],
                "checks": [
                    lambda proc: proc.returncode != StatusCode.SEGFAULT,
                    lambda proc: len(proc.stdout.splitlines()) == 1,
                    lambda proc: proc.stdout.splitlines()[0].startswith("rpm-"),
                ],
            },
            "Conflictname": {
                "cmd": [
                    self.rpm_path,
                    "-q",
                    "--conflicts",
                    "setup",
                    "--dbpath",
                    self.dbpath,
                ],
                "checks": [
                    lambda proc: proc.returncode != StatusCode.SEGFAULT,
                    lambda proc: len(proc.stdout.splitlines()) == 3,
                ],
            },
            "Obsoletename": {
                "cmd": [
                    self.rpm_path,
                    "-q",
                    "--obsoletes",
                    "coreutils",
                    "--dbpath",
                    self.dbpath,
                ],
                "checks": [
                    lambda proc: proc.returncode != StatusCode.SEGFAULT,
                    lambda proc: len(proc.stdout.splitlines()) >= 1,
                ],
            },
            "Providename": {
                "cmd": [
                    self.rpm_path,
                    "-q",
                    "--whatprovides",
                    "rpm",
                    "--dbpath",
                    self.dbpath,
                ],
                "checks": [
                    lambda proc: proc.returncode != StatusCode.SEGFAULT,
                    lambda proc: len(proc.stdout.splitlines()) == 1,
                    lambda proc: proc.stdout.splitlines()[0].startswith("rpm-"),
                ],
            },
            "Requirename": {
                "cmd": [
                    self.rpm_path,
                    "-q",
                    "--whatrequires",
                    "rpm",
                    "--dbpath",
                    self.dbpath,
                ],
                "checks": [
                    lambda proc: proc.returncode != StatusCode.SEGFAULT,
                    lambda proc: len(proc.stdout.splitlines()) >= 1,
                    lambda proc: any(
                        line.startswith("rpm-") or line.startswith("yum-")
                        for line in proc.stdout.splitlines()
                    ),
                ],
            },
            "Recommendname": None,
            "Dirnames": None,
            "Group": None,
            "Name": None,
            "Installtid": None,
            "Enhancename": None,  # rarely used
            "Filetriggername": None,  # rarely used
            "Suggestname": None,  # rarely used
            "Supplementname": None,  # rarely used
            "Transfiletriggername": None,  # rarely used
            "Triggername": None,  # rarely used
        }  # type: t.Dict[str, t.Optional[t.Dict[str, t.Union[t.Sequence[str], t.Sequence[t.Callable[[CompletedProcess], bool]]]]]]
    
        # Checks for Packages db corruption
        post_checks = [
            lambda proc: not any(
                "cannot open Packages database" in line
                for line in proc.stderr.splitlines()
            ),
            lambda proc: any(
                "missing index" in line for line in proc.stderr.splitlines()
            ),
        ]
    
        for index, config in rpmdb_indexes.items():
            # Skip over indexes with no defined checks / conditions
            if not config:
                continue
    
            try:
                # Skip over non existing indexes
                if not os.path.join(self.dbpath, index):
                    self.logger.info("{} does not exist".format(index))
                    continue
    
                self.logger.info("Attempting to selectively poke at %s index", index)
                if t:
                    self._poke_index(
                        t.cast(t.List[str], config["cmd"]),
                        t.cast(
                            t.List[t.Callable[[CompletedProcess], bool]],
                            config["checks"],
                        ),
                    )
                else:
                    # pyre-ignore[6]: config["cmd"] and config["checks"]
                    self._poke_index(config["cmd"], config["checks"])
    
            except DBIndexNeedsRebuild:
                self.status_logger.info(RepairAction.INDEX_REBUILD)
                index_path = os.path.join(self.dbpath, index)
                if os.path.isfile(index_path):
                    self.logger.info("%s index is out of whack, deleting it", index)
                    os.remove(index_path)
                else:
                    self.logger.info("%s index is missing", index)
    
                # Run the same command again, which should trigger a rebuild
                if t:
                    proc = self._poke_index(t.cast(t.List[str], config["cmd"]), [])
                else:
                    # pyre-ignore[6]: config["cmd"]
                    proc = self._poke_index(config["cmd"], [])
    
                # Sometimes single index rebuilds don't work, as rpm fails to
                # open Packages db. In that case we'll try a full recovery
                for check in post_checks:
                    if not check(proc):
                        self.logger.info("Granular index rebuild failed")
>                       raise DBNeedsRecovery()
E                       dcrpm.util.DBNeedsRecovery
dcrpm/rpmutil.py:278: DBNeedsRecovery
During handling of the above exception, another exception occurred:
args = (<tests.test_end_to_end.DcrpmIntegrationTest testMethod=test_rpmdb_centos7_missing_index>, '/tmp/tmpaindzqtc/rpm')
kwargs = {}
    def wrapper(*args, **kwargs):
        # type: (t.Any, t.Any) -> RT
        args += (os.path.join(temp_dir, "rpm"),)
>       result = t.cast(F, function)(*args, **kwargs)
tests/rpmdb.py:38: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_end_to_end.py:91: in test_rpmdb_centos7_missing_index
    run_result = self.dcrpm.run()
dcrpm/dcrpm.py:140: in run
    self.run_recovery()
dcrpm/dcrpm.py:194: in run_recovery
    lock_procs = pidutil.procs_holding_file(dbenv_lockfile)
dcrpm/pidutil.py:76: in procs_holding_file
    lsof = which("lsof")
dcrpm/util.py:133: in wrapper
    cache[key] = f(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
cmd = 'lsof'
    @memoize
    def which(cmd):
        # type: (str) -> str
        try:
            import shutil
    
            path = shutil.which(cmd)
            if not path:
>               raise DcRPMException("failed to find '{}'".format(cmd))
E               dcrpm.util.DcRPMException: failed to find 'lsof'
dcrpm/util.py:255: DcRPMException
------------------------------ Captured log call -------------------------------
WARNING  root:rpmutil.py:510 No db_backend found in macros, assuming bdb
ERROR    root:dcrpm.py:137 DB needs recovery
=========================== short test summary info ============================
FAILED tests/test_end_to_end.py::DcrpmIntegrationTest::test_rpmdb_centos7_missing_index

Comment 5 Davide Cavalca 2022-07-01 16:23:33 UTC
That test is broken on F37, I filed https://github.com/facebookincubator/dcrpm/issues/54 to get it sorted out upstream and will disable it for the time being.

Comment 6 Fedora Update System 2022-07-01 16:30:05 UTC
FEDORA-2022-6b7d15606f has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2022-6b7d15606f

Comment 7 Fedora Update System 2022-07-01 16:30:50 UTC
FEDORA-2022-6b7d15606f has been pushed to the Fedora 37 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.