Bug 1135740

Created attachment 933027 [details]
File: backtrace

Created attachment 933028 [details]
File: cgroup

Created attachment 933029 [details]
File: core_backtrace

Created attachment 933030 [details]
File: dso_list

Created attachment 933031 [details]
File: environ

Created attachment 933032 [details]
File: exploitable

Created attachment 933033 [details]
File: limits

Created attachment 933034 [details]
File: maps

Created attachment 933035 [details]
File: open_fds

Created attachment 933036 [details]
File: proc_pid_status

Created attachment 933037 [details]
File: var_log_messages

Looks like a hawkey crash. Reassigning.

Thread 1, frame #2 - PackageKit called hy_sack_load_yum_repo with NULL repo parameter.

-> reassigning back to Packagekit.

*** Bug 1066170 has been marked as a duplicate of this bug. ***

Another user experienced a similar problem:

Initial system start. 

reporter:       libreport-2.3.0
backtrace_rating: 4
cmdline:        /usr/libexec/packagekitd
crash_function: hy_repo_get_string
executable:     /usr/libexec/packagekitd
kernel:         3.17.7-300.fc21.x86_64
package:        PackageKit-1.0.3-4.fc21
reason:         packagekitd killed by SIGSEGV
runlevel:       N 3
type:           CCpp
uid:            0

Another user experienced a similar problem:

Packagekitd just crashed itself.

reporter:       libreport-2.3.0
backtrace_rating: 4
cmdline:        /usr/libexec/packagekitd
crash_function: hy_repo_get_string
executable:     /usr/libexec/packagekitd
kernel:         3.17.7-300.fc21.x86_64
package:        PackageKit-1.0.3-4.fc21
reason:         packagekitd killed by SIGSEGV
runlevel:       N 5
type:           CCpp
uid:            0

Another user experienced a similar problem:

It happened during startup.

reporter:       libreport-2.3.0
backtrace_rating: 4
cmdline:        /usr/libexec/packagekitd
crash_function: hy_repo_get_string
executable:     /usr/libexec/packagekitd
kernel:         3.17.8-300.fc21.i686
package:        PackageKit-1.0.3-4.fc21
reason:         packagekitd killed by SIGSEGV
runlevel:       N 5
type:           CCpp
uid:            0

Another user experienced a similar problem:

PackageKit keeps crashing without apparent reason.

reporter:       libreport-2.3.0
backtrace_rating: 4
cmdline:        /usr/libexec/packagekitd
crash_function: hy_repo_get_string
executable:     /usr/libexec/packagekitd
kernel:         3.17.8-300.fc21.i686
package:        PackageKit-1.0.3-4.fc21
reason:         packagekitd killed by SIGSEGV
runlevel:       N 5
type:           CCpp
uid:            0

Just saw this too on an F21 system, looks to happen when I launch Software; I can restart packagekit.service and see it running, launch Software, and it crashes.

I had some issues with offline updates just prior to encountering this.

hum - this seems to be 100%

This seems to be 100% reproducible for me ATM - I can trigger it every time by launching gnome-software, even after a 'pkcon repair' and a reboot.

Looking through the code and the backtrace I've got a theory that I caused this by doing:

rm -f /etc/yum.repos.d/fedora-rawhide-kernel-nodebug.repo

because I didn't want to use that repo any more. I suspect that hawkey and/or hif still know about that repo somehow, so they're trying to set it up. It seems plausible that the hif_source_get_repo (src) call in hif_sack_add_source() might return null in this case, then we wind up calling hy_sack_load_repo() with a null second parameter.

I'm not sure where to check and see what repos hawkey/hif 'think' exist, though. I don't know where they get that information.

OK, yeah, I'm starting to like my theory. My backtrace does end up in the same problem, but it comes down a different path:

#8  0x00007f91b8db1eda in pk_backend_job_thread_setup (thread_data=0x7f91bb267ab0) at pk-backend-job.c:815

#7  0x00007f91ad35cf07 in pk_backend_search_thread (job=0x7f91bb260370 [PkBackendJob], params=<optimized out>, user_data=<optimized out>)
    at pk-backend-hif.c:899

#6  0x00007f91ad35bcfe in hif_utils_create_sack_for_filters (job=job@entry=0x7f91bb260370 [PkBackendJob], filters=<optimized out>, create_flags=<optimized out>,
    create_flags@entry=HIF_CREATE_SACK_FLAG_USE_CACHE, state=0x7f91bb017400 [HifState], error=error@entry=0x7f91927c5da8) at pk-backend-hif.c:678

#5  0x00007f91ad35bcfe in hif_utils_create_sack_for_filters (error=0x7f91927c5da8, state=<optimized out>, flags=13, sack=0x7f918c1a1410, job=0x7f91bb260370 [PkBackendJob]) at pk-backend-hif.c:490

#4  0x00007f91accf7ee4 in hif_sack_add_sources (sack=sack@entry=0x7f918c1a1410, sources=0x7f91bafd4e40, permissible_cache_age=4294967295, flags=flags@entry=13, state=state@entry=0x7f91bb017240 [HifState], error=error@entry=0x7f91927c5da8)
    at hif-sack.c:160

#3  0x00007f91accf7d8e in hif_sack_add_source (sack=sack@entry=0x7f918c1a1410, src=src@entry=0x7f91bb17a2f0 [HifSource], permissible_cache_age=permissible_cache_age@entry=4294967295, flags=flags@entry=13, state=0x7f91bb205f90 [HifState], error=error@entry=0x7f91927c5da8) at hif-sack.c:109

#2  0x00007f91acade39d in hy_sack_load_yum_repo (sack=sack@entry=0x7f918c1a1410, repo=0x0, flags=flags@entry=3) at /usr/src/debug/hawkey/py3/src/sack.c:980

#1  0x00007f91acade39d in hy_sack_load_yum_repo (hrepo=0x0, sack=0x7f918c1a1410) at /usr/src/debug/hawkey/py3/src/sack.c:548

So looking in hif_utils_create_sack_for_filters() it's clear that it will use a cached 'sack' sometimes. In /var/cache/PackageKit/hif/ I see some very cache-y looking bits, including:

fedora-rawhide-kernel-nodebug.solv

though it's rather old:

"Sep 21 22:27"

but in /var/cache/PackageKit/hawkey there's some similar bits, much newer:

Jan 21 07:01 fedora-rawhide-kernel-nodebug.solv

so my bet here is that this code path isn't safe when a repo has been deleted out from under it; the cached sack has a source backed by a repo that doesn't actually exist any more, so when hif_source_get_repo() is called on that source the result is null, and that gets passed to hy_sack_load_repo(), causing the crash.

So I figured moving /var/cache/PackageKit/hawkey out of the way, restarting packagekit.service, and running gnome-software would work around this bug, but actually, it seems to cause the same crash. Richard's going to look into it further on Monday.

I easily reproduced this with this repo file:
http://paste.fedoraproject.org/180902/29755291/

It seems it's enough when you point a repo with file:// baseurl into a nonexistent directory.

commit 43188b72c5b88e0ec427dc0b24acae5f91c72390
Author: Richard Hughes <richard>
Date:   Tue Feb 3 15:15:49 2015 +0000

    If the enabled local source does not exist do not add it to the sack
    
    You can reproduce the problem setting baseurl=file:///notgoingtoexist in an
    enabled .repo file.
    
    Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1135740

Builds here: http://koji.fedoraproject.org/koji/taskinfo?taskID=8810185

libhif-0.1.8-3.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/libhif-0.1.8-3.fc21

Package libhif-0.1.8-4.fc21:
* should fix your issue,
* was pushed to the Fedora 21 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing libhif-0.1.8-4.fc21'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-1644/libhif-0.1.8-4.fc21
then log in and leave karma (feedback).

libhif-0.1.8-4.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.

*** Bug 1247753 has been marked as a duplicate of this bug. ***