Bug 2351472 - Core dump in `ggml_free (libggml.so.b3561 + 0x5d3f)`
Summary: Core dump in `ggml_free (libggml.so.b3561 + 0x5d3f)`
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: llama-cpp
Version: 41
Hardware: Unspecified
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Tom.Rix
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks: 2367864
TreeView+ depends on / blocked
 
Reported: 2025-03-12 08:54 UTC by Tomas Tomecek
Modified: 2025-05-21 19:05 UTC (History)
4 users (show)

Fixed In Version: llama-cpp-b3561-2.fc41
Clone Of:
: 2367864 (view as bug list)
Environment:
Last Closed: 2025-04-21 16:46:36 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)

Description Tomas Tomecek 2025-03-12 08:54:08 UTC
The combo of llama-cpp and python3-llama-cpp-python is now segfaulting in F41:

   #0  0x00007f5405f58d3f ggml_free (libggml.so.b3561 + 0x5d3f)
   #1  0x00007f54062c7056 ffi_call_unix64 (libffi.so.8 + 0x9056)
   #2  0x00007f54062c274d ffi_call_int.lto_priv.0 (libffi.so.8 + 0x474d)
   #3  0x00007f54062c564e ffi_call (libffi.so.8 + 0x764e)

Wrote a minimal reproducer and ran it via pytest:

Current thread 0x00007f6338feb740 (most recent call first):
  File "/usr/lib64/python3.13/site-packages/llama_cpp/llama.py", line 206 in __init__
  File "/usr/lib64/python3.13/site-packages/llama_cpp/llama.py", line 2213 in from_pretrained
  File "/home/tt/g/fedora-copr/logdetective/logdetective/utils.py", line 74 in initialize_model
  File "/home/tt/g/fedora-copr/logdetective/tests/test_utils.py", line 16 in test_process_log
  File "/usr/lib/python3.13/site-packages/_pytest/python.py", line 159 in pytest_pyfunc_call
  File "/usr/lib/python3.13/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/usr/lib/python3.13/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/usr/lib/python3.13/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/usr/lib/python3.13/site-packages/_pytest/python.py", line 1627 in runtest
  File "/usr/lib/python3.13/site-packages/_pytest/runner.py", line 174 in pytest_runtest_call
  File "/usr/lib/python3.13/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/usr/lib/python3.13/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/usr/lib/python3.13/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/usr/lib/python3.13/site-packages/_pytest/runner.py", line 242 in <lambda>
  File "/usr/lib/python3.13/site-packages/_pytest/runner.py", line 341 in from_call
  File "/usr/lib/python3.13/site-packages/_pytest/runner.py", line 241 in call_and_report
  File "/usr/lib/python3.13/site-packages/_pytest/runner.py", line 132 in runtestprotocol
  File "/usr/lib/python3.13/site-packages/_pytest/runner.py", line 113 in pytest_runtest_protocol
  File "/usr/lib/python3.13/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/usr/lib/python3.13/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/usr/lib/python3.13/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/usr/lib/python3.13/site-packages/_pytest/main.py", line 362 in pytest_runtestloop
  File "/usr/lib/python3.13/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/usr/lib/python3.13/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/usr/lib/python3.13/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/usr/lib/python3.13/site-packages/_pytest/main.py", line 337 in _main
  File "/usr/lib/python3.13/site-packages/_pytest/main.py", line 283 in wrap_session
  File "/usr/lib/python3.13/site-packages/_pytest/main.py", line 330 in pytest_cmdline_main
  File "/usr/lib/python3.13/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/usr/lib/python3.13/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/usr/lib/python3.13/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/usr/lib/python3.13/site-packages/_pytest/config/__init__.py", line 175 in main
  File "/usr/lib/python3.13/site-packages/_pytest/config/__init__.py", line 201 in console_main
  File "/usr/lib/python3.13/site-packages/pytest/__main__.py", line 9 in <module>
  File "<frozen runpy>", line 88 in _run_code
  File "<frozen runpy>", line 198 in _run_module_as_main

Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_int
egers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, _brotli, simplejson._speedups, markupsafe._speedups, yaml._yaml, zstandar
d.backend_c (total: 18)
zsh: illegal hardware instruction (core dumped)  python3 -m pytest -vv --showlocals ./tests/test_utils.py

I'm pretty astonished by this since we haven't updated neither packages since November.

What could possibly cause this?

Reproducible: Always




llama-cpp-b3561-1.fc41.x86_64
python3-llama-cpp-python-0.2.87-2.fc41.x86_64

Comment 1 Tom.Rix 2025-03-15 14:56:41 UTC
can you attach the test_utils.py reproducer.
I tried rebuilding python-llama-cpp-python and it's tests fine.

Comment 2 Tomas Tomecek 2025-03-17 16:30:08 UTC
from llama_cpp import Llama

model_name = "Mungert/gemma-3-1b-it-gguf"
suffix = "q4_k_s.gguf"

model_instance = Llama.from_pretrained(
    repo_id=model_name,
    filename=f"*{suffix}",  # this will download the quantized model only
    n_ctx=0,
    verbose=5,
    logits_all=True)
response = model_instance(
    prompt="Hi, how are you?",
    stream=False,
    max_tokens=0,
    logprobs=1)
print(response)


This is the minimal reproducer. It downloads real gemma3 model and uses it. I disabled all inference tests in python3-llama-cpp because we didn't have huggingface_hub library, but now that we have it, I should re-enable them.

Actual result:

zsh: illegal hardware instruction (core dumped)  python3 minimal.py

Expected result (with the stack from PyPI)::

...ton of logs...
llama_perf_context_print:        load time =     130.06 ms
llama_perf_context_print: prompt eval time =     129.89 ms /     7 tokens (   18.56 ms per token,    53.89 tokens per second)
llama_perf_context_print:        eval time =   45494.22 ms /   860 runs   (   52.90 ms per token,    18.90 tokens per second)
llama_perf_context_print:       total time =   49406.30 ms /   867 tokens
{'id': 'cmpl-12b6481a-0049-4de5-829e-b62b1711db8e', 'object': 'text_completion', 'created': 1742228430, 'model': '/home
...rest of the answer...

Comment 3 Miroslav Suchý 2025-04-08 12:43:45 UTC
This happens to me too on F42.

Comment 4 Fedora Update System 2025-04-11 16:07:16 UTC
FEDORA-2025-35a55f8467 (llama-cpp-b3561-2.fc41) has been submitted as an update to Fedora 41.
https://bodhi.fedoraproject.org/updates/FEDORA-2025-35a55f8467

Comment 5 Tom.Rix 2025-04-11 16:08:41 UTC
This change is required but may not resolve the issue, the simple testcase no longer runs.

Comment 6 Fedora Update System 2025-04-13 03:17:52 UTC
FEDORA-2025-35a55f8467 has been pushed to the Fedora 41 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-35a55f8467`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-35a55f8467

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 7 Fedora Update System 2025-04-21 16:46:36 UTC
FEDORA-2025-35a55f8467 (llama-cpp-b3561-2.fc41) has been pushed to the Fedora 41 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 8 Jiri Kortus 2025-05-09 12:22:55 UTC
Hello,

I'm bumping into this bug when running logdetective. Unfortunately the bug doesn't seem to have gone in the fixed version of llama-cpp, I can still reproduce the bug using the snippet from comment 2:

$ pytest ./test.py
============================================================================================= test session starts ==============================================================================================
platform linux -- Python 3.13.3, pytest-8.3.4, pluggy-1.5.0
rootdir: /tmp
plugins: anyio-3.7.1
collecting ... Fatal Python error: Illegal instruction

Current thread 0x00007f780a4bc740 (most recent call first):
  File "/usr/lib64/python3.13/site-packages/llama_cpp/llama.py", line 206 in __init__
  File "/usr/lib64/python3.13/site-packages/llama_cpp/llama.py", line 2213 in from_pretrained
  File "/tmp/test.py", line 8 in <module>
  File "/usr/lib/python3.13/site-packages/_pytest/assertion/rewrite.py", line 184 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "<frozen importlib._bootstrap>", line 1387 in _gcd_import
  File "/usr/lib64/python3.13/importlib/__init__.py", line 88 in import_module
  File "/usr/lib/python3.13/site-packages/_pytest/pathlib.py", line 587 in import_path
  File "/usr/lib/python3.13/site-packages/_pytest/python.py", line 493 in importtestmodule
  File "/usr/lib/python3.13/site-packages/_pytest/python.py", line 546 in _getobj
  File "/usr/lib/python3.13/site-packages/_pytest/python.py", line 284 in obj
  File "/usr/lib/python3.13/site-packages/_pytest/python.py", line 562 in _register_setup_module_fixture
  File "/usr/lib/python3.13/site-packages/_pytest/python.py", line 549 in collect
  File "/usr/lib/python3.13/site-packages/_pytest/runner.py", line 389 in collect
  File "/usr/lib/python3.13/site-packages/_pytest/runner.py", line 341 in from_call
  File "/usr/lib/python3.13/site-packages/_pytest/runner.py", line 391 in pytest_make_collect_report
  File "/usr/lib/python3.13/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/usr/lib/python3.13/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/usr/lib/python3.13/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/usr/lib/python3.13/site-packages/_pytest/runner.py", line 567 in collect_one_node
  File "/usr/lib/python3.13/site-packages/_pytest/main.py", line 835 in _collect_one_node
  File "/usr/lib/python3.13/site-packages/_pytest/main.py", line 970 in genitems
  File "/usr/lib/python3.13/site-packages/_pytest/main.py", line 809 in perform_collect
  File "/usr/lib/python3.13/site-packages/_pytest/main.py", line 347 in pytest_collection
  File "/usr/lib/python3.13/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/usr/lib/python3.13/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/usr/lib/python3.13/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/usr/lib/python3.13/site-packages/_pytest/main.py", line 336 in _main
  File "/usr/lib/python3.13/site-packages/_pytest/main.py", line 283 in wrap_session
  File "/usr/lib/python3.13/site-packages/_pytest/main.py", line 330 in pytest_cmdline_main
  File "/usr/lib/python3.13/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/usr/lib/python3.13/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/usr/lib/python3.13/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/usr/lib/python3.13/site-packages/_pytest/config/__init__.py", line 175 in main
  File "/usr/lib/python3.13/site-packages/_pytest/config/__init__.py", line 201 in console_main
  File "/usr/bin/pytest", line 8 in <module>

Extension modules: markupsafe._speedups, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, _brotli, yaml._yaml (total: 16)
Illegal instruction (core dumped)
$ rpm -q python3-llama-cpp-python llama-cpp
python3-llama-cpp-python-0.2.87-2.fc41.x86_64
llama-cpp-b3561-2.fc41.x86_64

I guess I shouldn't reopen this bug now that it's closed/errata, but rather clone it into a new one, right?


Note You need to log in before you can comment on or make changes to this bug.