Bug 2431943

How are you installing python3-torch-2.9.1-5.fc44, because on my machine it won't install without --skip-broken due to a failed python3-sympy dependency which is also https://bugzilla.redhat.com/show_bug.cgi?id=2432011

I think the package update that broke sympy had not propagated to my local mirror yet. I am in the same boat now, though; It has held me up from pushing the PR temporarily switching to clang.

When I looked deeper into this, by trying to change the optimization flags with GCC 16, that wasn't effective. I think that's because the PyTorch build system injects `-O2`in multiple places.

The specifics of the crash are that it's is inside a hash map implementation used to choose between different implementations of functions at runtime. As you might expect, an intimidating C++ backtrace with several lines of templated type information per frame resulted.

I regret not writing down the specific symbols or assembly that I saw. IIRC it was an inlined version of std::equal_to<some_types>:operator() and for some reason the compiler had implemented it by loading data at various offsets from a base address into the SIMD registers and doing a bunch of SIMD operations on said data. When I tried to manually probe the offending address in GDB, I got an error.

At this point, I'm not necessarily convinced the issue is GCC 16, specifically, it might just be exposing some issue in the existing code.

Can gnome-abrt/"Problem Reporting" create and upload a report?

Or just 'coredumpctl info` output for the corefile, and attach it. Also what is the HW platform? (Mac, rpi, etc).

I have worked around the sympy problem here
https://src.fedoraproject.org/rpms/python-torch/c/a62f1f295a83d991668b7bcc6bdb9183ef97d8b5?branch=rawhide
build has started, it usually takes a day or two to propagate.

Created attachment 2123998 [details]
Core Dump of PyTorch crashing during import on aarch64

After manually installing the latest sympy build, from https://bodhi.fedoraproject.org/updates/FEDORA-2026-e754c9e961 or https://koji.fedoraproject.org/koji/buildinfo?buildID=2923700 , I was able to generate a core file which is attached (compressed). Please take a look.

I captured this core on an NVIDIA DGX Spark system.

I've duplicated it to, on an x13s, so not related to anything nvidia. I see the crash in the c++/gcc stl_function.h:374 as called from the pytorch flat_hash_map.h:58.

At a quick glance it appears to be an out of bounds access trying to load a single element into v31. The address at x0+0x190 is ok, but the address at x0+1e0, which has crossed a page boundary by 32 bytes from the previous access is not.

Created attachment 2124529 [details]
fully decoded backtrace + OperatorEntry.cpp:309 find parameter decoding

Attached a complete backtrace and decoding of the OperatorEntry.cpp:309 _kernel->find(dispatch_key); call parameters which are crashing.

If this flat_hash behaves anything like the normal std::hash, then the iterator in find() needs to validate the node before trying to use its value.

Thanks for tracking down the issue; I'm going to trace through the sources to see if I can understand where this is coming from.

It looks like the way this is intended to work is that each bucket has a distance value which is -1 when its not used, and the distance from where the entry should have been placed when its filled. Which is 0 if its in the correct location. The hash is terminated with a pre-filled bucket, which isn't used for normal hash lookups because the num_slots_minus_one is used to compute the bucket, except that if you do land on that bucket in find/emplace then the distance check which terminates anytime it finds a bucket with a distance less than the current walk length will happily walk off the end because 0<=0, and the pointer is ++'ed to the next/invalid entry.

I added a iterator==end() check to abort find() and that moved the crash, to emplace() where a similar bit of logic is being used. Converting that to a iterator==end() break; I think will make the crash go away, but fundamentally that suggests that the num_slots_minus_one check is off by one.

Thanks, I'll take a look at num_slots_minus_one to see if I can apply the suggested fix, and to make sure I understand it. If I do, I'll check against latest upstream and see if I can file a bug or suggest a change there.

Yes, I've definitely created a workaround patch, understanding why this is suddenly happening or fixing the off by one error, is probably the right fix.

Created attachment 2127523 [details]
verify flat hash size before walking off the end.

I opened an upstream issue, for further discussion: https://github.com/pytorch/pytorch/issues/174230