Red Hat Bugzilla – Bug 484274
Octave is not built with 64 bit support
Last modified: 2012-10-02 18:36:22 EDT
Description of problem:
Matrix size is limited to 2GB on the x86_64 build. Fix: The 64 bit Fedora build of octave should include the --enable-64 compiler flag in order to permit large matrix operations.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. Find a 64 bit machine with >4GB memory
2. octave --eval 'b=17000; x=rand(b,b); size(x)'
error: memory exhausted or requested size too large for range of Octave's index type -- eval failed
It appears that he 64-bit extensions were deliberatly disabled a while back by the old maintainer, see the changelog here:
* Mon Jul 10 2006 Quentin Spencer 2.9.6-1
- New release. Remove old patches.
- Disable 64-bit extensions (some libraries don't support 64-bit indexing yet).
- Add gcc-gfortran to -devel dependencies (mkoctfile fails without it).
- Move octave-bug and octave-config from devel to main package.
- Fix categorization of info files (bug 196760).
I'm not sure if this is still a problem and if the --enable-64 configure can be re-enabled on x86_64.
Also not sure which libraries he was referring to, Cc'ing Quentin on this bug.
The --enable-64 flag uses 64 bit indexing for arrays, which allows for arrays to grow larger than the limits imposed by 32-bit indexing. I believe that this flag could even be theoretically used on a 32-bit processor--it doesn't enable any special 64-bit features. In order for that feature to be useful and even work properly, all of the dependent matrix and array math libraries (blas or atlas, lapack, suitesparse, and fftw) must be compiled to use 64 bit indexing as well. I recall from the octave mailing lists that there are some fortran flags that can force this when compiling blas and lapack, but this would produce libraries that are not compatible with the regular blas and lapack libraries (I don't know what it would require to support this in atlas and suitesparse). To really test this feature would require a lot of work and hardware that not everyone has access to (although it's getting less unusual now). You would need to compile all of the libraries, do some matrix math on arrays over 4 GB in size, and verify that you're getting the correct answers. I'm not aware of anyone ever reporting on the octave mailing lists that they had tested this feature, though periodically someone asks about support for such large arrays.
So, the short answer is that enabling this flag in Fedora doesn't make sense until all of the libraries and all of the other programs that use those libraries go to 64-bit indexing by default. That's probably not going to happen any time soon, so I suggest anyone really needing this feature compile it all from scratch using statically linked libraries.
Thank you Quentin for your input.
Alex I am closing this as WONTFIX for now. If in future we think this option needs to be worked out will re-open.
It sounds like it was decided "Oh, Geez. That might be a lot of work to look into. Let's not even try". I've run into this 32-bit index limit several times when working on different projects, and it's sad. A professional-grade scientific data processing and visualization package like Octave shouldn't have a 2GB index limit. Academics and those in industry routinely work with large datasets (Think 4GB-500GB), and with enough memory and swap space, we should be able to process such data. Please allow this bug to be re-opened, and perhapse elevated in priority.
(In reply to comment #4)
> It sounds like it was decided "Oh, Geez. That might be a lot of work to look
> into. Let's not even try". I've run into this 32-bit index limit several
> times when working on different projects, and it's sad. A professional-grade
> scientific data processing and visualization package like Octave shouldn't have
> a 2GB index limit. Academics and those in industry routinely work with large
> datasets (Think 4GB-500GB), and with enough memory and swap space, we should be
> able to process such data. Please allow this bug to be re-opened, and perhapse
> elevated in priority.
Nothing to stop you providing patches...
The 64 bit behavior seems to be substantially improved since this bug was first filed. The original test case of:
octave --eval 'b=17000; x=rand(b,b); size(x)'
now passes in a large machine.
It is curious, however, that even though the 4GB barrier has been breached, there is another memory limit that remains. The following test case, when run on a 64 bit machine with 144 GB of memory and version 3.4.3,
octave --eval 'b=46341; x=rand(b,b); size(x)'
fails with "error: memory exhausted or requested size too large for range of Octave's index type"
This seems to imply that there is a limit at the 17 GB point.