Bug 1082507 - Enable large arrays: Build octave such that it can use arrays larger than 2 GB
Summary: Enable large arrays: Build octave such that it can use arrays larger than 2 GB
Keywords:
Status: ASSIGNED
Alias: None
Product: Fedora
Classification: Fedora
Component: octave
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
Assignee: Orion Poplawski
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-03-31 08:16 UTC by Fabrice Allibe
Modified: 2019-12-16 10:05 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Feature: enable 64 bits indexing Reason: build octave such that it can use arrays larger than 2 GB Result (if any): Creating arrays larger than 2^31 should work
Clone Of:
Environment:
Last Closed: 2016-07-19 19:29:24 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Fabrice Allibe 2014-03-31 08:16:22 UTC
Description of problem:
the size of a single Octave array cannot exceed 2 GB of memory

octave is not built with "--enable-64":
# grep enable64 octave.spec
%global enable64 no
%configure --enable-shared --disable-static --enable-64=%enable64 \

Additional info:

http://wiki.octave.org/Enable_large_arrays:_Build_octave_such_that_it_can_use_arrays_larger_than_2Gb

www.gnu.org/software/octave/doc/interpreter/Compiling-Octave-with-64_002dbit-Indexing.html

http://lists.alioth.debian.org/pipermail/pkg-octave-devel/2013-May/010083.html

Comment 1 Susi Lehtola 2014-03-31 08:22:57 UTC
This will require 64-bit versions of at least ATLAS, qrupdate and suitesparse as well.

Comment 2 Jaroslav Reznik 2015-03-03 15:38:53 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 22 development cycle.
Changing version to '22'.

More information and reason for this action is here:
https://fedoraproject.org/wiki/Fedora_Program_Management/HouseKeeping/Fedora22

Comment 3 Fabrice Allibe 2015-03-06 14:37:31 UTC
Found a recent blog post about an "Octave with 64-bit indexing" built on Ubuntu 14.04: http://calaba.tumblr.com/post/107087607479/octave-64

The guy explains well why the need of 64-bit indexing, especially for stuff around big data analysis.

He is even providing a github repository about this octave recompilation: https://github.com/calaba/octave-3.8.2-enable-64-ubuntu-14.04

Maybe it can help unlock the situation on Fedora/RHEL side ?

Comment 4 Susi Lehtola 2015-03-08 21:27:05 UTC
Lol, nifty guide, building software as root :D

We can't do anything before the prerequisites are available as 64-bit versions. Maybe you'd like to file the necessary bug reports?

Also, since the option is still marked experimental, I'm not sure we'd want to replace the existing package with a version with 64-bit indices just yet.

Comment 5 Fedora End Of Life 2016-07-19 19:29:24 UTC
Fedora 22 changed to end-of-life (EOL) status on 2016-07-19. Fedora 22 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 6 Justin Walker 2018-01-12 23:41:04 UTC
I would like to request this, although I don't know how to change the version to 27 without cloning the bug...

I haven't upgraded to Fedora 27 yet, but I found that the current version of octave is still built without --enable-64 (https://kojipkgs.fedoraproject.org//packages/octave/4.2.1/4.fc27.2/data/logs/x86_64/build.log), even though it appears that ATLAS is built with the -b 64 flag, enabling 64-bit width integers (https://kojipkgs.fedoraproject.org//packages/atlas/3.10.3/1.fc27/data/logs/x86_64/build.log). This would lead me to believe that the bug upon which this one depends (https://bugzilla.redhat.com/show_bug.cgi?id=1354228) could be possibly be closed. And this bug could then be resolved, too, after a new build.

BTW, there's some evidence of people doing just this (http://octave.1599824.n4.nabble.com/Octave-64-bit-indexing-built-with-ATLAS-td4684889.html).

Comment 7 Susi Lehtola 2018-01-13 13:52:11 UTC
(In reply to Justin Walker from comment #6)
> I would like to request this, although I don't know how to change the
> version to 27 without cloning the bug...
> 
> I haven't upgraded to Fedora 27 yet, but I found that the current version of
> octave is still built without --enable-64
> (https://kojipkgs.fedoraproject.org//packages/octave/4.2.1/4.fc27.2/data/
> logs/x86_64/build.log), even though it appears that ATLAS is built with the
> -b 64 flag, enabling 64-bit width integers
> (https://kojipkgs.fedoraproject.org//packages/atlas/3.10.3/1.fc27/data/logs/
> x86_64/build.log). 

Nope. If you look at the build log, it's just building the code in 64-bit mode (-m64 flag to gcc). What it should be doing in addition is redefining 4 byte integers to 8 bytes, see your last link:

[...]
F2CDEFS = -DAdd_ -DF77_INTEGER=int64_t -DStringSunStyle
[...]
F77FLAGS = -O -mavx -m64 -fPIC -finteger-4-integer-8
F77NOOPT = -O0 -finteger-4-integer-8
[...]

The problem is that the BLAS/LAPACK routines take size arguments that are integers, and if they are 4 bytes long then you get a size limit of a few billion elements in arrays.

> This would lead me to believe that the bug upon which
> this one depends (https://bugzilla.redhat.com/show_bug.cgi?id=1354228) could
> be possibly be closed. And this bug could then be resolved, too, after a new
> build.

Nope, the blocking issue still remains: a 64-bit integer (8 bytes) version of ATLAS needs to be built.

Comment 8 Orion Poplawski 2019-11-03 04:01:14 UTC
So, I think there are two levels of 64-bit support:

- octave with 64-bit array indexes but with 32-bit integer BLAS/suitesparse/etc.

- octave with 64-bit array indexes but with 64-bit integer BLAS/suitesparse/etc.


I believe we can simply enable the first one now.  In fact, octave 5.1.0 defaults to that and we have continued to disable it.

For the second though, there is still work to be done.  octave's own fortran BLAS checks do not appear to work with openblas64 for example.  However, we have switched to openblas so I'm dropping the link on the atlas bug.

Comment 9 Orion Poplawski 2019-11-03 04:14:13 UTC
So just to be explicit - in the first case pure-octave arrays could be larger than 2GB, but operations that use:

 * BLAS
 * LAPACK
 * QRUPDATE
 * SuiteSparse
 * ARPACK
 * GLPK
 * Qhull

are still limited - so I'm not sure how use it will be.


Note You need to log in before you can comment on or make changes to this bug.