The normal mmap paths all avoid creating a mapping where the pgoff inside the mapping could wrap around due to overflow. However, an expanding mremap() can take such a non-wrapping mapping and make it bigger and cause a wrapping condition. There is also another case where we expand mappings hiding in plain sight: the automatic stack expansion.
The wrapping condition can cause a BUG_ON() due to terminally confusing the vma_prio_tree code.
Upstream patches:
982134ba62618c2d69fbbbd166d0a11ee3b7e3d8 mremap
a626ca6a656450e9f4df91d0dda238fff23285f4 stack expansion downwards
42c36f63ac1366ab0ecc2d5717821362c259f517 stack expansion upwards
References:
http://www.spinics.net/lists/stable-commits/msg11385.htmlhttp://www.spinics.net/lists/linux-mm/msg17093.htmlhttp://groups.google.com/group/fa.linux.kernel/msg/9e43ab898c5e6d16
Acknowledgements:
Red Hat would like to thank Robert Swiecki for reporting this issue.