Simo, can you provide more details. What's wrong with shadow-utils? Who write these large UID records into lastlog?

FreeIPA allocates a random uid range for its use between 200k and 2G.

In my case I have a user with uid 1280000008, if I log in with that user the lastlog file size jumps to almost 400G. The actual file is sparse so it doesn't use all that space.

But normal applications that do not understand sparse will read 400G on copy/backup/etc ...

It seem to me that someone believes that UIDs are still limited to 65k or something and decided to treat the lastlog file as a memory table where the uid number is the index.
When a very high UID logs in a very big index is created so a seek into the file is done to a very high location and then some small value is written there.

This logic is fine when you have only low UIDs in the system, but with very high UIDs it is just a problem as most programs do not really understand sparse files.

I wonder if the lastlog parsing needs to be so fast and can't be changed to be just a list and use a traverse function to write to it.

HTH,
Simo.

"someone believes" - lastlog file format as a sparse file was defined at the time when 65k UID numbers were the norm. If we want to redefine the file format we will have to rewrite all the applications that handle it.

Additional points for consideration:
The fixed record size format with random access have the nice property that the writes to the individual records do not mess up the other records. This avoids the need to perform the copy+rewrite+rename when updated records are written. An alternative is to do append-only format - such as wtmp - however this requires to read of the whole file to determine the last records for each user and it would probably also require rotation of the file so it would not grow out of bounds.

Third alternative would be to use regular database file however I do not think that many upstreams of utilities such as login, gdm, sshd would be too fond of this dependency.

(In reply to comment #4)
> "someone believes" - lastlog file format as a sparse file was defined at the
> time when 65k UID numbers were the norm.

Yeah I realize that, but reality has changed now :-)

> If we want to redefine the file format
> we will have to rewrite all the applications that handle it.

I guess you can also close WONTFIX if you think a sparse file is not an issue.
I can turn this bug into a FreeIPA doc bug if you think the effort is not worth it.

It's really too bad applications directly access lastlog instead of going through a helper library that would have abstracted access away so that apps didn't need to know the format.

I suppose it is worth fixing this issue. 

First, we had a problem when nfsnobody UID was added to lastlog db. We used a "workaround" to get thru it. Unfortunately we can't use the same trick with IPA users. The problem is back.

Someone could claim that lastlog is a valid sparse file and other tools should start handling it properly. Unfortunately I'm afraid it won't work in all cases. People who use tools will not expect there is >400GB sparse file on the system. :)

Just a quick search in bugzilla:

https://bugzilla.redhat.com/show_bug.cgi?id=156809
https://bugzilla.redhat.com/show_bug.cgi?id=213450
https://bugzilla.redhat.com/show_bug.cgi?id=192560
https://bugzilla.redhat.com/show_bug.cgi?id=138676
https://bugzilla.redhat.com/show_bug.cgi?id=769601

and there are even more.


This issue is not doable in RHEL-6.3 time frame. I see it more like Fedora Feature first and than we can backport it into RHEL. It's also not task for few days or weeks because it requires changes in other packages too.

This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.

This also effect abrt, causing it to crash, which is of course ironic,  abrt fails to egrep lastlog and tallylog multiple times because of their size. I am going to open a bug with the abrt folks, but this is something you should be aware of.

-Erinn

This request was erroneously removed from consideration in Red Hat Enterprise Linux 6.4, which is currently under development.  This request will be evaluated for inclusion in Red Hat Enterprise Linux 6.4.

This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated
in the current release, Red Hat is unable to address this
request at this time.

Red Hat invites you to ask your support representative to
propose this request, if appropriate, in the next release of
Red Hat Enterprise Linux.

I am afraid we cannot change the lastlog format in RHEL-6. That would be too disruptive and it would require changes in multiple packages such as util-linux, coreutils (probably), pam, shadow-utils and maybe even many more.

This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated
in the current release, Red Hat is unable to address this
request at this time.

Red Hat invites you to ask your support representative to
propose this request, if appropriate, in the next release of
Red Hat Enterprise Linux.

This is impossible to fix in released RHEL.

Ok is this fixed in RHEL 7? Or Fedora? Should the bug be moved over there?

-Erinn

Unfortunately this is not fixed in RHEL7 either.

The Fedora rawhide bug is bug 951564