Bug 951564 - Document the lastlog file format limitation
Summary: Document the lastlog file format limitation
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: shadow-utils
Version: rawhide
Hardware: All
OS: Linux
medium
medium
Target Milestone: ---
Assignee: Iker Pedrosa
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: sync-to-jira review
Depends On:
Blocks: 827429
TreeView+ depends on / blocked
 
Reported: 2013-04-12 13:38 UTC by Tomas Mraz
Modified: 2021-05-12 16:12 UTC (History)
25 users (show)

Fixed In Version: shadow-utils-4.8.1-8.fc34 shadow-utils-4.8.1-6.fc33
Doc Type: Enhancement
Doc Text:
Clone Of: 771286
Environment:
Last Closed: 2021-04-24 19:45:16 UTC
Type: ---


Attachments (Terms of Use)

Description Tomas Mraz 2013-04-12 13:38:11 UTC
+++ This bug was initially created as a clone of Bug #771286 +++

Description of problem:
I will point you to this e-mail for the details of the issue. But the short of it is large UID = large lastlog file (sparse file but still), this creates problems for backups programs and integrity checkers (though of course they should handle sparse files). 

https://www.redhat.com/archives/freeipa-users/2012-January/msg00010.html

This is also occuring in Fedora. 

Simo knows more of the technical details than I by a long shot. 

Thanks,
-Erinn

--- Additional comment from Simo Sorce on 2012-01-03 16:18:13 CET ---

FreeIPA allocates a random uid range for its use between 200k and 2G.

In my case I have a user with uid 1280000008, if I log in with that user the lastlog file size jumps to almost 400G. The actual file is sparse so it doesn't use all that space.

But normal applications that do not understand sparse will read 400G on copy/backup/etc ...

It seem to me that someone believes that UIDs are still limited to 65k or something and decided to treat the lastlog file as a memory table where the uid number is the index.
When a very high UID logs in a very big index is created so a seek into the file is done to a very high location and then some small value is written there.

This logic is fine when you have only low UIDs in the system, but with very high UIDs it is just a problem as most programs do not really understand sparse files.

I wonder if the lastlog parsing needs to be so fast and can't be changed to be just a list and use a traverse function to write to it.

HTH,
Simo.

--- Additional comment from Peter Vrabec on 2012-01-06 16:00:57 CET ---

I suppose it is worth fixing this issue. 

First, we had a problem when nfsnobody UID was added to lastlog db. We used a "workaround" to get thru it. Unfortunately we can't use the same trick with IPA users. The problem is back.

Someone could claim that lastlog is a valid sparse file and other tools should start handling it properly. Unfortunately I'm afraid it won't work in all cases. People who use tools will not expect there is >400GB sparse file on the system. :)

Just a quick search in bugzilla:

https://bugzilla.redhat.com/show_bug.cgi?id=156809
https://bugzilla.redhat.com/show_bug.cgi?id=213450
https://bugzilla.redhat.com/show_bug.cgi?id=192560
https://bugzilla.redhat.com/show_bug.cgi?id=138676
https://bugzilla.redhat.com/show_bug.cgi?id=769601

and there are even more.


This issue is not doable in RHEL-6.3 time frame. I see it more like Fedora Feature first and than we can backport it into RHEL. It's also not task for few days or weeks because it requires changes in other packages too.

--- Additional comment from Tomas Mraz on 2013-04-12 15:30:49 CEST ---

I am afraid we cannot change the lastlog format in RHEL-6. That would be too disruptive and it would require changes in multiple packages such as util-linux, coreutils (probably), pam, shadow-utils and maybe even many more.

Comment 4 Nara 2015-06-10 13:41:03 UTC
When using the pam_tally2.so module I am seeing the same result for the tallylog.  Would this be related?

We bound to an Active Directory using beyondtrust's PowerBroker Identity Services (http://www.beyondtrust.com/Home/FreeTrials/#tab2), previously likewise.  The domain accounts have UIDs in the hundred millions, up to trillions. 

As soon as they are recognized lastlog and tallylog explode up to 100+ GB.

I'm sure the same result would be found if we went through the manually steps to bind to the AD, but i can't test that right now.

But the pattern is the same.
# ls -hl /var/log/lastlog
... ... 528G ... /var/log/lastlog
# du -h /var/log/lastlog
32K /var/log/lastlog
# ls -hl /var/log/tallylog
... ... 116G ... /var/log/tallylog
# du -h /var/log/tallylog
12K /var/log/tallylog

Comment 5 Tomas Mraz 2015-06-10 16:17:53 UTC
You have to use pam_faillock.so module instead of pam_tally2.so.

Comment 6 Daniel Walsh 2016-11-28 21:06:01 UTC
This is a very old issue and is now causing havoc with docker.  Is anyone looking at this issue?

Comment 7 Tomas Mraz 2016-11-29 10:44:52 UTC
Currently nobody. Also please understand that it can be changed in the next major RHEL release at the earliest. We cannot change it in RHEL6 and 7.

Comment 8 Daniel Walsh 2016-11-29 15:24:27 UTC
Sure although we potentially could do something in containers rather then in full userspace.  But if we don't do something soon, this will not get fixes in RHEL8.

Comment 9 Justin 2018-08-15 22:43:55 UTC
This lastlog file is causing me tremendous grief cause the / partition to appear full so yum wont run or all sorts of other issues.  Only solution is to check the file, delete it and reboot.  That pretty much sucks.

Comment 10 Aaron Nel 2018-10-26 09:58:34 UTC
Just wanted to call attention to this bug again. 

Also, affect our systems. 
Getting disusage alerts/root full alerts for a 43G lastlog file.

Using IPA, with uids like '100 000 000'

Comment 11 Edray Mashiri 2018-10-26 11:07:57 UTC
Having the same issue on our servers, also using IPA.

Comment 12 Ladislav Furman 2018-10-26 11:10:55 UTC
Same issue occurs hen using Centrify DC suite with enabled generating UIDs off of AD SID.

Comment 13 Justin 2018-10-26 12:29:34 UTC
This is a really frustrating issue.  Reported back in 2012 and still not fixed.  There is not even a fix/workaround to prevent this one file from causing all sorts of problems with monitoring tools and almost anything that checks disk space.  How about image based backup utilities and docker images seem to be struggling with this also.

http://www.noah.org/wiki/Lastlog_is_gigantic

ls -lh /var/log/lastlog
-rw-r--r--. 1 root root 441G Oct 26 05:03 /var/log/lastlog

Joining RH systems to an AD domain should be common enough for this to get fixed.

Comment 14 Robin Hack 2018-10-26 14:03:18 UTC
Hello.

There is freezed proof of concept called liblastlog2:
https://github.com/marmolak/liblastlog2

Yeah it needs some polish after years.

However I'm not sure how upstreams will be opened to adopt this approach.

You need to propose changes at least to:
openssh (portable)
gnome/kde
pamd
systemd
... and almost any other project which uses old lastlog operations.

I have had experimerimantal patches with support for openssh but not proposed.

Comment 15 Patsev Anton 2020-07-23 09:44:58 UTC
Having the same issue on our servers

ls -lh /var/log/lastlog 
-rw-r--r--. 1 root root 494G Jul 23 11:53 /var/log/lastlog

Comment 16 Bastien Nocera 2020-08-03 13:54:28 UTC
The fprintd PAM module, which authenticates users through fingerprint readers,
would like to be able to know when the last time a particular user logged in
using a certain type of authentication.

Eg. we'd like to force the user to enter their password when the machine has
been rebooted or when they haven't used that password in X hours, but for that
we'd need the authentication methods to be recorded, and have a way of accessing
that information programmatically.

Comment 18 Fedora Update System 2021-04-07 07:30:23 UTC
FEDORA-2021-45b039fc92 has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2021-45b039fc92

Comment 19 Fedora Update System 2021-04-07 18:16:05 UTC
FEDORA-2021-45b039fc92 has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-45b039fc92`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-45b039fc92

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 20 Fedora Update System 2021-04-24 19:45:16 UTC
FEDORA-2021-45b039fc92 has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 21 Fedora Update System 2021-04-24 20:05:40 UTC
FEDORA-2021-45b039fc92 has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 22 Fedora Update System 2021-04-26 13:24:41 UTC
FEDORA-2021-6761b1adac has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2021-6761b1adac

Comment 23 Fedora Update System 2021-04-27 01:26:43 UTC
FEDORA-2021-6761b1adac has been pushed to the Fedora 33 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-6761b1adac`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-6761b1adac

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 24 Fedora Update System 2021-05-12 13:43:34 UTC
FEDORA-2021-6761b1adac has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 25 Fedora Update System 2021-05-12 16:12:20 UTC
FEDORA-2021-6761b1adac has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.