Vivek any ideas on this one?

I ran above useradd command in rhel6.5 container and it works fine. No large sparse file was created.

So does that mean this happens only if run during docker build using RUN command?

Upstream comment from unclejack says that a large 32GB sparse file is created. It is not clear who creates that sparse file. I assumed it will be useradd, but I can't see that.

Is it possible that docker commit is not handling sparse files well?

wtmp file?

This is only a problem during `docker build`

/var/log/lastlog is a sparse file.  When writing to the file for a very large UID, it must pad the intervening entries.

This is why passing '-l' to useradd works around the problem, that option tells useradd to skip updating /var/log/lastlog

I thought this might be issue with docker commit and sparse files and I am running into an strange issue when I create a sparse file in container and then try to commit container.

[root@vm2-f22 rhvgoyal-docker]# docker commit 2b7c139146d8
Error response from daemon: open /var/lib/docker/devicemapper/mnt/2b7c139146d8fb0cd6110c05f12ccd5d99e4cf59d6108e1ec97233f1f776a931-init/rootfs/usr/share/zoneinfo/America/Indiana: no such file or directory

So I tried this with overlayfs backend now. I create a sparse file of 1G size and do a docker commit and resultant image is 1.26GB in size. That means docker commit can't handle sparse files well and bloats these.

$ docker run -ti fedora bash
$ truncate -s 1G test.txt
$ exit
$ docker commit <above-container-id>
$ docker images

And resultant image is 1.26GB.


Docker history shows that top most layer itself is 1.074GB. Which confirms that docker has bloated the sparse file.

I am testing with latest upstream docker. That means problem is still present. So this first needs to be fixed upstream.

docker diff must show a huge file.  Perhaps golang tar can not handle sparse files?

Golang archive/tar can extract from a sparse archive, but can not create a sparse archive. Of all tar implementations, only GNU tar creates sparse archives. Many only barely support extracting sparse.

Now docker has re-opened this PR.

https://github.com/docker/docker/issues/5419

vbatts, sounds like fixing this is not going to be easy. Will require fundamental changes to golang archive/tar to be able to create sparse files.

(In reply to Vivek Goyal from comment #14)
> Now docker has re-opened this PR.
> 
> https://github.com/docker/docker/issues/5419
> 
> vbatts, sounds like fixing this is not going to be easy. Will require
> fundamental changes to golang archive/tar to be able to create sparse files.

I've contemplated these changes before, but it's not straight forward and would be a slight departure from the API.
Though this lack of sparse file handling does relate our previous discussions of block level CoW.

Really needs to be fixed upstream.

No movement on this.

https://github.com/docker/docker/issues/20707

No movement on this since May

Nalin any chance we can do a better job on this with container/storage?

Encountered this as well on RHEL 7.3.

There is a sparse file inside the container backed by overlay2/xfs.

# du -sh trinity-testfile2
18M	trinity-testfile2

# ll trinity-testfile2
----r--r--. 1 test test 2704441078443517033 Sep 30 16:06 trinity-testfile2

# filefrag trinity-testfile2 
trinity-testfile2: 10 extents found

Then, "docker commit" is running out of disk space because there is huge file (still growing) in the host during docker-untar.

# du -sh
...
28G	/var/lib/docker-latest/overlay2/907fba5d781faf78bb48680dfe31646bd60b893216ada69f5a9a14daab3559b2/diff/home/test/trinity-testfile2
...

We may probably have this fixed in Fedora and RHEL-8 with https://bugzilla.redhat.com/show_bug.cgi?id=951564.

I understand this is causing issues though. There's an issue upstream in Golang about having a Tar writer which supports sparse files (https://github.com/golang/go/issues/13548). Yet, that issue is stuck for more than 1 year now.

Docker side, we may implement a tar writer wrapper which could handle sparse files, yes. We'll run into troubles when hashing the resulting tar files though, since docker relies on content addressable hashes. I'd say this isn't really doable in docker and by going with a custom wrapper we may end up relying on something which could change upstream in Golang (making any docker layer created with our wrapper) "invalid".

I'd love to point out that, while there are tools which supports sparse files (rsync?, tar), that's not the default behavior either (tar -S, for instance).

The proposal in https://bugzilla.redhat.com/show_bug.cgi?id=951564 is to change how the lastlog db is built (from my understanding at least). That makes most sense here, as pointed out in that BZ, people don't really expect to have a 400G lastlog files lying around and tooling not supporting sparse files).

Let's keep this open for now (still). But really we should look forward having lastlog structure modified so it won't end up too large to be tarred up by docker (for instance...).

Sadly you are linking a very old bugzilla that has had no action in the last year.

I know :( unfortunately, as explained above, that's one (if not the only) viable option to me.

Interestingly, ppl at Docker ran into this today with https://github.com/docker/docker/issues/28920 and they realized it could ddos their own docker hub when auto-building images

so what is the status and state of this bug? It sounds like no forward momentum?
Do we have a scope for a fix in 7.4? If not, then it should be re-targeted to a different / future release.

There is no easy fix. Perhaps for lastlog, we could just replace it with an empty file.  Perhaps docker commit should look for sparse files and create empty files if they exist.  Not sure if there is an easy way to check if a file is sparse.  Wrote a script with some notes I found on the internet.

cat /usr/bin/sparse
#!/bin/sh
sparse() {
	file=$1
	if [ "$((`stat -c '%b*%B-%s' -- "$file"`))" -lt 0 ]; then
	    echo "$file" is sparse
	fi
}
while IFS='' read -r file || [[ -n "$file" ]]; do
      sparse "$file"
done


You can execute something like

# find /var/log -type f ! -size 0 | sparse
/var/log/lastlog is sparse

Would Docker be interested in a patch that blocked the import of sparse files on docker commit.

As far as I know this bug is not fixed.

I am providing devel cond_nack for this. This issue is still an open problem and is not fixed even upstream. Can't be fixed yet.

Development Management has reviewed and declined this request. You may appeal this decision by reopening this request.

(In reply to Daniel Walsh from comment #30)
> Would Docker be interested in a patch that blocked the import of sparse
> files on docker commit.

certainly not. Blocking or clobbering sparse files with an empty file would certainly be unpredictable behavior for many apps.

There might be a future with fixed Sparse file support in golang's `archive/tar`. I'm not giving up hope there.

As for detecting files on disk, it's the same in golang as your bash example

```golang
package main

import (
  "errors"
  "fmt"
  "log"
  "os"
  "path/filepath"
  "syscall"
)

func main() {
  err := filepath.Walk(".", func(path string, stat os.FileInfo, err error) error {
    if err != nil {
      return err 
    }   
    if !stat.Mode().IsRegular() {
      return nil 
    }   
    statT := stat.Sys().(*syscall.Stat_t)
    if statT == nil {
      return errors.New("farts: " + path)
    }   
    if ((statT.Blksize * statT.Blocks) - statT.Size) < 0 { 
      fmt.Printf("%s: { Blksize: %d * Blocks: %d - Size: %d = %d}\n", path, statT.Blksize, statT.Blocks, statT.Size, (statT.Blksize*statT.Blocks)-statT.Size)
    }   

    return nil 
  })  
  if err != nil {
    log.Fatal(err)
  }
}
```

Just checking whether the file size is larger that the currently allocated blocks on disk. The next step is then using things like SEEK_HOLE options to jump to the next allocated blocks and counting the offset.

Any movement on this?

https://github.com/golang/go/issues/13548 is still an active design discussion

upstream golang compiler have merged the first support for sparse files. It is queued to be in go1.10. Currently you can review this API here https://tip.golang.org/pkg/archive/tar/

go1.10 release may not happen until beginning of Feb 2018, and at that, this API for sparse file support may change before then, so it is not something to backport.

Given its a new API, that means docker will require changes too? If yes, that means first go 1.10 will be released, then docker will make changes and then we will backport everything in rhel. So this sounds more like april/may 2018 to me.

Please note that lastlog is not the only thing which uses sparse files.

I just ran into this problem while trying to build container with LMDB instance inside. LMDB is extensively used by OpenLDAP and other projects and it creates sparse files by design.

no update, this depends on golang

(In reply to Antonio Murdaca from comment #62)
> no update, this depends on golang

correct. There has been an ongoing effort upstream to get sparse file support. It _almost_ landed for go1.10, but was pulled out at the last minute for further API review. The conversation to re-add it is here https://github.com/golang/go/issues/22735

We have no plans to ship another version of Docker at this time. RHEL7 is in final support stages where only security fixes will get released.  Customers should move to using Podman which is available starting in RHEL 7.6.