Bug 698702

Strange, the algorithm didn't change since before f-13.  It seems to kinda-sorta work here where I'm sitting.  I'm using the following makefile to test:

all: $(shell rm -vf tmp > /dev/stderr) $(shell seq -f job-%g 100)
job-%:
	@echo "1" >> tmp
	@echo "enter $@; $$(wc -l<tmp) jobs; load $$(cat /proc/loadavg)"
	@for i in $$(seq $*0); do for j in $$(seq 10); \
        do for k in $$(seq 10); do :; \
	done; done; done
	@sed -i 1d tmp
	@echo "exit $@; $$(wc -l<tmp) jobs; load $$(cat /proc/loadavg)"

# Umm, in my locale, I have to write 1.45 as 1,45, and...
$ ./make-3.82/make -j 10 -l 1,45 -f ~/tmp/load.mk
удалён «tmp» # ... this just says "tmp erased".
enter job-1; 4 jobs; load 0.51 1.14 1.17 7/321 18340
enter job-2; 5 jobs; load 0.51 1.14 1.17 8/321 18341
[...]

The output oscillates between 1 and about 8 jobs in flight here, and the load is about 1.2 to 1.9.  make takes the system-level load into account, so if you have just finished building, or someone else is occupying the machine, it will keep launching one job at a time at the beginning, before the load values get down a bit.  Anyway, it seems to be doing approximately the right thing.

Yes, I seem to have misidentified what exactly wasn't working as expected; passing the -l option seems to be irrelevant to the issue.  I apologize for wasting your time.

The actual problem seems to be the combination of remaking the Makefile and -j N from MAKEFLAGS in the environment.  Given this makefile:

----
things = thing1 thing2 thing3 thing4

all: $(things)

Makefile: foo.c
        touch Makefile

clean:
        touch foo.c

$(things):
        @echo 
        @echo $@ start MAKEFLAGS: $(MAKEFLAGS); sleep 4; echo $@ end
----

"make clean; make -j4" works fine.
"make clean; MAKEFLAGS=-j make" also works fine.
"make clean; MAKEFLAGS=-j4 make" breaks; the MAKEFLAGS passed down to sub-makes has no -j option, so we only get one job at a time.

I believe this is intentional, due to jobserver.  From the manual:

> The ‘-j’ option is a special case (see Parallel Execution). If you set it to
> some numeric value ‘N’ and your operating system supports it (most any UNIX
> system will; others typically won't), the parent make and all the sub-makes
> will communicate to ensure that there are only ‘N’ jobs running at the same
> time between them all.

Also, I can't seem to be able to reproduce it.

$ cat load.mk
all: batch-1 batch-2 batch-3 batch-4
erase:;	$(shell rm -vf tmp > /dev/stderr)
batch-%: erase
	sleep .$$(($$RANDOM / 32))
	$(MAKE) -f load.mk $(shell seq -f job-%g $$((10 + $*)) 10 100)

job-%:
	@echo "1" >> tmp
	@echo "enter $@; $$(wc -l<tmp) jobs; $(MAKEFLAGS)"
	@for i in $$(seq $*0); do for j in $$(seq 10); do for k in $$(seq 10); do :; \
	done; done; done
	@sed -i 1d tmp
	@echo "exit $@; $$(wc -l<tmp) jobs; $(MAKEFLAGS)"

$ MAKEFLAGS=-j4 make -f load.mk
[...]
enter job-11; 4 jobs; w --jobserver-fds=3,4 -j
exit job-13; 3 jobs; w --jobserver-fds=3,4 -j
enter job-23; 4 jobs; w --jobserver-fds=3,4 -j
exit job-11; 3 jobs; w --jobserver-fds=3,4 -j
enter job-21; 4 jobs; w --jobserver-fds=3,4 -j
exit job-12; 3 jobs; w --jobserver-fds=3,4 -j
enter job-22; 4 jobs; w --jobserver-fds=3,4 -j
exit job-14; 3 jobs; w --jobserver-fds=3,4 -j
[...]

("make -f load.mk -j 4" behaves the same way)

$ rpm -q make
make-3.82-3.fc14.x86_64

Your load.mk doesn't seem to rebuild itself; that seems to necessary to reproduce the bug.

Duh, I missed your example makefile :-/  It indeed reproduces the problem, and I don't think it's related to the special handling of -j anymore.

Created attachment 515601 [details]
Erm... let's call it a fix

I'm turning this in reluctantly.  It removes the issue that you see, it doesn't break the test suite, but I had to turn off a bit of logic that someone put there for reasons that I don't understand.  I'll pass this through upstream and see what they have to say.

It is longterm annoying, could there be a Fedora backport before a new upstream release?  (F-16 would be enough for me.)

make-3.82-8.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/make-3.82-8.fc16

Package make-3.82-8.fc16:
* should fix your issue,
* was pushed to the Fedora 16 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing make-3.82-8.fc16'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2011-15358
then log in and leave karma (feedback).

make-3.82-8.fc16 has been pushed to the Fedora 16 stable repository.  If problems still persist, please make note of it in this bug report.