My qemu-kvm command line:
# /usr/libexec/qemu-kvm -M rhel6.5.0 -cpu SandyBridge -enable-kvm -m 4096 -smp 4,sockets=2,cores=2,threads=1 -no-kvm-pit-reinjection -name sluo -uuid 43425b70-86e5-4664-bf2c-3b76699b8bec -rtc base=localtime,clock=host,driftfix=slew -device virtio-serial-pci,id=virtio-serial0,max_ports=16,vectors=0,bus=pci.0,addr=0x3 -chardev socket,id=channel1,path=/tmp/helloworld1,server,nowait -device virtserialport,chardev=channel1,name=com.redhat.rhevm.vdsm.1,bus=virtio-serial0.0,id=port1,nr=1 -chardev socket,id=channel2,path=/tmp/helloworld2,server,nowait -device virtserialport,chardev=channel2,name=com.redhat.rhevm.vdsm.2,bus=virtio-serial0.0,id=port2,nr=2 -drive file=/home/rhel6.5-64-copy.qcow2,if=none,id=drive-system-disk,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop,serial="QEMU-DISK1" -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-system-disk,id=system-disk,bootindex=1 -device virtio-balloon-pci,id=ballooning,bus=pci.0,addr=0x5 -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -netdev tap,id=hostnet0,vhost=off,script=/etc/qemu-ifup -device virtio-net-pci,netdev=hostnet0,id=virtio-net-pci0,mac=2C:41:38:B6:40:21,bus=pci.0,addr=0x6,bootindex=2 -k en-us -boot menu=on -qmp tcp:0:4444,server,nowait -serial unix:/tmp/ttyS0,server,nowait -vnc :1 -spice port=5931,disable-ticketing -monitor stdio

My host cpu info:
# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                8
On-line CPU(s) list:   0-7
Thread(s) per core:    2
Core(s) per socket:    4
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 42
Stepping:              7
CPU MHz:               3401.000
BogoMIPS:              6782.68
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              8192K
NUMA node0 CPU(s):     0-7

As far as I can understand, what's actually happening is that you're pushing the balloon operation to the limit and it's failing. Even if free tells you that you have used only 442MB, you also got memory cached and memory usage can vary a little bit during the balloon operation.

When the balloon driver fails to allocate memory, it prints that "Out of puff" message and goes to sleep a little bit and then retries. It keeps trying and never return a failure to the user/host.

I wouldn't say it's a bug, it's how it was designed to work. To change this behavior we could hack it (say, print the message only once) or change it to gave up on trying and return a failure to the user somehow (would require changes on QEMU, and possibly on the virtio-spec). I'd say both solutions are unlikely to fly on upstream and I doubt are worth trying.

I'll keep this open, but IMO this is very low priority.

I'm revisiting this bug, let me confirm something: you keep getting those messages even _after_ the operation finished? I mean, you do the following:

1. balloon 600
2. info balloon and query-balloon report the operation finished at 600MB
3. you log into the guest and free also reports the operation finished
4. you keep getting the error messages in the guest

If this is what happens then this might be a bug. If you get those messages while the operation is taking place then comment 3 applies.

Hi, Sibiao

Could you please check Luiz's comment 4 and reply it?  Thanks.

(In reply to Luiz Capitulino from comment #4)
> I'm revisiting this bug, let me confirm something: you keep getting those
> messages even _after_ the operation finished? I mean, you do the following:
> 
> 1. balloon 600
> 2. info balloon and query-balloon report the operation finished at 600MB
> 3. you log into the guest and free also reports the operation finished
> 4. you keep getting the error messages in the guest
> 
> If this is what happens then this might be a bug. If you get those messages
> while the operation is taking place then comment 3 applies.
The guest kernel will flood with prompting message "virtio_balloon virtio2: Out of puff! Can't get 256 pages" during the balloon operation changing process.

And if the balloon operation finished at 600MB, there is no any error message display any more.

Best Regards,
sluo

As explained in comment 3 this is how the balloon was designed to work. If you push the balloon operation to the limit (that's, you reduce the guest to the point where it may start running out of memory) then those messages will be printed and two things may happen: the guest dies out of memory or the balloon operation eventually succeeds. The latter happened in your case.