Fixed upstream by v0.9.6-165-g5d784bd:

commit 5d784bd6d7b19314b0908aec6b46bfe377aeba42
Author: Jiri Denemark <jdenemar@redhat.com>
Date:   Fri Oct 14 21:24:18 2011 +0200

    Clarify semantics of virDomainMigrate2
    
    Explicitly disallow conflicts between domain name from dxml and dname.

test on
kernel-2.6.32-223.el6.x86_64
libvirt-0.9.9-1.el6.x86_64
qemu-kvm-0.12.1.2-2.213.el6.x86_64

on new.xml only change the domain name to "new"
<domain type='qemu' id='9'>
  <name>new</name>
  <uuid>43ed7ac4-20e3-8c6c-8d9d-b9c672c0eb46</uuid>
...

1. migrate only with --xml changing domain name
# virsh migrate --live kvm-rhel6u2-x86_64-new  qemu+ssh://10.66.83.197/system --xml new.xml 
error: invalid argument: target domain name doesn't match source name nor destination name

2. migrate with --xml and --dname and they setting the different domain name
# virsh migrate --live kvm-rhel6u2-x86_64-new  qemu+ssh://10.66.83.197/system --xml new.xml --dname new2
error: invalid argument: target domain name doesn't match source name nor destination name

3. migrate with --xml and --dname and they setting the same domain name
# virsh migrate --live kvm-rhel6u2-x86_64-new  qemu+ssh://10.66.83.197/system --xml new.xml --dname new
error: internal error Incoming cookie data had unexpected name kvm-rhel6u2-x86_64-new vs new

For the first 2 tests it works as expected, but for the third test, it should not report error

Also I can not find doc on virsh help migrate or virsh man page to explain --dname and --xml conflict for changing domain name

I also find a bug that with setting non exist xml with --xml, virsh migrate will not report error
# virsh migrate --live kvm-rhel6u2-x86_64-new  qemu+ssh://10.66.83.197/system --xml non-exist.xml 
even if it return non zero value

so change the bug status to assigned

(In reply to comment #3)
> I also find a bug that with setting non exist xml with --xml, virsh migrate
> will not report error
> # virsh migrate --live kvm-rhel6u2-x86_64-new  qemu+ssh://10.66.83.197/system
> --xml non-exist.xml 
> even if it return non zero value
> 
I have already filed a new bug for this issue. -- bug 773208

Hmm, the issue you see feels like bug 740533 has returned :-/

Oh, I wasn't quite right. The bug is caused by a patch for this BZ. Destination libvirtd expects to see the original name in the XML and new name in dname parameter. However, if a user provides an XML with the new name, destination daemon will never see the original name.

This issue should be now fixed upstream by v0.9.9-53-g5e31e71:

commit 5e31e713657e2582857c66d121de2fe6c1f3df8c
Author: Jiri Denemark <jdenemar@redhat.com>
Date:   Mon Jan 16 10:24:42 2012 +0100

    Clarify semantics of virDomainMigrate{,ToURI}2
    
    Commit 5d784bd6d7b19314b0908aec6b46bfe377aeba42 was a nice attempt to
    clarify the semantics by requiring domain name from dxml to either match
    original name or dname. However, setting dxml domain name to dname
    doesn't really work since destination host needs to know the original
    domain name to be able to use it in migration cookies. This patch
    requires domain name in dxml to match the original domain name. The
    change should be safe and backward compatible since migration would fail
    just a bit later in the process.

Verify pass on
libvirt-0.9.10-1.el6.x86_64
qemu-kvm-0.12.1.2-2.225.el6.x86_64
kernel-2.6.32-230.el6.x86_64

1. migrate only with --xml changing domain name
# virsh migrate --live kvm-rhel6u2-x86_64-new  qemu+ssh://10.66.83.197/system
--xml new.xml 

error: invalid argument: target domain name doesn't match source name

2. migrate with --xml and --dname and they setting the different domain name, both of them are different from original name
# virsh migrate --live kvm-rhel6u2-x86_64-new  qemu+ssh://10.66.83.197/system
--xml new.xml --dname new2

error: invalid argument: target domain name doesn't match source name

3. migrate with --xml and --dname and they setting the same domain name which different from original name 
# virsh migrate --live kvm-rhel6u2-x86_64-new  qemu+ssh://10.66.83.197/system
--xml new.xml --dname new

error: invalid argument: target domain name doesn't match source name

4. migrate with --xml with original domain name, and --dname with new domain name, migration succeed with no error, on target, the domain name changed to new name

The conclusion is that only --dname can change the domain name during migration, the domain name in dxml must be the same with original name

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2012-0748.html