From Bugzilla Helper: User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0 Description of problem: Iscsi works on RHEL5 B2 and within RHEL5 B2 xen kernels. Iscsi should work within Xen guest if Xen is configured with a separate ethernet to the guest. Version-Release number of selected component (if applicable): 2.6.18-1.2747.el5xen How reproducible: Always Steps to Reproduce: 1.et-virt06.lab was setup with 2 ethernets/bridges in Xen. 2.6.18-1.2747.el5xen. xenbr0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:2084 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:304319 (297.1 KiB) TX bytes:0 (0.0 b) xenbr1 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:605618 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:52540992 (50.1 MiB) TX bytes:0 (0.0 b) [root@et-virt06 xen]# more r5g2_iscsi # Automatically generated xen config file name = "v5g2" memory = "512" disk = [ 'phy:/dev/VolGroup00/g2vol,xvda,w', ] vif = [ 'mac=00:16:3e:4e:8b:ec, bridge=xenbr1', 'mac=00:16:3e:4e:8b:bc, bridge=xenbr0' ] nographic=1 uuid = "ca491918-0c5d-b5a6-3a18-b27a6aa6b29c" bootloader="/usr/bin/pygrub" vcpus=1 on_reboot = 'restart' on_crash = 'restart' [root@et-virt06 xen]# 2. Alter /etc/xen/scripts to setup 2nic - help from Jan Mark on this [root@et-virt06 scripts]# more network-shak #!/bin/sh # Exit if anything goes wrong. set -e # First arg is the operation. OP=$1 shift script=/etc/xen/scripts/network-bridge.xen case ${OP} in start) $script start vifnum=1 bridge=xenbr1 netdev=eth1 $script start vifnum=0 bridge=xenbr0 netdev=eth0 ;; stop) $script stop vifnum=1 bridge=xenbr1 netdev=eth1 $script stop vifnum=0 bridge=xenbr0 netdev=eth0 ;; status) $script status vifnum=1 bridge=xenbr1 netdev=eth1 $script status vifnum=0 bridge=xenbr0 netdev=eth0 ;; *) echo 'Unknown command: ' ${OP} echo 'Valid commands are: start, stop, status' exit 1 esac 3. Add to /etc/xen/xend-config.sxp (network-script network-shak) 4. startup guest 5. setup 2nd nic in guest to private net to talk to iscsi device. 6. install iscsi in guest 7. iscsiadm for discovery of luns iscsiadm -m discovery -t sendtargets -p 10.10.10.5 8. restart iscsi for discovery - 9. CRASH Actual Results: root@dhcp78-229 tools]# service iscsi restart Stopping iSCSI initiator service: [ OK ] Turning off network shutdown.Starting iSCSI initiator service: SCSI subsystem initialized [ OK ] Setting up iSCSI targets: Unable to handle kernel NULL pointer dereference at 0000000000000078 RIP: [<ffffffff8833d3a6>] :scsi_mod:scsi_calculate_bounce_limit+0x15/0x49 PGD f2e2067 PUD f2d2067 PMD 0 Oops: 0000 [1] SMP last sysfs file: /class/scsi_host/host0/scan CPU 0 Modules linked in: iscsi_tcp libiscsi scsi_transport_iscsi scsi_mod nfs lockd fscache nfs_acl autofs4 i2c_dev i2c_core hidp rfcomm l2cap bluetooth sunrpc xennet ip_conntrack_netbios_ns ipt_REJECT iptable_filter ip_tables xt_state ip_conntrack nfnetlink xt_tcpudp ip6table_filter ip6_tables x_tables ipv6 parport_pc lp parport pcspkr dm_snapshot dm_zero dm_mirror dm_mod xenblk ext3 jbd ehci_hcd ohci_hcd uhci_hcd Pid: 2094, comm: iscsid Not tainted 2.6.18-1.2746.el5xen #1 RIP: e030:[<ffffffff8833d3a6>] [<ffffffff8833d3a6>] :scsi_mod:scsi_calculate_bounce_limit+0x15/0x49 RSP: e02b:ffff88000f223c30 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff88001035b8e8 RCX: 0000000000000071 RDX: 0000000000000001 RSI: 0000000000000400 RDI: ffff880010505000 RBP: ffff880010505000 R08: ffff88001fde3000 R09: ffff880000e13080 R10: ffffffff8833e8dc R11: 0000000000000048 R12: ffff88000f0bf428 R13: ffff880010505000 R14: 0000000000000000 R15: ffff88000f0bf400 FS: 00002aaaaaac6270(0000) GS:ffffffff8058e000(0000) knlGS:0000000000000000 CS: e033 DS: 0000 ES: 0000 Process iscsid (pid: 2094, threadinfo ffff88000f222000, task ffff88000f8897a0) Stack: ffffffff8833da3c ffff88001014f000 ffff88001014f000 ffff88000f0bf400 ffffffff8833f629 ffff880010505170 00000001883391fe ffff880010505000 0000000000000000 0000000000000000 Call Trace: [<ffffffff8833da3c>] :scsi_mod:scsi_alloc_queue+0x65/0xb6 [<ffffffff8833f629>] :scsi_mod:scsi_alloc_sdev+0x12e/0x1d2 [<ffffffff8833f858>] :scsi_mod:scsi_probe_and_add_lun+0x10d/0x9ca [<ffffffff88340458>] :scsi_mod:scsi_alloc_target+0x21e/0x320 [<ffffffff8834069d>] :scsi_mod:__scsi_scan_target+0xc3/0x5e2 [<ffffffff8024b61c>] sscanf+0x49/0x51 [<ffffffff88340e66>] :scsi_mod:scsi_scan_target+0x6c/0x83 [<ffffffff8836864f>] :scsi_transport_iscsi:iscsi_user_scan+0x65/0x94 [<ffffffff883417ac>] :scsi_mod:store_scan+0x9b/0xc5 [<ffffffff802e89e1>] sysfs_write_file+0xb9/0xe8 [<ffffffff8021624c>] vfs_write+0xce/0x174 [<ffffffff80216a48>] sys_write+0x45/0x6e [<ffffffff8025c65d>] tracesys+0xa7/0xb2 Code: 83 78 78 00 75 0c 48 8b 05 75 08 32 f8 48 c1 e0 0c c3 48 8b RIP [<ffffffff8833d3a6>] :scsi_mod:scsi_calculate_bounce_limit+0x15/0x49 RSP <ffff88000f223c30> CR2: 0000000000000078 <0>Kernel panic - not syncing: Fatal exception Expected Results: [root@et-virt06 ~]# service iscsi restart Stopping iSCSI initiator service: [ OK ] Turning off network shutdown.Starting iSCSI initiator service: [ OK ] Setting up iSCSI targets: iscsiadm: [ OK ] eth0 Link encap:Ethernet HWaddr 00:13:72:4C:A2:4A inet addr:10.10.10.23 Bcast:10.10.10.255 Mask:255.255.255.0 inet6 addr: fe80::213:72ff:fe4c:a24a/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:98768 errors:0 dropped:0 overruns:0 frame:0 TX packets:92595 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:9974079 (9.5 MiB) TX bytes:8338626 (7.9 MiB) Additional info:
Verified issue with 2747 kernel -------- [root@dhcp78-229 ~]# ifconfig eth0 Link encap:Ethernet HWaddr 00:16:3E:4E:8B:EC inet addr:192.168.78.229 Bcast:192.168.79.255 Mask:255.255.252.0 inet6 addr: fe80::216:3eff:fe4e:8bec/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:837 errors:0 dropped:0 overruns:0 frame:0 TX packets:67 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:95801 (93.5 KiB) TX bytes:9169 (8.9 KiB) eth1 Link encap:Ethernet HWaddr 00:16:3E:4E:8B:BC inet addr:10.10.10.22 Bcast:10.10.10.255 Mask:255.255.255.0 inet6 addr: fe80::216:3eff:fe4e:8bbc/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:101 errors:0 dropped:0 overruns:0 frame:0 TX packets:53 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:9102 (8.8 KiB) TX bytes:7228 (7.0 KiB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:8 errors:0 dropped:0 overruns:0 frame:0 TX packets:8 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:560 (560.0 b) TX bytes:560 (560.0 b) [root@dhcp78-229 ~]# service iscsi start Turning off network shutdown.Starting iSCSI initiator service: SCSI subsystem initialized [ OK ] Setting up iSCSI targets: iscsiadm: main: node [iqn.2001-05.com.equallogic:6-8a0900-769a80002-538273305d24537d-red6,10.10.10.5,3260] iscsiadm: Looking for config file /var/lib/iscsi/nodes/iqn.2001-05.com.equallogic:6-8a0900-769a80002-538273305d24537d-red6/10.10.10.5,3260 iscsiadm: updated 'node.name', '' => 'iqn.2001-05.com.equallogic:6-8a0900-769a80002-538273305d24537d-red6' iscsiadm: updated 'node.transport_name', 'tcp' => 'tcp' iscsiadm: updated 'node.tpgt', '-1' => '1' iscsiadm: updated 'node.active_conn', '1' => '1' iscsiadm: updated 'node.startup', 'manual' => 'automatic' iscsiadm: updated 'node.session.initial_cmdsn', '0' => '0' iscsiadm: updated 'node.session.auth.authmethod', 'None' => 'None' iscsiadm: updated 'node.session.timeo.replacement_timeout', '120' => '120' iscsiadm: updated 'node.session.err_timeo.abort_timeout', '10' => '10' iscsiadm: updated 'node.session.err_timeo.reset_timeout', '30' => '30' iscsiadm: updated 'node.session.iscsi.InitialR2T', 'No' => 'No' iscsiadm: updated 'node.session.iscsi.ImmediateData', 'Yes' => 'Yes' iscsiadm: updated 'node.session.iscsi.FirstBurstLength', '262144' => '262144' iscsiadm: updated 'node.session.iscsi.MaxBurstLength', '16776192' => '16776192' iscsiadm: updated 'node.session.iscsi.DefaultTime2Retain', '0' => '0' iscsiadm: updated 'node.session.iscsi.DefaultTime2Wait', '0' => '0' iscsiadm: updated 'node.session.iscsi.MaxConnections', '1' => '0' iscsiadm: updated 'node.session.iscsi.MaxOutstandingR2T', '1' => '1' iscsiadm: updated 'node.session.iscsi.ERL', '0' => '0' iscsiadm: updated 'node.conn[0].address', '' => '10.10.10.5' iscsiadm: updated 'node.conn[0].port', '3260' => '3260' iscsiadm: updated 'node.conn[0].startup', 'manual' => 'automatic' iscsiadm: updated 'node.conn[0].tcp.window_size', '524288' => '524288' iscsiadm: updated 'node.conn[0].tcp.type_of_service', '0' => '0' iscsiadm: updated 'node.conn[0].timeo.login_timeout', '15' => '15' iscsiadm: updated 'node.conn[0].timeo.auth_timeout', '45' => '45' iscsiadm: updated 'node.conn[0].timeo.active_timeout', '5' => '5' iscsiadm: updated 'node.conn[0].timeo.idle_timeout', '60' => '60' iscsiadm: updated 'node.conn[0].timeo.ping_timeout', '5' => '5' iscsiadm: updated 'node.conn[0].timeo.noop_out_interval', '0' => '0' iscsiadm: updated 'node.conn[0].timeo.noop_out_timeout', '0' => '0' iscsiadm: updated 'node.conn[0].iscsi.MaxRecvDataSegmentLength', '131072' => '65536' iscsiadm: updated 'node.conn[0].iscsi.HeaderDigest', 'None,CRC32C' => 'None' iscsiadm: updated 'node.conn[0].iscsi.DataDigest', 'None' => 'None' iscsiadm: updated 'node.conn[0].iscsi.IFMarker', 'No' => 'No' iscsiadm: updated 'node.conn[0].iscsi.OFMarker', 'No' => 'No' Unable to handle kernel NULL pointer dereference at 0000000000000078 RIP: [<ffffffff8823a3a6>] :scsi_mod:scsi_calculate_bounce_limit+0x15/0x49 PGD 14055067 PUD 1404d067 PMD 0 Oops: 0000 [1] SMP last sysfs file: /class/scsi_host/host0/scan CPU 0 Modules linked in: iscsi_tcp libiscsi scsi_transport_iscsi scsi_mod autofs4 i2c_dev i2c_core hidp rfcomm l2cap bluetooth sunrpc xennet ipv6 parport_pc lp parport pcspkr dm_snapshot dm_zero dm_mirror dm_mod xenblk ext3 jbd ehci_hcd ohci_hcd uhci_hcd Pid: 1792, comm: iscsid Not tainted 2.6.18-1.2747.el5xen #1 RIP: e030:[<ffffffff8823a3a6>] [<ffffffff8823a3a6>] :scsi_mod:scsi_calculate_bounce_limit+0x15/0x49 RSP: e02b:ffff880014065c30 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff8800140818e8 RCX: 0000000000000071 RDX: 0000000000000001 RSI: 0000000000000400 RDI: ffff880014ee2800 RBP: ffff880014ee2800 R08: ffff88001fde3000 R09: ffff880000e13080 R10: ffffffff8823b8dc R11: 0000000000000048 R12: ffff880014783828 R13: ffff880014ee2800 R14: 0000000000000000 R15: ffff880014783800 FS: 00002aaaaaac6270(0000) GS:ffffffff8058e000(0000) knlGS:0000000000000000 CS: e033 DS: 0000 ES: 0000 Process iscsid (pid: 1792, threadinfo ffff880014064000, task ffff88001467b080) Stack: ffffffff8823aa3c ffff8800147b9000 ffff8800147b9000 ffff880014783800 ffffffff8823c629 ffff880014ee2970 00000001882361fe ffff880014ee2800 0000000000000000 0000000000000000 Call Trace: [<ffffffff8823aa3c>] :scsi_mod:scsi_alloc_queue+0x65/0xb6 [<ffffffff8823c629>] :scsi_mod:scsi_alloc_sdev+0x12e/0x1d2 [<ffffffff8823c858>] :scsi_mod:scsi_probe_and_add_lun+0x10d/0x9ca [<ffffffff8823d458>] :scsi_mod:scsi_alloc_target+0x21e/0x320 [<ffffffff8823d69d>] :scsi_mod:__scsi_scan_target+0xc3/0x5e2 [<ffffffff8024b61c>] sscanf+0x49/0x51 [<ffffffff8823de66>] :scsi_mod:scsi_scan_target+0x6c/0x83 [<ffffffff8826564f>] :scsi_transport_iscsi:iscsi_user_scan+0x65/0x94 [<ffffffff8823e7ac>] :scsi_mod:store_scan+0x9b/0xc5 [<ffffffff802e89e1>] sysfs_write_file+0xb9/0xe8 [<ffffffff8021624c>] vfs_write+0xce/0x174 [<ffffffff80216a48>] sys_write+0x45/0x6e [<ffffffff8025c65d>] tracesys+0xa7/0xb2 Code: 83 78 78 00 75 0c 48 8b 05 75 38 42 f8 48 c1 e0 0c c3 48 8b RIP [<ffffffff8823a3a6>] :scsi_mod:scsi_calculate_bounce_limit+0x15/0x49 RSP <ffff880014065c30> CR2: 0000000000000078 <0>Kernel panic - not syncing: Fatal exception [root@et-virt06 ~]#
This was on virtual list: [PATCH] x86_64, pci_iommu_alloc, dma_ops and iSCSI Here is the mail and patch. markmc found the bug and made the patch and I think pushed upstream and in RHEL. I ccd him because he knows more about this than me. Hi, With iSCSI root on x86_64 DomU I was seeing an oops in scsi_calculate_bounce_limit() where PCI_DMA_BUS_IS_PHYS() was called. Issue is that dma_ops is NULL. Attached patch fixes it for me - call no_iommu_init() as a fallback. This is what upstream Xen does, except upstream does it in x86_64/mm/init-xen.c:mem_init() whereas, for some reason, we have this stuff in i386/kernel/pci-dma-xen.c:pci_iommu_init(). Not sure where the divergence from upstream came from. Cheers, Mark. --- ./arch/i386/kernel/pci-dma-xen.c.x86_64-no-iommu 2006-10-02 10:58:11.000000000 +0100 +++ ./arch/i386/kernel/pci-dma-xen.c 2006-10-02 10:58:51.000000000 +0100 @@ -18,6 +18,7 @@ #include <asm/tlbflush.h> #include <asm-i386/mach-xen/asm/swiotlb.h> #include <asm/bug.h> +#include <asm/proto.h> #ifdef __x86_64__ int iommu_merge __read_mostly = 0; @@ -64,6 +65,7 @@ #ifdef CONFIG_SWIOTLB pci_swiotlb_init(); #endif + no_iommu_init(); } __init int iommu_setup(char *p)
Yep, it's bug #209429 *** This bug has been marked as a duplicate of 209429 ***
This bug was closed prematurely; can somebody please verify if the issue is still present on RHEL-5?
Yes, I cloned #209429 for RHEL5 as #215581, so this is actually a dup of the latter *** This bug has been marked as a duplicate of 215581 ***
Great, thanks.
I will retry using nightly/RHEL5-Server-20070120.nightly/5/i386
Everything installed nicely now to a 500 MB slab local xvd for /boot then to iscsi storage, see all devices, choose sda1 for / 50GB and 4GB for swap. Install worked fine. Reboot initially hung on >audit(1169502268.280:3): policy loaded auid=4294967295 >SELinux: initialized (dev usbfs, type usbfs), uses genfs_contexts After manually disabling SElinux using init level 1 I am now seeing hang when startup tries to up loopback device >Bringing up loopback interface: [ OK ] If I manually bring up the loopback after init level 1, it works/boots The problem appears to be getting a 2nd nic for the building as a network after it used eth0 for iSCSI. [FULL trace below] collecting xm dump-core Loading xenblk.ko module Registering block device major 202 xvda: xvda1 Loading dm-mod.ko module device-mapper: ioctl: 4.11.0-ioctl (2006-09-14) initialised: dm-devel Loading dm-mirror.ko module Loading dm-zero.ko module Loading dm-snapshot.ko module Waiting for driver initialization. Making device-mapper control node Bringing up eth0 Attaching to iSCSI storage iscsistart: transport class version 2.0-685. iscsid version 2.0-742 iscsistart: version 2.0-742 scsi0 : iSCSI Initiator over TCP/IP iscsistart: connection0:0 is operational now Vendor: EQLOGIC Model: 100E-00 Rev: 3.1 Type: Direct-Access ANSI SCSI revision: 05 SCSI device sda: 111175680 512-byte hdwr sectors (56922 MB) sda: Write Protect is off SCSI device sda: drive cache: write through SCSI device sda: 111175680 512-byte hdwr sectors (56922 MB) sda: Write Protect is off SCSI device sda: drive cache: write through sda: sda1 sd 0:0:0:0: Attached scsi disk sda Scanning logical volumes Reading all physical volumes. This may take a while... Found volume group "VolGroup00" using metadata type lvm2 Activating logical volumes 2 logical volume(s) in volume group "VolGroup00" now active Creating root device. Mounting root filesystem. EXT3-fs: INFO: recovery required on readonly filesystem. EXT3-fs: write access will be enabled during recovery. kjournald starting. Commit interval 5 seconds EXT3-fs: recovery complete. EXT3-fs: mounted filesystem with ordered data mode. Setting up other filesystems. Setting up new root fs no fstab.sys, mounting internal defaults Switching to new root and running init. unmounting old /dev unmounting old /proc unmounting old /sys SELinux: Disabled at runtime. SELinux: Unregistering netfilter hooks audit(1169503010.730:2): selinux=0 auid=4294967295 INIT: version 2.86 booting Welcome to Red Hat Enterprise Linux Server Press 'I' to enter interactive startup. Setting clock (utc): Mon Jan 22 16:56:55 EST 2007 [ OK ] Starting udev: [ OK ] Loading default keymap (us): [ OK ] Setting hostname localhost.localdomain: [ OK ] Setting up Logical Volume Management: 2 logical volume(s) in volume group "VolGroup00" now active [ OK ] Checking filesystems Checking all file systems. [/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/VolGroup00/LogVol00 /dev/VolGroup00/LogVol00: clean, 83153/12799776 files, 881347/12795904 blocks [/sbin/fsck.ext3 (1) -- /boot] fsck.ext3 -a /dev/xvda1 /boot: recovering journal /boot: clean, 35/128520 files, 33472/514048 blocks [ OK ] Remounting root filesystem in read-write mode: [ OK ] Mounting local filesystems: [ OK ] Enabling local filesystem quotas: [ OK ] Enabling /etc/fstab swaps: [ OK ] INIT: Entering runlevel: 5 Entering non-interactive startup Starting background readahead: [ OK ] Checking for hardware changes [ OK ] Applying ip6tables firewall rules: [ OK ] Applying iptables firewall rules: [ OK ] Loading additional iptables modules: ip_conntrack_netbios_ns [ OK ] Bringing up loopback interface: [ OK ]