Bug 472771

Summary: Samba crash causes random outages on rest of server
Product: [Fedora] Fedora Reporter: Max E <max>
Component: kernelAssignee: Kernel Maintainer List <kernel-maint>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: medium    
Version: 9CC: gdeschner, kernel-maint, quintela, ssorce
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2008-12-02 14:25:24 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
output from dmesg showing samba crash none

Description Max E 2008-11-24 14:38:50 UTC
Created attachment 324482 [details]
output from dmesg showing samba crash

Description of problem:
I have now experienced this problem three times - so this is not just a one off. I run an HP Proliant DL380 with 4G of RAM, running Fedora 9 on a 2.6.25.11-97.fc9.x86_64 kernel and the version of Samba is Samba version 3.2.0-17.fc9.  My server is essentially a glorified file server, which hosts an array of 56TB of storage, which is configured using LVM.  The only Samba share is the one looking at the array [see config later on].

At random times [i.e. no pattern so far], my production team are reporting that the 'screen is blank' and I find that X-Windows seems to have died.  NFS has also stopped working, but dmesg reports the following [see attachment].  I have tried restarting X (by init 3 and then init 5), which works, but restarting NFS does not start.  Samba does seem to continue working, but at a heavily reduced speed.  

The only other oddity is that I am running a high-bandwidth 10G CX4 network card on the system, which can be tempremential.  lspci reports it is a:

0b:00.0 Ethernet controller: NetXen Incorporated NXB-10GCX4 10 Gigabit Ethernet PCIe Adapter with CX4 copper interface (rev 25)

The Samba configuration has been checked using testparm reports:

Load smb config files from /etc/samba/smb.conf
Processing section "[Eon4]"
Loaded services file OK.
Server role: ROLE_STANDALONE
Press enter to see a dump of your service definitions
[global]
	workgroup = EDITSHARE_GROUP
	server string = Samba Server Version %v
	interfaces = localhost, eth4
	passdb backend = tdbsam
	log file = /var/log/samba/log.%m
	max log size = 50
	socket options = TCP_NODELAY SO_RCVBUF=131072 SO_SNDBUF=131072
	guest ok = Yes

[Eon4]
	comment = Share To Be Accessable By Everyone
	path = /EonStor
	valid users = admin
	read only = No
	guest ok = No


Version-Release number of selected component (if applicable):

Samba 3.2.0-17.fc9

How reproducible:

Reproduces roughly one a week.  Very reproducible, but I can cannot manually trigger the problem.

Steps to Reproduce:
1.  Wait until users report that server is not responsive
2.  type dmesg
3.
  
Actual results:

Random crashing, and non-functionality of the rest of the server, that cannot be fixed by just restarting services.  Rebooting server is the only fix at the moment.

Expected results:

A smooth running Linux server...please.

Additional info:

Samba configuration file:

# This is the main Samba configuration file. You should read the
# smb.conf(5) manual page in order to understand the options listed
# here. Samba has a huge number of configurable options (perhaps too
# many!) most of which are not shown in this example
#
# For a step to step guide on installing, configuring and using samba, 
# read the Samba-HOWTO-Collection. This may be obtained from:
#  http://www.samba.org/samba/docs/Samba-HOWTO-Collection.pdf
#
# Many working examples of smb.conf files can be found in the 
# Samba-Guide which is generated daily and can be downloaded from: 
#  http://www.samba.org/samba/docs/Samba-Guide.pdf
#
# Any line which starts with a ; (semi-colon) or a # (hash) 
# is a comment and is ignored. In this example we will use a #
# for commentry and a ; for parts of the config file that you
# may wish to enable
#
# NOTE: Whenever you modify this file you should run the command "testparm"
# to check that you have not made any basic syntactic errors. 
#
#---------------
# SELINUX NOTES:
#
# If you want to use the useradd/groupadd family of binaries please run:
# setsebool -P samba_domain_controller on
#
# If you want to share home directories via samba please run:
# setsebool -P samba_enable_home_dirs on
#
# If you create a new directory you want to share you should mark it as
# "samba-share_t" so that selinux will let you write into it.
# Make sure not to do that on system directories as they may already have
# been marked with othe SELinux labels.
#
# Use ls -ldZ /path to see which context a directory has
#
# Set labels only on directories you created!
# To set a label use the following: chcon -t samba_share_t /path
#
# If you need to share a system created directory you can use one of the
# following (read-only/read-write):
# setsebool -P samba_export_all_ro on
# or
# setsebool -P samba_export_all_rw on
#
# If you want to run scripts (preexec/root prexec/print command/...) please
# put them into the /var/lib/samba/scripts directory so that smbd will be
# allowed to run them.
# Make sure you COPY them and not MOVE them so that the right SELinux context
# is applied, to check all is ok use restorecon -R -v /var/lib/samba/scripts
#
#--------------
#
#======================= Global Settings =====================================

[global]

# ----------------------- Netwrok Related Options -------------------------
#
# workgroup = NT-Domain-Name or Workgroup-Name, eg: MIDEARTH
#
# server string is the equivalent of the NT Description field
#
# netbios name can be used to specify a server name not tied to the hostname
#
# Interfaces lets you configure Samba to use multiple interfaces
# If you have multiple network interfaces then you can list the ones
# you want to listen on (never omit localhost)
#
# Hosts Allow/Hosts Deny lets you restrict who can connect, and you can
# specifiy it as a per share option as well
#
	workgroup = EDITSHARE_GROUP
	server string = Samba Server Version %v

	netbios name = EON4

	interfaces = localhost eth4
;	hosts allow = 127. 192.168.10

# --------------------------- Logging Options -----------------------------
#
# Log File let you specify where to put logs and how to split them up.
#
# Max Log Size let you specify the max size log files should reach

	# logs split per machine
	log file = /var/log/samba/log.%m
	# max 50KB per log file, then rotate
	max log size = 50

# ----------------------- Standalone Server Options ------------------------
#
# Scurity can be set to user, share(deprecated) or server(deprecated)
#
# Backend to store user information in. New installations should 
# use either tdbsam or ldapsam. smbpasswd is available for backwards 
# compatibility. tdbsam requires no further configuration.

	security = user
	passdb backend = tdbsam


# ----------------------- Domain Members Options ------------------------
#
# Security must be set to domain or ads
#
# Use the realm option only with security = ads
# Specifies the Active Directory realm the host is part of
#
# Backend to store user information in. New installations should 
# use either tdbsam or ldapsam. smbpasswd is available for backwards 
# compatibility. tdbsam requires no further configuration.
#
# Use password server option only with security = server or if you can't
# use the DNS to locate Domain Controllers
# The argument list may include:
#   password server = My_PDC_Name [My_BDC_Name] [My_Next_BDC_Name]
# or to auto-locate the domain controller/s
#   password server = *


;	realm = MY_REALM

;	password server = <NT-Server-Name>

# ----------------------- Domain Controller Options ------------------------
#
# Security must be set to user for domain controllers
#
# Backend to store user information in. New installations should 
# use either tdbsam or ldapsam. smbpasswd is available for backwards 
# compatibility. tdbsam requires no further configuration.
#
# Domain Master specifies Samba to be the Domain Master Browser. This
# allows Samba to collate browse lists between subnets. Don't use this
# if you already have a Windows NT domain controller doing this job
#
# Domain Logons let Samba be a domain logon server for Windows workstations. 
#
# Logon Scrpit let yuou specify a script to be run at login time on the client
# You need to provide it in a share called NETLOGON
#
# Logon Path let you specify where user profiles are stored (UNC path)
#
# Various scripts can be used on a domain controller or stand-alone
# machine to add or delete corresponding unix accounts
#

;	domain master = yes 
;	domain logons = yes

	# the login script name depends on the machine name
;	logon script = %m.bat
	# the login script name depends on the unix user used
;	logon script = %u.bat
;	logon path = \\%L\Profiles\%u
	# disables profiles support by specifing an empty path
;	logon path =          

;	add user script = /usr/sbin/useradd "%u" -n -g users
;	add group script = /usr/sbin/groupadd "%g"
;	add machine script = /usr/sbin/useradd -n -c "Workstation (%u)" -M -d /nohome -s /bin/false "%u"
;	delete user script = /usr/sbin/userdel "%u"
;	delete user from group script = /usr/sbin/userdel "%u" "%g"
;	delete group script = /usr/sbin/groupdel "%g"


# ----------------------- Browser Control Options ----------------------------
#
# set local master to no if you don't want Samba to become a master
# browser on your network. Otherwise the normal election rules apply
#
# OS Level determines the precedence of this server in master browser
# elections. The default value should be reasonable
#
# Preferred Master causes Samba to force a local browser election on startup
# and gives it a slightly higher chance of winning the election
;	local master = no
;	os level = 33
;	preferred master = yes

#----------------------------- Name Resolution -------------------------------
# Windows Internet Name Serving Support Section:
# Note: Samba can be either a WINS Server, or a WINS Client, but NOT both
#
# - WINS Support: Tells the NMBD component of Samba to enable it's WINS Server
#
# - WINS Server: Tells the NMBD components of Samba to be a WINS Client
#
# - WINS Proxy: Tells Samba to answer name resolution queries on
#   behalf of a non WINS capable client, for this to work there must be
#   at least one	WINS Server on the network. The default is NO.
#
# DNS Proxy - tells Samba whether or not to try to resolve NetBIOS names
# via DNS nslookups.

;	wins support = yes
;	wins server = w.x.y.z
;	wins proxy = yes

;	dns proxy = yes

# --------------------------- Printing Options -----------------------------
#
# Load Printers let you load automatically the list of printers rather
# than setting them up individually
#
# Cups Options let you pass the cups libs custom options, setting it to raw
# for example will let you use drivers on your Windows clients
#
# Printcap Name let you specify an alternative printcap file
#
# You can choose a non default printing system using the Printing option

;	load printers = yes
;	cups options = raw

;	printcap name = /etc/printcap
	#obtain list of printers automatically on SystemV
;	printcap name = lpstat
;	printing = cups

# --------------------------- Filesystem Options ---------------------------
#
# The following options can be uncommented if the filesystem supports
# Extended Attributes and they are enabled (usually by the mount option
# user_xattr). Thess options will let the admin store the DOS attributes
# in an EA and make samba not mess with the permission bits.
#
# Note: these options can also be set just per share, setting them in global
# makes them the default for all shares

;	map archive = no
;	map hidden = no
;	map read only = no
;	map system = no
;	encrypt passwords = yes
	guest ok = yes
;	guest account = nobody
;	username map = /etc/samba/smbusers
;	store dos attributes = yes

# Performance enhancement - ME 30/10/08

 socket options = TCP_NODELAY SO_RCVBUF=131072 SO_SNDBUF=131072


#============================ Share Definitions ==============================

;[homes]
;	comment = Home Directories
;	browseable = no
;	writable = yes
;	valid users = %S
;	valid users = MYDOMAIN\%S

;[printers]
;	comment = All Printers
;	path = /var/spool/samba
;	browseable = no
;	guest ok = no
;	writable = No
;	printable = yes

# Un-comment the following and create the netlogon directory for Domain Logons
;	[netlogon]
;	comment = Network Logon Service
;	path = /var/lib/samba/netlogon
;	guest ok = yes
;	writable = no
;	share modes = no


# Un-comment the following to provide a specific roving profile share
# the default is to use the user's home directory
;	[Profiles]
;	path = /var/lib/samba/profiles
;	browseable = no
;	guest ok = yes


# A publicly accessible directory, but read only, except for people in
# the "staff" group
;	[public]
;	comment = Public Stuff
;	path = /home/samba
;	public = yes
;	writable = yes
;	printable = no
;	write list = +staff



;This is a public share accessible by everyone. This share is LVM configured on 
;/dev/EonStore/lvstorage00 and is on xfs filesystem mounted on /EonStore
;Please run the the command lvdisplay for more info on configured LVM
;[public]
;        comment = Storage on EonStore
;        path = /EonStore
;        public = yes
;        writable = yes
;        printable = no

[Eon4]
	comment = Share To Be Accessable By Everyone
	path = /EonStor
	writeable = yes
	browseable = yes
	guest ok = no
	valid users = admin

Comment 1 Simo Sorce 2008-11-24 14:52:06 UTC
This is a crash in the kernel. It does not matter which component can "cause" it (and honestly I do not see any evidence it can be samba either).

Reassigning to kernel.

Comment 2 Max E 2008-11-24 15:04:50 UTC
Apologies - I thought that the top of the dmesg message might have been the trigger and hence why I pinged it in your direction Simon.

smbd: page allocation failure. order:3, mode:0x4020
Pid: 2833, comm: smbd Not tainted 2.6.25.11-97.fc9.x86_64 #1

I await feedback from the Kernel boys (and girls).

Comment 3 Chuck Ebbert 2008-11-24 21:11:01 UTC
Is there some reason you did not update to the latest kernel before reporting this bug?

Comment 4 Max E 2008-11-26 11:21:17 UTC
Right - I have now updated the kernel to 2.6.27.5-37.fc9.x86_64; are you guys happy to allow me to put this on hold, and see whether this problem is still occurring?

Comment 5 Chuck Ebbert 2008-11-27 06:05:36 UTC
It's probably fixed in 2.6.27 -- the netxen driver was buggy in older kernels. If the error doesn't recur within a few days, close this bug report.

Comment 6 Max E 2008-12-02 14:25:24 UTC
Closing call - new kernel seems to have fixed the problem.  Thanks for your understanding guys!