Bug 450543

Summary: glx-gears segfaults running metacity or compiz, i945
Product: [Fedora] Fedora Reporter: Tom London <selinux>
Component: mesaAssignee: Adam Jackson <ajax>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: low Docs Contact:
Priority: low    
Version: 10CC: ajax, dominik, dwalsh, lordmorgul, tomek
Target Milestone: ---Keywords: SELinux
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2008-12-30 03:39:55 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Xorg.0.log for above crashes
none
Xorg.0.log
none
/var/log/Xorg.0.log with X, gdm and compiz running
none
output of 'runcon -t unconfined_execmem_t valgrind glxgears'
none
run of 'runcon -t unconfined_execmem_t valgrind glxgears' with debuginfo packages none

Description Tom London 2008-06-09 13:53:14 UTC 
Description of problem:
glxgears segfaults on Thinkpad X60 running Rawhide.

I run without any xorg.conf.

Here is stacktrace:

[tbl@localhost ~]$ gdb glxgears
GNU gdb Fedora (6.8-10.fc10)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...
(gdb) run
Starting program: /usr/bin/glxgears 
[Thread debugging using libthread_db enabled]
[New Thread 0xb7fc66c0 (LWP 4631)]

Program received signal SIGSEGV, Segmentation fault.
0x003a49d4 in emit_1ub (p=<value optimized out>, b0=<value optimized out>)
    at x86/rtasm/x86sse.c:64
64	   *csr++ = b0;
Missing separate debuginfos, use: debuginfo-install libXdamage.i386
libXxf86vm.i386 libdrm.i386
(gdb) where
#0  0x003a49d4 in emit_1ub (p=<value optimized out>, b0=<value optimized out>)
    at x86/rtasm/x86sse.c:64
#1  0x003a6804 in x86_push (p=0xbfce18e4, reg=
      {file = 0, idx = 5, mod = 3, disp = 0}) at x86/rtasm/x86sse.c:313
#2  0x00347a01 in _tnl_generate_sse_emit (ctx=0x944cb20)
    at tnl/t_vertex_sse.c:359
#3  0x00346dc3 in choose_emit_func (ctx=0x944cb20, count=162, 
    dest=0xb20f1020 "") at tnl/t_vertex.c:136
#4  0x00346fb0 in _tnl_build_vertices (ctx=0x944cb20, start=0, end=162, 
    newinputs=4294967295) at tnl/t_vertex.c:418
#5  0x0033b046 in run_render (ctx=0x944cb20, stage=0x948f80c)
    at tnl/t_vb_render.c:290
#6  0x00333fe1 in _tnl_run_pipeline (ctx=0x944cb20) at tnl/t_pipeline.c:158
#7  0x002452d1 in intelRunPipeline (ctx=0x944cb20) at intel_tris.c:906
#8  0x00334a2c in _tnl_draw_prims (ctx=0x944cb20, arrays=0x947e428, 
    prim=0x94bf640, nr_prims=2, ib=0x0, min_index=0, max_index=161)
    at tnl/t_draw.c:402
#9  0x00333744 in vbo_save_playback_vertex_list (ctx=0x944cb20, data=0x94bf274)
    at vbo/vbo_save_draw.c:220
#10 0x002be007 in execute_list (ctx=0x944cb20, list=<value optimized out>)
    at main/dlist.c:5727
#11 0x002c0492 in _mesa_CallList (list=1) at main/dlist.c:6811
#12 0x0031e908 in neutral_CallList (i=1) at main/vtxfmt_tmp.h:298
---Type <return> to continue, or q <return> to quit---
#13 0x0804994f in do_draw () at glxgears.c:254
#14 0x0804a39f in main (argc=1, argv=0xbfce22b4) at glxgears.c:304
(gdb) quit
The program is running.  Exit anyway? (y or n) y
[tbl@localhost ~]$ 


Version-Release number of selected component (if applicable):
mesa-libGLU-devel-7.1-0.31.fc9.i386
mesa-libGL-devel-7.1-0.31.fc9.i386
mesa-libOSMesa-7.1-0.31.fc9.i386
mesa-debuginfo-7.1-0.31.fc9.i386
mesa-libGL-7.1-0.31.fc9.i386
mesa-libGLU-7.1-0.31.fc9.i386
mesa-libOSMesa-devel-7.1-0.31.fc9.i386
glx-utils-7.1-0.31.fc9.i386

How reproducible:
Every time

Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:
Comment 1 Tom London 2008-06-09 13:54:33 UTC 
Created attachment 308687 [details]
Xorg.0.log for above crashes

Here is Xorg.0.log generated during the above session.
Comment 2 Tom London 2008-06-09 13:59:25 UTC 
Figure this out a bit:

glxgears is generating an execmem AVC.  Putting system into permissive mode
makes this work:

type=AVC msg=audit(1213018957.010:27): avc:  denied  { execmem } for  pid=4631
comm="glxgears" scontext=unconfined_u:unconfined_r:unconfined_t:s0
tcontext=unconfined_u:unconfined_r:unconfined_t:s0 tclass=process
type=SYSCALL msg=audit(1213018957.010:27): arch=40000003 syscall=192 success=no
exit=-13 a0=0 a1=a00000 a2=7 a3=22 items=0 ppid=4628 pid=4631 auid=500 uid=500
gid=500 euid=500 suid=500 fsuid=500 egid=500 sgid=500 fsgid=500 tty=pts1 ses=1
comm="glxgears" exe="/usr/bin/glxgears"
subj=unconfined_u:unconfined_r:unconfined_t:s0 key=(null)
type=AVC msg=audit(1213019478.587:28): avc:  denied  { execmem } for  pid=5045
comm="glxgears" scontext=unconfined_u:unconfined_r:unconfined_t:s0
tcontext=unconfined_u:unconfined_r:unconfined_t:s0 tclass=process
type=SYSCALL msg=audit(1213019478.587:28): arch=40000003 syscall=192 success=no
exit=-13 a0=0 a1=a00000 a2=7 a3=22 items=0 ppid=3284 pid=5045 auid=500 uid=500
gid=500 euid=500 suid=500 fsuid=500 egid=500 sgid=500 fsgid=500 tty=pts1 ses=1
comm="glxgears" exe="/usr/bin/glxgears"
subj=unconfined_u:unconfined_r:unconfined_t:s0 key=(null)
type=ANOM_ABEND msg=audit(1213019478.589:29): auid=500 uid=500 gid=500 ses=1
subj=unconfined_u:unconfined_r:unconfined_t:s0 pid=5045 comm="glxgears" sig=11
Comment 3 Tom London 2008-06-27 15:02:35 UTC 
glxgears works fine if run with LIBGL_ALWAYS_INDIRECT=1

No execmem AVC then, but it is much slower:

[tbl@localhost ~]$ LIBGL_ALWAYS_INDIRECT=1 glxgears
3438 frames in 5.0 seconds = 685.601 FPS
3418 frames in 5.0 seconds = 683.218 FPS
^C
[tbl@localhost ~]$ 
[tbl@localhost ~]$ glxgears
Segmentation fault
[tbl@localhost ~]$ 

<<< Put system in permissive mode >>>

[tbl@localhost ~]$ glxgears
4682 frames in 5.0 seconds = 936.400 FPS
4606 frames in 5.0 seconds = 921.170 FPS
^C
[tbl@localhost ~]$
Comment 4 Tom London 2008-08-28 18:27:47 UTC 
This continues with:

[tbl@localhost ~]$ rpm -qa mesa\* glx\* xorg-x11-server\* xorg-x11-drv-i810
mesa-libOSMesa-7.1-0.37.fc10.i386
glx-utils-7.1-0.37.fc10.i386
xorg-x11-server-utils-7.4-3.fc10.i386
xorg-x11-server-common-1.4.99.906-10.fc10.i386
mesa-libGLU-devel-7.1-0.37.fc10.i386
mesa-dri-drivers-7.1-0.37.fc10.i386
mesa-libGL-7.1-0.37.fc10.i386
xorg-x11-drv-i810-2.4.2-1.fc10.i386
xorg-x11-server-Xorg-1.4.99.906-10.fc10.i386
mesa-libGLU-7.1-0.37.fc10.i386
mesa-libOSMesa-devel-7.1-0.37.fc10.i386
mesa-libGL-devel-7.1-0.37.fc10.i386
[tbl@localhost ~]$ 

Same behavior/performance as in #3, and same AVC:

type=AVC msg=audit(1219947405.708:30): avc:  denied  { execmem } for  pid=4211 comm="glxgears" scontext=unconfined_u:unconfined_r:unconfined_t:s0 tcontext=unconfined_u:unconfined_r:unconfined_t:s0 tclass=process
type=SYSCALL msg=audit(1219947405.708:30): arch=40000003 syscall=192 success=yes exit=14258176 a0=0 a1=a00000 a2=7 a3=22 items=0 ppid=3503 pid=4211 auid=500 uid=500 gid=500 euid=500 suid=500 fsuid=500 egid=500 sgid=500 fsgid=500 tty=pts1 ses=1 comm="glxgears" exe="/usr/bin/glxgears" subj=unconfined_u:unconfined_r:unconfined_t:s0 key=(null)

or

#============= unconfined_t ==============
allow unconfined_t self:process execmem;

Running this as "unconfined_execmem_t" works:

[tbl@localhost ~]$ runcon -t unconfined_execmem_t glxgears
4075 frames in 5.0 seconds = 814.913 FPS
4293 frames in 5.0 seconds = 858.460 FPS
XIO:  fatal IO error 104 (Connection reset by peer) on X server ":0.0"
      after 33695 requests (33562 known processed) with 0 events remaining.
[tbl@localhost ~]$ 

Since the performance is pretty bad with LIBGL_ALWAYS_INDIRECT=1 (slower and very bad flicker), I'm presuming that this 'execmem' should be allowed or fixed.
Comment 5 Daniel Walsh 2008-08-29 19:07:16 UTC 
I am not seeing this on my rawhide box.  It seems to be working fine without any avc's.

rpm -qa mesa\* glx\* xorg-x11-server\* xorg-x11-drv-i810
mesa-libGL-7.1-0.37.fc10.x86_64
xorg-x11-drv-i810-2.4.2-1.fc10.x86_64
xorg-x11-server-common-1.4.99.906-10.fc10.x86_64
mesa-libGLU-7.1-0.37.fc10.x86_64
xorg-x11-server-utils-7.4-3.fc10.x86_64
mesa-libGLU-devel-7.1-0.37.fc10.x86_64
xorg-x11-server-Xorg-1.4.99.906-10.fc10.x86_64
mesa-dri-drivers-7.1-0.37.fc10.x86_64
mesa-libGL-devel-7.1-0.37.fc10.x86_64
glx-utils-7.1-0.37.fc10.x86_64
xorg-x11-server-Xephyr-1.4.99.906-10.fc10.x86_64
Comment 6 Tom London 2008-08-29 23:30:18 UTC 
Not sure its important, but I'm also running compiz:

compizconfig-python-0.7.6-1.fc10.i386
compiz-fusion-extras-0.7.6-6.fc10.i386
compiz-manager-0.6.0-8.fc10.noarch
compizconfig-backend-gconf-0.7.6-1.fc10.i386
compiz-fusion-extras-gnome-0.7.6-6.fc10.i386
compiz-gnome-0.7.6-10.fc10.i386
compiz-0.7.6-10.fc10.i386
compiz-fusion-0.7.6-6.fc10.i386
compiz-fusion-gnome-0.7.6-6.fc10.i386

I start it with a slightly augmented "fusion-icon" script:
#!/bin/bash

LIBGL_ALWAYS_INDIRECT=1 
export LIBGL_ALWAYS_INDIRECT 
exec /usr/bin/fusion-icon &

like:
[tbl@localhost ~]$ fusion-icon.sh
[tbl@localhost ~]$  * Detected Session: gnome
 * Searching for installed applications...
 * Intel detected, exporting: INTEL_BATCH=1
 * Using the GTK Interface
 * Starting Compiz
 ... executing: compiz --replace --sm-disable --ignore-desktop-hints ccp --indirect-rendering


I'm running this on a Thinkpad X60 (with intel 945 graphics).
Comment 7 Andrew Farris 2008-09-15 00:44:27 UTC 
I'm seeing a similar problem running anything glx right now on rawhide, freshly installed 20080912.  Any glx app crashes with a segmentation fault, and throws up an AVC for execmem.  I have glxinfo reporting direct rendering yes.

But if I export LIBGL_ALWAYS_INDIRECT then I do not get the AVC or the segmentation fault, but nothing can draw to screen either (just blank black windows show up).

With indirect, glxgears reports a framerate (just black window showing) of just 20 fps.

I cannot run compiz right now either (starting compiz crashes X), so this is happening with metacity running.  No xorg.conf is being used.

00:00.0 Host bridge: Intel Corporation Mobile 945GM/PM/GMS, 943/940GML and 945GT Express Memory Controller Hub (rev 03)
00:02.0 VGA compatible controller: Intel Corporation Mobile 945GM/GMS, 943/940GML Express Integrated Graphics Controller (rev 03)
00:02.1 Display controller: Intel Corporation Mobile 945GM/GMS/GME, 943/940GML Express Integrated Graphics Controller (rev 03)

glx-utils-7.2-0.1.fc10.i386
mesa-libGLU-7.2-0.1.fc10.i386
mesa-dri-drivers-7.2-0.1.fc10.i386
xorg-x11-server-Xephyr-1.5.0-6.fc10.i386
xorg-x11-server-utils-7.4-3.fc10.i386
xorg-x11-drv-i810-2.4.2-8.fc10.i386
xorg-x11-server-common-1.5.0-6.fc10.i386
mesa-libGL-7.2-0.1.fc10.i386
xorg-x11-server-Xorg-1.5.0-6.fc10.i386

This is a Macbook 2,1 (blackbook) 2.16G c2 duo machine.  F9 was installed up until the 12th and running compiz and glx very nicely.  I've tried booting with nomodeset, but I think thats still turned off in the kernel anyway?
Comment 8 Andrew Farris 2008-09-15 00:45:37 UTC 
Created attachment 316701 [details]
Xorg.0.log

Startup log without config file.
Comment 9 Tom London 2008-09-15 00:58:59 UTC 
Created attachment 316702 [details]
/var/log/Xorg.0.log with X, gdm and compiz running

My current status:

Running latest rawhide: 

[root@tlondon Download]# rpm -qa mesa\* xorg-x11-drv-i810 glx\*
mesa-libGL-devel-7.2-0.2.fc10.i386
glx-utils-7.2-0.2.fc10.i386
mesa-dri-drivers-7.2-0.2.fc10.i386
xorg-x11-drv-i810-2.4.2-8.fc10.i386
mesa-libGLU-devel-7.2-0.2.fc10.i386
mesa-libGLU-7.2-0.2.fc10.i386
mesa-libGL-7.2-0.2.fc10.i386
[root@tlondon Download]# 

I can run compiz only by running it with LIBGL_ALWAYS_INDIRECT=1 as:

alias fusion-icon="LIBGL_ALWAYS_INDIRECT=1 /usr/bin/fusion-icon&"

However, compiz-type functions (like cube rotate, screen refresh, window moves) are very slow (it can take 3-5 seconds for the cube to rotate).

Running glxgears shows slow and highly erratic performance, and while running the cursor vanishes.  Only way to kill glxgears is by entering ESC:

[tbl@tlondon ~]$ runcon -t unconfined_execmem_t glxgears
23 frames in 5.9 seconds =  3.881 FPS
10 frames in 6.9 seconds =  1.449 FPS
18 frames in 5.9 seconds =  3.051 FPS
12 frames in 6.6 seconds =  1.806 FPS
9 frames in 8.2 seconds =  1.099 FPS
118 frames in 6.4 seconds = 18.317 FPS
10 frames in 6.0 seconds =  1.665 FPS
10 frames in 6.0 seconds =  1.670 FPS
5 frames in 6.0 seconds =  0.832 FPS
5 frames in 5.3 seconds =  0.944 FPS
[tbl@tlondon ~]$ 

Again, no xorg.conf; Thinkpad X60 (intel 945 graphics).
Comment 10 Tom London 2008-09-15 01:08:38 UTC 
Decided to rerun the above with compiz off, so I logged off and back in again, this time, just with metacity.

Running the same command as above, I get a segfault the first time.  Works the second time, but still badly slow:

[tbl@tlondon ~]$ runcon -t unconfined_execmem_t glxgears
Segmentation fault
[tbl@tlondon ~]$ runcon -t unconfined_execmem_t glxgears
22 frames in 6.3 seconds =  3.492 FPS
22 frames in 8.4 seconds =  2.612 FPS
7 frames in 5.0 seconds =  1.399 FPS
14 frames in 5.0 seconds =  2.799 FPS
[tbl@tlondon ~]$ 

Here is what /var/log/messages says about the segfault:
Sep 14 18:03:40 tlondon kernel: glxgears[9135]: segfault at c ip 07f01949 sp bfef6030 error 4 in libdrm.so.2.3.0[7efa000+f000]

I'll try to run with gdb and get a better stack trace.
Comment 11 Tom London 2008-09-15 01:25:09 UTC 
Created attachment 316704 [details]
output of 'runcon -t unconfined_execmem_t valgrind glxgears'

I decided to try running glxgears with valgrind.

I run this command: "runcon -t unconfined_execmem_t valgrind glxgears"

Surprisingly, this runs much faster, smoother, better than without valgrind:
375 frames in 5.0 seconds = 74.829 FPS
424 frames in 5.0 seconds = 84.703 FPS
445 frames in 5.0 seconds = 88.955 FPS

Valgrind complains about lots of errors:
==10330== ERROR SUMMARY: 5927052 errors from 108 contexts (suppressed: 51 from 1)

Not sure if any of this is compelling....

I'm having some difficulty downloading the debuginfo packages.  When I get this done, I'll try to rerun....
Comment 12 Tom London 2008-09-15 02:11:46 UTC 
Created attachment 316705 [details]
run of 'runcon -t unconfined_execmem_t valgrind glxgears' with debuginfo packages

OK.  Managed to download debuginfo packages and reran 'runcon -t unconfined_execmem_t valgrind glxgears'

Again, performance is much improved:
376 frames in 5.0 seconds = 75.067 FPS
429 frames in 5.0 seconds = 85.737 FPS
442 frames in 5.0 seconds = 88.258 FPS
440 frames in 5.0 seconds = 87.995 FPS
442 frames in 5.0 seconds = 88.252 FPS
510 frames in 5.0 seconds = 101.999 FPS

This run says "==12394== ERROR SUMMARY: 10000000 errors from 107 contexts (suppressed: 39 from 1)"

Could this be related to: https://bugzilla.redhat.com/show_bug.cgi?id=461646  ?
Comment 13 Tom London 2008-09-17 18:31:08 UTC 
Admit I'm confused....

Travelled for the last 2 days, and today I find:

[tbl@tlondon ~]$ runcon -t unconfined_execmem_t glxgears
1572 frames in 5.0 seconds = 314.217 FPS
1642 frames in 5.0 seconds = 328.323 FPS
1633 frames in 5.0 seconds = 326.408 FPS
1644 frames in 5.0 seconds = 328.628 FPS
[tbl@tlondon ~]$ 

Still get the execmem AVC, but performance is snappy again.

[tbl@tlondon ~]$ rpm -qa mesa\* glx\* gdm\* xorg-x11-server\* libdri\*
gdm-2.23.92-7.fc10.i386
xorg-x11-server-Xorg-1.5.0-6.fc10.i386
mesa-libGL-devel-7.2-0.2.fc10.i386
gdm-user-switch-applet-2.23.92-7.fc10.i386
glx-utils-7.2-0.2.fc10.i386
mesa-dri-drivers-7.2-0.2.fc10.i386
xorg-x11-server-utils-7.4-3.fc10.i386
mesa-libGLU-devel-7.2-0.2.fc10.i386
mesa-libGLU-7.2-0.2.fc10.i386
mesa-debuginfo-7.2-0.2.fc10.i386
xorg-x11-server-common-1.5.0-6.fc10.i386
mesa-libGL-7.2-0.2.fc10.i386
[tbl@tlondon ~]$ 

Ghost?  Gone? (except for the AVC)
Comment 14 Andrew Farris 2008-09-18 20:56:07 UTC 
I see roughly the same performance numbers you're seeing here, when running with changed context, but glxgears is still producing only a blank black window (it claims it is rendering frames around 300fps).  I still cannot run compiz without crashing X, still see glxgears segfault without changing the context.  The AVC is still there though.
Comment 15 Andrew Farris 2008-09-18 21:10:00 UTC 
$-> runcon -t unconfined_execmem_t 'glxgears'
1940 frames in 5.0 seconds = 387.876 FPS
1931 frames in 5.0 seconds = 386.080 FPS
1933 frames in 5.0 seconds = 386.586 FPS
1933 frames in 5.0 seconds = 386.484 FPS
7159 frames in 5.0 seconds = 1431.458 FPS
1832 frames in 5.0 seconds = 366.385 FPS
1906 frames in 5.0 seconds = 381.009 FPS
8621 frames in 5.0 seconds = 1724.160 FPS
9393 frames in 5.0 seconds = 1877.216 FPS
9179 frames in 5.0 seconds = 1835.724 FPS
3499 frames in 5.0 seconds = 699.708 FPS
1915 frames in 5.0 seconds = 382.806 FPS

The faster sections are where I've hidden the black glxgears window behind a terminal, and it clearly effects it.  Thats pretty normal afaik, but the fact that its just a solid black output and still being effected by whether the window is visible may be useful (its actually drawing?).
Comment 16 Tom London 2008-10-15 19:03:57 UTC 
I did a "clean install"of Spin1 on a Thinkpad X61 (newer than the X60!), fully updated it, and I am not seeing this problem there:

mesa-libGL-7.2-0.8.fc10.x86_64
gdm-user-switch-applet-2.24.0-9.fc10.x86_64
glx-utils-7.2-0.8.fc10.x86_64
xorg-x11-server-common-1.5.2-4.fc10.x86_64
mesa-libGLU-7.2-0.8.fc10.x86_64
xorg-x11-server-Xorg-1.5.2-4.fc10.x86_64
gdm-2.24.0-9.fc10.x86_64
xorg-x11-server-utils-7.4-3.fc10.x86_64
mesa-dri-drivers-7.2-0.8.fc10.x86_64

selinux-policy-3.5.12-1.fc10.noarch
selinux-policy-targeted-3.5.12-1.fc10.noarch

Also, starting compiz via "fusion-icon" no longer seems to need the "LIBGL_ALWAYS_INDIRECT=1" passed in the environment.

[Thinkpad X61 has Intel 965 vs. Intel 945 on X60.]

[tbl@tlondon incomplete]$ glxgears
1231 frames in 5.0 seconds = 246.032 FPS
1257 frames in 5.0 seconds = 251.242 FPS
1269 frames in 5.0 seconds = 253.637 FPS
1268 frames in 5.0 seconds = 253.586 FPS
[tbl@tlondon incomplete]$
Comment 17 Dominik 'Rathann' Mierzejewski 2008-10-19 12:49:49 UTC 
I can confirm the execmem selinux denial on 32bit x86 (Intel Atom).

glx-utils-7.2-0.8.fc10.i386
selinux-policy-targeted-3.5.13-1.fc10.noarch
Comment 18 Tomasz Torcz 2008-10-21 11:51:06 UTC 
*** Bug 463078 has been marked as a duplicate of this bug. ***
Comment 19 Tom London 2008-11-19 22:40:13 UTC 
This has been working for quite some time for me.

Close?
Comment 20 Bug Zapper 2008-11-26 02:23:32 UTC 
This bug appears to have been reported against 'rawhide' during the Fedora 10 development cycle.
Changing version to '10'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 21 Tom London 2008-12-30 03:39:55 UTC 
Believe this has been "working" for some time.

Closing.