Bug 1970168

Summary: ipa-trust-add fails with "not enough quota"
Product: [Fedora] Fedora Reporter: Robbie Harwood <rharwood>
Component: freeipaAssignee: IPA Maintainers <ipa-maint>
Status: CLOSED WORKSFORME QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 34CC: abokovoy, anoopcs, asn, contribs, frenaud, ftrivino, gdeschner, iboukris, ipa-maint, jarrpa, jcholast, jhrozek, jstephen, lmohanty, madam, mhjacks, pvoborni, rcritten, sbose, ssorce, twoerner
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1982211 1982212 (view as bug list) Environment:
Last Closed: 2021-10-29 16:49:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1982211, 1982212    

Description Robbie Harwood 2021-06-10 00:53:48 UTC
[root@ipaserver ~]# echo "vagrant" | ipa trust-add --type=ad ad.test --admin vagrant --password --two-way=true
ipa: ERROR: CIFS server communication error: code "3221225495", message "{Not Enough Quota} Not enough virtual memory or paging file quota is available to complete the specified operation." (both may be "None")
[root@ipaserver ~]# 

Injecting a stack print, the callsite is:

  File "/usr/share/ipa/wsgi.py", line 59, in application
    return api.Backend.wsgi_dispatch(environ, start_response)
  File "/usr/lib/python3.9/site-packages/ipaserver/rpcserver.py", line 296, in __call__
    return self.route(environ, start_response)
  File "/usr/lib/python3.9/site-packages/ipaserver/rpcserver.py", line 308, in route
    return app(environ, start_response)
  File "/usr/lib/python3.9/site-packages/ipaserver/rpcserver.py", line 917, in __call__
    response = super(jsonserver_session, self).__call__(environ, start_response)
  File "/usr/lib/python3.9/site-packages/ipaserver/rpcserver.py", line 517, in __call__
    response = super(jsonserver, self).__call__(environ, start_response)
  File "/usr/lib/python3.9/site-packages/ipaserver/rpcserver.py", line 473, in __call__
    response = self.wsgi_execute(environ)
  File "/usr/lib/python3.9/site-packages/ipaserver/rpcserver.py", line 400, in wsgi_execute
    result = command(*args, **options)
  File "/usr/lib/python3.9/site-packages/ipalib/frontend.py", line 471, in __call__
    return self.__do_call(*args, **options)
  File "/usr/lib/python3.9/site-packages/ipalib/frontend.py", line 499, in __do_call
    ret = self.run(*args, **options)
  File "/usr/lib/python3.9/site-packages/ipalib/frontend.py", line 821, in run
    return self.execute(*args, **options)
  File "/usr/lib/python3.9/site-packages/ipaserver/plugins/trust.py", line 759, in execute
    full_join = self.validate_options(*keys, **options)
  File "/usr/lib/python3.9/site-packages/ipaserver/plugins/trust.py", line 868, in validate_options
    self.trustinstance = ipaserver.dcerpc.TrustDomainJoins(self.api)
  File "/usr/lib/python3.9/site-packages/ipaserver/dcerpc.py", line 1742, in __init__
    self.__populate_local_domain()
  File "/usr/lib/python3.9/site-packages/ipaserver/dcerpc.py", line 1756, in __populate_local_domain
    ld.retrieve(FQDN)
  File "/usr/lib/python3.9/site-packages/ipaserver/dcerpc.py", line 992, in retrieve
    self.init_lsa_pipe(remote_host)
  File "/usr/lib/python3.9/site-packages/ipaserver/dcerpc.py", line 891, in init_lsa_pipe
    self._pipe = self.__gen_lsa_connection(binding)                                                                                                                                                                                                                                                                            
  File "/usr/lib/python3.9/site-packages/ipaserver/dcerpc.py", line 870, in __gen_lsa_connection
    raise assess_dcerpc_error(e)
  File "/usr/lib/python3.9/site-packages/ipaserver/dcerpc.py", line 179, in assess_dcerpc_error
    traceback.print_stack(file=f)

I've previously captured an strace, but it doesn't seem helpful: https://rharwood.fedorapeople.org/strace

I observe no network traffic on any interface, other than: DNS, STP, SSH, LDAP, and HTTPS (on 443).

Tarball of debug logs with debug level 50: https://rharwood.fedorapeople.org/var_log_samba.tar.gz

Should you reproduce this locally, I have a repo: https://github.com/frozencemetery/ad-testing/ (vagrant up ipaserver, vagrant ssh ipaserver, sudo ./install_trust.sh).

I'm personally at a loss as to how to debug this further.  I don't seem to be able to find where the lsa stuff comes from, and no matter which smbd I attach to with gdb (even all of them), I can't seem to catch the connection.  Is it possible that there is no connection, somehow?

Comment 1 Andreas Schneider 2021-07-14 10:03:57 UTC
Error loading module '/usr/lib64/samba/idmap/sss.so': /usr/lib64/samba/idmap/sss.so: cannot open shared object file: No such file or directory

looks like idmap_sss is not installed and it can't allocate IDs ...

Comment 2 Alexander Bokovoy 2021-07-14 10:05:02 UTC
sssd-winbind-idmap is missing in IPA dependencies, it looks like.

Comment 3 Florence Blanc-Renaud 2021-07-15 14:29:31 UTC
Fixed upstream
master:
https://pagure.io/freeipa/c/24afb10c30577be6092ab699fd7f6eeef9fa62b2

Comment 4 Florence Blanc-Renaud 2021-07-15 16:23:10 UTC
Fixed upstream
ipa-4-9:
https://pagure.io/freeipa/c/1a4f459b81bc77cdf233b65f41d0f76dbb5f2fce

Comment 5 Robbie Harwood 2021-07-15 18:11:05 UTC
I appreciate you looking into this issue.  However, adding the package doesn't fix the issue for me:

[root@ipaserver vagrant]# rpm -qv sssd-winbind-idmap
sssd-winbind-idmap-2.5.2-1.fc34.x86_64
[root@ipaserver vagrant]# kinit admin
Password for admin: 
[root@ipaserver vagrant]# ipa trust-add --type=ad ad.test --admin vagrant --password
Active Directory domain administrator's password: 
ipa: ERROR: CIFS server communication error: code "3221225495", message "{Not Enough Quota} Not enough virtual memory or paging file quota is available to complete the specified operation." (both may be "None")
[root@ipaserver vagrant]# 

Samba logs at level 50: https://rharwood.fedorapeople.org/samba2.tar.gz

I see some things in log.winbindd-idmap - looks like there was trouble allocating a GID, maybe?

Comment 6 Alexander Bokovoy 2021-07-15 18:41:55 UTC
It should not allocate any GID, that's intentional, sss idmap module is read-only.
I'll look at the logs tomorrow.

Comment 7 Robbie Harwood 2021-08-03 19:32:49 UTC
Andreas pointed out that my logs aren't actually at the right level, so here's another attempt: http://rharwood.fedorapeople.org/samba3.tar.gz

Comment 8 Andreas Schneider 2021-08-11 15:41:33 UTC
Can someone from IPA Team please look into this. I dunno where this error is coming from.

When I looked into logs with Robbie we at least that winbind is unable to allocate gids, however this is how Samba is set up. The smb.conf (net conf list) has

idmap config * : range = 0-0

so winbind wont allocate any gid.

Comment 9 Rob Crittenden 2021-08-11 19:24:52 UTC
This configuration appears to be by design: https://github.com/freeipa/freeipa/blob/master/doc/designs/adtrust/samba-domain-member.md

Comment 10 Florence Blanc-Renaud 2021-08-12 14:55:25 UTC
Robbie,
in the config steps found in https://github.com/frozencemetery/ad-testing/ I don't see the forwarders config.
You can try to add the following steps:
- on ipa server: ipa dnsforwardzone-add ad.test --forwarder 192.168.3.2 --forward-policy only
- on ipa server: also add the ipaserver address in /etc/hosts (192.168.3.3 ipaserver.ipa.test) to force using this address (the vagrant image contains 2 IP addresses and I'm afraid this may confuse the system).

Comment 11 Robbie Harwood 2021-08-12 20:35:47 UTC
Thanks for taking a look.  I've made the suggested changes (repo updated accordingly).

The error is different now:

# echo vagrant | ipa trust-add --type=ad ad.test --admin vagrant --password
ipa: ERROR: Cannot find specified domain or server name
# 

I'm unsure what it's trying to resolve - the system can resolve adserver.ad.test just fine (it pings).  Happy to provide more logs but I'm not sure what would be useful: there doesn't appear to be anything new in /var/log and an strace of the `ipa trust-add` didn't show any resolution failures.

Comment 12 Florence Blanc-Renaud 2021-08-13 10:26:21 UTC
Hi Robbie,
in my env it's working if I modify the ipa-server-install command and add --ip-address 192.168.3.3 (in order to force binding only to this address).

Comment 13 Florence Blanc-Renaud 2021-09-28 09:18:21 UTC
@rharwood 
Did you manage to solve the issue with my proposal from #c12?

Comment 14 Robbie Harwood 2021-10-29 16:49:28 UTC
Your proposal allowed it to get further, thanks.