| Summary: | httpd crashes while loading ldap attributes | ||
|---|---|---|---|
| Product: | Apache httpd-2 | Reporter: | Bernd Leinfelder <bernd.asf> |
| Component: | mod_authz_ldap | Assignee: | Apache HTTPD Bugs Mailing List <bugs> |
| Status: | RESOLVED FIXED | ||
| Severity: | critical | CC: | covener |
| Priority: | P2 | Keywords: | PatchAvailable |
| Version: | 2.2.6 | ||
| Target Milestone: | --- | ||
| Hardware: | Other | ||
| OS: | AIX | ||
| Attachments: |
Patch - optimized string handling in util_ldap.c
Patch - optimized attribute handling |
||
With additional debugging I found something new: The problem is in apr_strings.c in function apr_pstrcat, when memory is requested from apr_palloc. The last call to apr_palloc before the coredump returns 0, so memcpy tries to copy to an invalid destination. apr_palloc returns NULL if and only if allocator_alloc return NULL in apr_pools.c. This is teh case if malloc fails. So the question is: What makes malloc fail on AIX? >What makes malloc fail on AIX?
1) unavoidable address space limitations (I think the max heap is 2GB with 32-bit processes)
2) ulimits
3) starting Apache with httpd instead of apachectl, which skips the LDR_CNTRL setting in envvars, and
you're left with a default address space layout which doesn't allow much heap
(LDR_CNTRL necessity applies only to 32-bit builds)
4) out of paging space
5) ???
Created attachment 21240 [details]
Patch - optimized string handling in util_ldap.c
Problem was caused by a problematic algorithm in util_ldap.c, where for each
attribue value apr_pstrcat was called. with each call new memory was allocated,
so that in the end this operation caused an allocation of 260 mb for storing
180 kb. This was optimized in uldap_cache_getuserdn.
Can you please provide your patch as a unified diff? Jeff, your are right. We started httpd directly and so the data segment was limited to 256 MB. See http://publib.boulder.ibm.com/infocenter/tivihelp/v2r1/index.jsp?topic=/com.ibm.itame.doc/am60_perftune64.htm With specifying the LDR_CNTRLs the process didn't crash anymore, but grew to a size of 760 mb during retrieving the ldap attributes. Thanks for this very helpful hint. Created attachment 21242 [details]
Patch - optimized attribute handling
Regardless of the process doesn't crash anymore with correct LDR-CNTRLs I still
think that this is bug in util_ldap.c
If you look at this piece of code:
- while (values && values[j]) {
- str = str ? apr_pstrcat(r->pool, str, "; ", values[j], NULL)
- : apr_pstrdup(r->pool, values[j]);
So with each iteration an new string is allocated with apr_pstrcat whereas the
old string is not returned to the pool.
In my case I want to retrieve 3'500 attribute values. So this loop has 3'500
iterations and 3'500 strings are allocated, each a bit larger than the one
before. This is exponential memory consumption.
In the end this loop consumes about 760 mb; in my patched version it takes only
about 800 kb. Still not optimal, but much better than before.
Patch now in unified diff format as requested.
|
We wrote an authorisation module for integrating apache in our company's security framework. This module is based on mod_authnz_ldap, and we can reproduce our problem with mod_authnz_ldap as well. The problem occurs when a large list of attribute-values is loaded from ldap. In this case, the account, that causes the crash, has an ldap attribute "slResolvedPermissions" with 3'500 values. These values are concatenated in modules/ldap/util_ldap.c:uldap_cache_getuserdn() to a string of approx. 200k size. During this the process receives a SegFault and coredumps after concatenating 2893 values and a string size of approx. 178'947 bytes. Backtrace: (gdb) bt #0 0x0000f050 in ?? () #1 0xd21ceddc in apr_pstrcat (a=0x20239f20, __ellip=<incomplete type>) at strings/apr_strings.c:165 #2 0x200c734c in uldap_cache_getuserdn (r=0x20239f58, ldc=0x200dd1b8, url=0x20242808 "ldap://sphinxps:392/ou=accounts,dc=swisslife,dc=ch?cn,slResolvedPermissions", basedn=0x20242868 "ou=accounts,dc=swisslife,dc=ch", scope=2, attrs=0x20242888, filter=0x2ff205f8 "(&(objectclass=*)(cn=IXED))", binddn=0x2ff225f8, retvals=0x2ff225fc) at util_ldap.c:1240 #3 0x20075378 in authz_ldap_check_user_access (r=0x20239f58) at mod_authnz_ldap.c:590 #4 0x1003bdec in ap_run_auth_checker (r=0x20239f58) at request.c:78 #5 0x1003e2c0 in ap_process_request_internal (r=0x20239f58) at request.c:202 #6 0x10043940 in ap_process_request (r=0x20239f58) at http_request.c:256 #7 0x1004c32c in ap_process_http_connection (c=0x2022dfd0) at http_core.c:184 #8 0x10035000 in ap_run_process_connection (c=0x2022dfd0) at connection.c:43 #9 0x10035a34 in ap_process_connection (c=0x2022dfd0, csd=0x2022def8) at connection.c:178 #10 0x1000bda0 in child_main (child_num_arg=0) at prefork.c:640 #11 0x1000b77c in make_child (s=0x20025780, slot=0) at prefork.c:680 #12 0x1000c930 in ap_mpm_run (_pconf=0x200238e0, plog=0x20051a50, s=0x20025780) at prefork.c:956 #13 0x10000e8c in main (argc=2, argv=0x2ff22b10) at main.c:730 Please not that line numbers in util_ldap.c are not accurate because of some debug statements. For reproducing this problem please set up an ldap server with an user as described above; cn=username and a huge list of attributes slResolvedPermissions. Configure a directory with the following .hraccess: AuthType basic AuthBasicProvider file AuthName "Secret Stuff" AuthUserFile /home/bele/apache/htdocs/spxtest/secret/htpasswd AuthLDAPURL ldap://sphinxps:392/ou=accounts,dc=company,dc=com?cn,slResolvedPermissions AuthLDAPBindDN "uid=Directory Reader,ou=Directory Users,dc=company,dc=com" AuthLDAPBindPassword XXXXXX Require ldap-attribute slResolvedPermissions=AV-K.BEST_AUSGABE.read I could not reproduce this problem on Solaris, so this is probably AIX specific. Compiler was xlc and alternatively gcc 4. Ldap library is openldap 2.3.39, httpd is 2.2.6.