Bug 52402

Summary: balancer setup crashes apache during start
Product: Apache httpd-2 Reporter: Mario <JBlond>
Component: mod_proxy_balancerAssignee: Apache HTTPD Bugs Mailing List <bugs>
Status: CLOSED FIXED    
Severity: blocker CC: colinpauladams, gls, micha, paulzhou2004, s9flschr, zl
Priority: P1 Keywords: APIBug, ErrorMessage, FixedInTrunk, PatchAvailable
Version: 2.4.1   
Target Milestone: ---   
Hardware: PC   
OS: All   
Attachments: proposed patch for httpd-2.4.2

Description Mario 2011-12-30 20:16:47 UTC
Apache crashes during startup. httpd -S fails /shows nothing, also httpd -M fails shows nothing. apache installed as service. version 2.3.16 beta (VC9 x64)
OS: win7 ultimate x64

error log:
Starting the Apache2.4 service
The Apache2.4 service is running.
[proxy:debug] [pid 3632:tid 428] mod_proxy.c(1932): AH01147: Defining worker 'http://servera:80' for balancer 'balancer://mycluster'
[Fri Dec 30 20:55:42.972797 2011] [proxy:debug] [pid 3632:tid 428] mod_proxy.c(1937): AH01148: Defined worker 'http://servera' for balancer 'balancer://mycluster'
[Fri Dec 30 20:55:42.972797 2011] [proxy:debug] [pid 3632:tid 428] mod_proxy.c(1932): AH01147: Defining worker 'http://serverb:80' for balancer 'balancer://mycluster'
[Fri Dec 30 20:55:42.972797 2011] [proxy:debug] [pid 3632:tid 428] mod_proxy.c(1937): AH01148: Defined worker 'http://serverb' for balancer 'balancer://mycluster'
[Fri Dec 30 20:55:42.972797 2011] [proxy:debug] [pid 3632:tid 428] mod_proxy.c(1932): AH01147: Defining worker 'http://serverc:80' for balancer 'balancer://mycluster'
[Fri Dec 30 20:55:42.972797 2011] [proxy:debug] [pid 3632:tid 428] mod_proxy.c(1937): AH01148: Defined worker 'http://serverc' for balancer 'balancer://mycluster'
[Fri Dec 30 20:55:42.972797 2011] [proxy:debug] [pid 3632:tid 428] mod_proxy.c(1932): AH01147: Defining worker 'http://serverd:80' for balancer 'balancer://mycluster'
[Fri Dec 30 20:55:42.972797 2011] [proxy:debug] [pid 3632:tid 428] mod_proxy.c(1937): AH01148: Defined worker 'http://serverd' for balancer 'balancer://mycluster'
[Fri Dec 30 20:55:42.987797 2011] [proxy_balancer:debug] [pid 3632:tid 428] mod_proxy_balancer.c(742): AH01178: Doing balancers create: 544, 1 (6)
[Fri Dec 30 20:55:42.988797 2011] [proxy_balancer:debug] [pid 3632:tid 428] mod_proxy_balancer.c(796): AH01184: Doing workers create: balancer://mycluster (s96f9938e_mycluster), 512, 4
[Fri Dec 30 20:55:43.003798 2011] [proxy_balancer:debug] [pid 3632:tid 428] mod_proxy_balancer.c(742): AH01178: Doing balancers create: 544, 1 (6)
[Fri Dec 30 20:55:43.003798 2011] [proxy_balancer:debug] [pid 3632:tid 428] mod_proxy_balancer.c(796): AH01184: Doing workers create: balancer://mycluster (s96f9938e_mycluster), 512, 4
[Fri Dec 30 20:55:43.004798 2011] [proxy_balancer:emerg] [pid 3632:tid 428] (22)Invalid argument: AH01186: worker slotmem_grab failed
AH00020: Configuration Failed


loaded modules:
mod_alias
mod_auth_basic
mod_authn_core
mod_authn_file
mod_authz_core
mod_authz_groupfile
mod_authz_host
mod_authz_user
mod_autoindex
mod_cgi
mod_deflate
mod_dir
mod_env
mod_filter
mod_headers
mod_include
mod_lbmethod_bybusyness
mod_lbmethod_byrequests
mod_lbmethod_bytraffic
mod_log_config
mod_mime
mod_negotiation
mod_proxy
mod_proxy_balancer
mod_proxy_express
mod_proxy_http
mod_setenvif
mod_slotmem_plain
mod_slotmem_shm
mod_socache_memcache
mod_socache_shmcb
mod_vhost_alias

test setup

<VirtualHost *:80>
    ServerAdmin info@localhost
    ServerName localhost
    ServerAlias servera serverb serverc serverd
    DocumentRoot "/Apache23/htdocs"
</VirtualHost>



<Proxy balancer://mycluster>
	BalancerMember http://servera:80
	BalancerMember http://serverb:80
	BalancerMember http://serverc:80
	BalancerMember http://serverd:80
</Proxy>
	ProxyPass / balancer://mycluster
	ProxyPassReverse / balancer://mycluster 

<Location /balancer-manager>
	SetHandler balancer-manager
</Location> 


see also http://old.nabble.com/Notice%3A-Intent-to-T-R-httpd-2.3.15-this-week.-td32590965.html
Comment 1 Mario 2012-01-01 18:00:30 UTC
Commenting out all BalancerMember http://... let apache run again, and I can see than the Load Balancer Manager page. Clicking on the balancer://mycluster link (http://localhost/balancer-manager?b=mycluster) crashes the connection, but apache keeps running.

in the windows event log
Protokollname: Application
Quelle:        Application Error
Datum:         01.01.2012 18:58:15
Ereignis-ID:   1000
Aufgabenkategorie:(100)
Ebene:         Fehler
Schlüsselwörter:Klassisch
Benutzer:      Nicht zutreffend
Computer:      zengarden
Beschreibung:
Name der fehlerhaften Anwendung: httpd.exe, Version: 2.3.16.0, Zeitstempel: 0x4eebe6cb
Name des fehlerhaften Moduls: mod_proxy_balancer.so, Version: 2.3.16.0, Zeitstempel: 0x4eebe6a6
Ausnahmecode: 0xc0000005
Fehleroffset: 0x0000000000002db0
ID des fehlerhaften Prozesses: 0x86c
Startzeit der fehlerhaften Anwendung: 0x01ccc8aee7051bb9
Pfad der fehlerhaften Anwendung: C:\Apache23\bin\httpd.exe
Pfad des fehlerhaften Moduls: C:\Apache23\modules\mod_proxy_balancer.so
Berichtskennung: 2d7508eb-34a2-11e1-90ed-f0def11c30f7
Ereignis-XML:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Level>2</Level>
    <Task>100</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2012-01-01T17:58:15.000000000Z" />
    <EventRecordID>1259</EventRecordID>
    <Channel>Application</Channel>
    <Computer>zengarden</Computer>
    <Security />
  </System>
  <EventData>
    <Data>httpd.exe</Data>
    <Data>2.3.16.0</Data>
    <Data>4eebe6cb</Data>
    <Data>mod_proxy_balancer.so</Data>
    <Data>2.3.16.0</Data>
    <Data>4eebe6a6</Data>
    <Data>c0000005</Data>
    <Data>0000000000002db0</Data>
    <Data>86c</Data>
    <Data>01ccc8aee7051bb9</Data>
    <Data>C:\Apache23\bin\httpd.exe</Data>
    <Data>C:\Apache23\modules\mod_proxy_balancer.so</Data>
    <Data>2d7508eb-34a2-11e1-90ed-f0def11c30f7</Data>
  </EventData>
</Event>
Comment 2 Graham Leggett 2012-01-01 18:34:59 UTC
This isn't a crash, but rather a configuration error:

[Fri Dec 30 20:55:43.004798 2011] [proxy_balancer:emerg] [pid 3632:tid 428]
(22)Invalid argument: AH01186: worker slotmem_grab failed
AH00020: Configuration Failed
Comment 3 Mladen Turk 2012-01-02 08:16:18 UTC
Hmm, I remember I once wrote experimental module that used provider logic and it crashed on windows. Think this has to do something with dso logic and which pool has been used. Let me see if I can find that code and see how I fixed that. Might be this is the same cause.
Comment 4 Mario 2012-01-17 18:53:17 UTC
Just saying it is a crash. The pid file still exist. Also there is a popup from windows telling me there is a crash and there is the event log entry.
Comment 5 Mladen Turk 2012-02-03 08:47:03 UTC
OK, so the reason is mod_slotmem_shm

I added some debugging and the reason is that win32 calls config twice.
That cause mod_slotmem_shm to try to allocate shared storage twice, and
the second avocation fails because it already reached limit in first call.
Comment 6 Mladen Turk 2012-02-03 14:17:13 UTC
Fixed both in the trunk and 2.4.x
Could you check if r1240170 fixes the issue on your side?
Comment 7 Mario 2012-02-07 22:15:55 UTC
Now lbmethod_bybusyness_module nor lbmethod_byrequests_module nor lbmethod_bytraffic_module. No issue while building.
Without that modules I get

C:\Apache2x\bin>httpd -S
AH00526: Syntax error on line 209 of C:/Apache2x/conf/httpd.conf:
BalancerMember Can't find 'byrequests' lb method

Than I tried to load each module single, but they don't load at all.

C:\Apache2x\bin>httpd -S
httpd: Syntax error on line 121 of C:/Apache2x/conf/httpd.conf: Cannot load C:/Apache2x/mod_lbmethod_bybusyness.so into server: Das angegebene Modul wurde
nicht gefunden.


C:\Apache2x\bin>httpd -S
httpd: Syntax error on line 122 of C:/Apache2x/conf/httpd.conf: Cannot load C:/Apache2x/mod_lbmethod_byrequests.so into server: Das angegebene Modul wurde
nicht gefunden.


C:\Apache2x\bin>httpd -S
httpd: Syntax error on line 123 of C:/Apache2x/conf/httpd.conf: Cannot load C:/Apache2x/mod_lbmethod_bytraffic.so into server: Das angegebene Modul wurde n
icht gefunden.



Positive I can say that the module itself now load without apache crashing
Comment 8 Rainer Jung 2012-02-12 18:11:02 UTC
(In reply to comment #7)
> Now lbmethod_bybusyness_module nor lbmethod_byrequests_module nor
> lbmethod_bytraffic_module. No issue while building.
> Without that modules I get
> 
> C:\Apache2x\bin>httpd -S
> AH00526: Syntax error on line 209 of C:/Apache2x/conf/httpd.conf:
> BalancerMember Can't find 'byrequests' lb method

It is expected, that using method "byrequests" without having lbmethod_byrequests_module does not work.

> Than I tried to load each module single, but they don't load at all.
> 
> C:\Apache2x\bin>httpd -S
> httpd: Syntax error on line 121 of C:/Apache2x/conf/httpd.conf: Cannot load
> C:/Apache2x/mod_lbmethod_bybusyness.so into server: Das angegebene Modul wurde
> nicht gefunden.

> C:\Apache2x\bin>httpd -S
> httpd: Syntax error on line 122 of C:/Apache2x/conf/httpd.conf: Cannot load
> C:/Apache2x/mod_lbmethod_byrequests.so into server: Das angegebene Modul wurde
> nicht gefunden.
> 
> C:\Apache2x\bin>httpd -S
> httpd: Syntax error on line 123 of C:/Apache2x/conf/httpd.conf: Cannot load
> C:/Apache2x/mod_lbmethod_bytraffic.so into server: Das angegebene Modul wurde n
> icht gefunden.

Fpr non-Germans "The given module has not been found".

Mario: did you notice the strange path? Modules usually go to a "modules" directory. It seems your httpd.conf doesn't have the LoadModule lines correct. Can you please double check, how the LoadModule lines for those modules look like, and whether the installed modules files actually are in the correct directories?

Regards,

Rainer
Comment 9 Gregg L. Smith 2012-02-13 03:26:50 UTC
I'm to blame on this. I see the problem in installwinconf.awk. 

Fixed Trunk: r1243404
2.4: r1243405
Comment 10 Mario 2012-02-13 10:32:31 UTC
Works on non localhost. Thanks for the hint with the paths and the fix svn.
With external servers works fine now. 

With the original setup on only localhost I have now

--- snip ---
Bad Request

Your browser sent a request that this server could not understand.
Size of a request header field exceeds server limit.

X-Forwarded-Server
--- /snip ---
Comment 11 Mario 2012-02-13 10:40:16 UTC
I changed 

LimitRequestFields      0
LimitRequestFieldsize	12392

But than I get 
[Mon Feb 13 11:38:37.294511 2012] [proxy_balancer:emerg] [pid 9324:tid 376] (22)Invalid argument: AH01186: worker slotmem_grab failed
[Mon Feb 13 11:38:37.295511 2012] [:emerg] [pid 9324:tid 376] AH00020: Configuration Failed, exiting
[Mon Feb 13 11:38:37.308512 2012] [mpm_winnt:crit] [pid 5980:tid 528] AH00419: master_main: create child process failed. Exiting.
[Mon Feb 13 11:38:38.202563 2012] [mpm_winnt:notice] [pid 9712:tid 452] AH00364: Child: All worker threads have exited.
Comment 12 Mario 2012-02-27 21:25:59 UTC
With apache 2.4.1 it works if I change the balancer root to /test

e.g.
ProxyPass /test/ balancer://mycluster
ProxyPassReverse /test/ balancer://mycluster

Having the the old way 
ProxyPass / balancer://mycluster
ProxyPassReverse / balancer://mycluster

I get a 502. Any idea how to fix this?
Comment 13 msubbareddy 2012-03-06 16:26:45 UTC
Unable to start the apache webserver (httpd). Getting terminated in between.
How to start the webserver?

Linux: Ubuntu 11.10 i386 x32 bit
Apache2: 2.4.1

Apache2 source got compiled and installed, when try to start, failed due to:

[proxy_balancer:emerg] [pid 2325] AH01177: Failed to lookup provider 'shm' for 'slotmem': is mod_slotmem_shm loaded??
[:emerg] [pid 2325] AH00020: Configuration Failed, exiting

Results on running: 
./httpd -S

VirtualHost configuration:
ServerRoot: "/usr/local/apache-2.4.1"
Main DocumentRoot: "/usr/local/apache-2.4.1/htdocs"
Main ErrorLog: "/usr/local/apache-2.4.1/logs/error_log"
Mutex default: dir="/usr/local/apache-2.4.1/logs" mechanism=default
Mutex mpm-accept: using_defaults
Mutex proxy-balancer-shm: using_defaults
PidFile: "/usr/local/apache-2.4.1/logs/httpd.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
User: name="daemon" id=1
Group: name="daemon" id=1

error_log:
[proxy_balancer:emerg] [pid 2705] AH01177: Failed to lookup provider 'shm' for 'slotmem': is mod_slotmem_shm loaded??
[:emerg] [pid 2705] AH00020: Configuration Failed, exiting
Comment 14 msubbareddy 2012-03-06 22:18:54 UTC
Server is runnable, upon compiling without --enable-proxy --enable-proxy-balancer --enable-proxy-http options.
Comment 15 msubbareddy 2012-03-07 02:03:10 UTC
Got it, this is no issue with proxy enabled option.
Its basically required to enable the corresponding required module!

[proxy_balancer:emerg] [pid 29584] AH01177: Failed to lookup provider 'shm' for 'slotmem': is mod_slotmem_shm loaded??

by uncommenting the below entry, server has ran without any issue by using 2.4.1 without any additional patches.

LoadModule slotmem_shm_module modules/mod_slotmem_shm.so
Comment 16 Eric Covener 2012-03-07 02:33:59 UTC
msubbareddy: don't hijack,much less close, existing bug reports opened by other users.
Comment 17 Florian S. 2012-03-09 12:22:08 UTC
It looks like each balancer creates the SHM for its number of workers (1
in my case) using its unique ->sname. After that, each worker grabs a
slot from this.

I have several virtual hosts pointing to the same balancer (by rewrite
[P]). So, the second time, the balancer calls shm's slotmem_create again
with its name, that already exists -> kind-of noop.

Now, the SHM under the balancer's name has still only 1 slot (that is
already occupied).

When the worker now wants to (re-)grab its slot, I get:

[slotmem_shm:debug] mod_slotmem_shm.c(575): AH02293: slotmem(...) grab
failed. Num 1/num_free 0 [proxy_balancer:emerg] (22)Invalid argument:
AH01186: worker slotmem_grab failed

Does anybody know the intention behind this?
Comment 18 paulzhou 2012-03-20 01:49:57 UTC
I also encounter such issue. The virtual hosts configuration is as below:

<VirtualHost *:80>
   ServerAdmin webmaster@www.mydomain.sites
   ServerName www.mydomain.sites
   ProxyPass /google http://www.google.com/
   ErrorLog "logs/www.mydomain.sites-error_log"
   CustomLog "logs/www.mydomain.sites-access_log" common
</VirtualHost>
<Proxy balancer://testBalancer>
BalancerMember http://www.app03.v2/
BalancerMember http://www.app03.v22/
</Proxy>

Linux: Ubuntu 10.10 i686 x32 bit
Apache2: 2.4.1

By above configuration, it crash apache during start.
error log:
[proxy_balancer:emerg] [pid 11746:tid 3077961472] (22)Invalid argument: AH01186: worker slotmem_grab failed


Any help is appreciated.
Comment 19 Arun Janarthanan 2012-03-26 14:40:24 UTC
All,

I had similar error when the balancer included outside from Virtual host, for testing I did include the balancer file inside the virtual host and it did work and nolonger getting the error.



<VirtualHost abc.com:80>

Include conf/abc.com.conf

Include conf/balancer_ab.conf
Include conf/balancer_xyz.conf

</VirtualHost>
Comment 20 Micha Lenk 2012-05-22 13:44:52 UTC
This issue still isn't fixed.

What about applying the patch proposed on the mailinglist some time ago?
http://mail-archives.apache.org/mod_mbox/httpd-dev/201204.mbox/%3C1334306253.1695.50.camel%40florian-desktop%3E
Comment 21 Zisis Lianas 2012-06-13 15:53:22 UTC
I also stumbled across this issue today in httpd-2.4.2...

See http://www.mail-archive.com/dev%40httpd.apache.org/msg54912.html
Comment 22 Zisis Lianas 2012-06-13 16:38:06 UTC
Just tested the patch against my httpd-2.4.2 installation and it resolves the issue.

Patch:
http://mail-archives.apache.org/mod_mbox/httpd-dev/201204.mbox/%3C1334306253.1695.50.camel%40florian-desktop%3E


Maybe some commiter can have a look on this?
Comment 23 Zisis Lianas 2012-06-13 16:46:12 UTC
Created attachment 28934 [details]
proposed patch for httpd-2.4.2
Comment 24 Mario 2012-06-18 13:51:30 UTC
The patch works for me, too!
Comment 25 manny 2012-07-09 14:34:43 UTC
The nice folks over at Apache Lounge built the patched module for me ( http://www.apachelounge.com/viewtopic.php?t=4805 ) but the error persists:

[proxy_balancer:emerg] [pid 6496:tid 620] (22)Invalid argument: AH01186: worker slotmem_grab failed

Does anything obvious appear wrong with my configuration?

<VirtualHost _default_:9000> 

  ProxyTimeout 900 
  SSLProxyEngine On 
  ProxyPreserveHost On 

  <Proxy balancer://my_cluster> 
    BalancerMember https://paz:9000 route=nA 
    BalancerMember https://baz:9000 route=nB 
    ProxySet lbmethod=byrequests stickysession=FOO_STICKY nofailover=Off 
  </Proxy> 

  ProxyPass /foo/   balancer://my_cluster/foo/ 
  Header add Set-Cookie "FOO_STICKY=s.%{BALANCER_WORKER_ROUTE}e;path=/;secure" env=BALANCER_ROUTE_CHANGED 

</VirtualHost>
Comment 26 Micha Lenk 2012-10-10 16:44:28 UTC
This seems to be still present in Apache 2.4.3.
Is there any news on this issue?
What can I do to help resolving this issue?
Comment 27 Jim Jagielski 2012-11-01 14:43:13 UTC
Trunk handles this differently and that code path is projected to be backported. Can you check that trunk works for you?
Comment 28 Micha Lenk 2012-11-02 11:16:33 UTC
Jim, I haven't tested yet with trunk. But do you know which commit fixed the issue in trunk?
Comment 29 Zisis Lianas 2012-11-06 18:55:38 UTC
(In reply to comment #27)
> Trunk handles this differently and that code path is projected to be
> backported. Can you check that trunk works for you?

Jim, just tested httpd-trunk (r1406270) and I can confirm that the issue is fixed now.
Comment 30 Colin Adams 2012-12-20 16:48:38 UTC
I am getting just this problem (AH01186: worker slotmem_grab failed) with 2.4.3 on Windows XP (64-bit).

The last comment seems to say that there is a fix, but I infer that it will require compiling from source. Is that correct?
Comment 31 Zisis Lianas 2012-12-24 11:44:24 UTC
That's right - you can build from sources to get this fixed or wait for the 2.4.4 release, which will fix the issue, see:
http://svn.apache.org/viewvc/httpd/httpd/branches/2.4.x/CHANGES?revision=1422877&view=markup
Comment 32 Stefan Fritsch 2013-03-03 16:48:42 UTC
fixed in 2.4.4
Comment 33 Vincent Gerris 2017-05-05 09:29:10 UTC
Has this regressed somehow into 2.4.25 ?
I needed to change the config to have BalancerMember directives included within the VirtualHost.
When it is not done, on OSx I get 

[Thu May 04 16:54:49.918766 2017] [slotmem_shm:error] [pid 63687:tid 140736233759680] (24)Too many open files: AH02611: create: apr_shm_create(/somedir/logs/slotmem-shm-p3e3cb24c.shm) failed
[Thu May 04 16:54:49.918794 2017] [proxy_balancer:emerg] [pid 63687:tid 140736233759680] (24)Too many open files: AH01179: balancer slotmem_create failed