The apr_socket_send() function in the UNIX version of sendrecv.c does a write() loop on the socket. If the write fails with EAGAIN, it does an apr_wait_for_io_or_timeout() and tries the write loop once more. If that write also fails with EAGAIN, it returns to the caller, instead of continuing to try. The result is that an HTTP GET may abort prematurely, and only return the first 64K of a file.
Moving to the APR project.
I believe that apr_socket_send is documented to return EAGAIN and that the caller is responsible for calling it again. An alternative would be to add a apr_socket_sendfull() that would handle EAGAIN and keep trying to send in all cases.
Per dev@httpd discussion, that is by design; if select/poll indicates writability and the subsequence write() call fails with EAGAIN there is some other problem. Paul mentioned it may be a Darwin issue.... (I'd also check it's not another issue with the O_NONBLOCK-inheritance-over-accept-detection too)
To help explore the possibility of a Darwin issue ... Current probes indicate that no code is waiting in select or poll, or that poll/select/etc are returning right away indicating that the socket is ready for writing. What mechanism does apache2 use to determine when a socket is writable?
It uses poll() if available or select() otherwise. I don't know which it is on Darwin. Jeff commented on dev@httpd: > Another thing to check is if apr_poll() is telling the I/O routine > that data is ready when in fact it is not. I recall some recent > complaints about APR using poll() on OS X 10.3, where poll() has some > negative attributes (I don't recall details). You need to answer some of these question before we can help you any more on this. Specifically have you checked whether this is an O_NONBLOCK-inheritance issue? What's the output from running: http://people.apache.org/~jorton/nonblock.c
The output is: bound to 0.0.0.0:53696
That means that APR has detected that O_NONBLOCK is inherited across accept(), which is expected for a BSDish platform. So, the other question: which is used by APR on Darwin, poll or select? Have you tried forcing the decision one way or the other?
It appears APR prefers poll() on Mac OS X 10.4 (note difference from Mac OS X 10.3, where bug 29985 was encountered). Summary of poll-related log messages configuring Apache 2.0.53 on Mac OS X 10.4 Server: checking for poll... yes checking poll.h usability... yes checking poll.h presence... yes checking for poll.h... yes checking sys/poll.h usability... yes checking sys/poll.h presence... yes checking for sys/poll.h... yes checking for POLLIN in poll.h sys/poll.h... yes I'm not certain this is the best way to force it not to use poll(), but after running ./configure I set the following in httpd-2.0.53/srclib/apr/include/arch/unix/apr_private.h: #define HAVE_POLL 0 #define HAVE_POLLIN 0 #define HAVE_POLL_H 0 Built this way, apache2 still behaves the same way: y:/tmp al$ curl -o 1.2m.html http://xx.apple.com:8080/1.2M.html % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 4 1280k 4 65536 0 0 4339 0 0:05:02 0:00:15 0:04:47 0 curl: (18) transfer closed with 1245184 bytes remaining to read
From a kernel developer: Select does not guarantee writability; merely that at the time that the select wakeup was issued (not the time the caller runs) the socket was writable. There are many reasons that the socket may become unwritable again (resource shortages, competing access), and the caller should simply accept this and re- select. This implies we aren't doing the right thing in APR.
>There are many reasons that the socket may become >unwritable again (resource shortages, competing access), Competing access isn't an issue here. There is one thread checking for writability, and no other threads will consume the writeability characteristic. It seems very odd that an obscure problem such as a resource shortage (lack of mbufs ?) is the explanation for what is going on, since since this problem is so easily recreatable for you. Is there any way to get a syscall trace of the httpd process during the failure scenario so we can see this explicitly?
I'm not sure that resource contention is in fact a problem; the point was that getting EWOULDBLOCK (EAGAIN) on write after poll() tells you that it's writeable is valid behavior and that we should probably deal with that case by continuing to retry rather than giving up and sending the errno up to the caller. In this case, httpd isn't aware that it's dealing with a non-blocking socket, so that it can get the EAGAIN seems like a bug in APR. I worked around this (in r160348 on httpd HEAD) by having httpd retry the write if it gets EAGAIN, but I think that the fix probably should move down to APR, since httpd shouldn't have to anticipate an EAGAIN from APR.
Created attachment 15135 [details] try again Attaching Al's patch to APR for reference. Rather than trying one, and then if failure, try one more time, just keep trying.
The above patch would have to be applied to the other functions as well, if it's appropriate. The question is whether apr_socket_send() and friends are supposed to yiled EAGAIN to the caller if the caller is doing blocking I/O.
1) using "#define HAVE_POLL 0" in apr_private.h will not force use of select; it's a defined-or-not constant, not a defined-to-1-or-0 constant. I'd really like to see conclusive results of using select() over poll() first. There are many independent reports that poll() is broken (not to mention unsupported) in Darwin, e.g. http://curl.haxx.se/mail/lib-2005-05/0122.html Preferable way to force use of select is: export ac_cv_func_poll=no ./configure ... (and check apr_private.h does *not* define HAVE_POLL at all after that) 2) echoing other comment: our kernel guys here agree that this kind of situation *can* occur in rare circumstance (e.g. memory pressure) etc with other kernels also. That does not seem to be what is happening here; but it seems APR should indeed cope with it in _recv() and _send() (and ...) at least. A loop would be better than a goto, of course!
OK, I configured Apache 2.0.54 as Joe suggested, and verified that the unix version of apr_private.h does not define HAVE_POLL. Built that way, Apache no longer encounters the problem on Mac OS X 10.4. I was able to download not only the 1.2M file but also a roughly 2.3G file without incident: [y:/tmp] albegley% curl -o 2.3G.html http://xx.apple.com:8080/2.3G.html % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 2465M 100 2465M 0 0 10.0M 0 0:04:04 0:04:04 --:--:-- 10.4M
OK, I checked in the fix to prevent use of poll() on Darwin too. Thanks for investigating this. http://svn.apache.org/viewcvs?rev=178386&view=rev
Joe, I fixed APR's usage of poll/select earlier yesterday (http://svn.apache.org/viewcvs.cgi? view=rev&rev=178340) which removes the need to disable poll() on Darwin. Anyway, it at least makes Al's problem go away. The POLLIN/POLLHUP issue mentioned in the curl list message doesn't seem to be related... The upshot being that disabling poll() may not be necessary.
For reference; see http://mail-archives.apache.org/mod_mbox/apr-dev/200505.mbox/%3c222760D2-EF04-4481-B731-C90D7123CA2D@wsanchez.net%3e for the conclusion.