This may not be related;

But after each writev() call, mmap() is called and throws INVAL. An strace of an active connection to a dummy index.html page: 

7006  mmap(NULL, 7950, PROT_READ, MAP_SHARED, 17, 0) = 0x2aab472b1000
7006  munmap(0x2aab472b1000, 7950)      = 0
7006  writev(13, [{iov_base="\27\3\3 \21\362|k\202\250&S\274\343\272\217\356r\375\322\233G\232Eu1\215B\236sc\321"..., iov_len=8214}], 1) = 8214
7006  mmap(NULL, 72624, PROT_READ, MAP_SHARED, 17, 0x1f0e) = -1 EINVAL (Invalid argument)
7006  lseek(17, 7950, SEEK_SET)         = 7950
7006  read(17, "ackground-dim.has-background-dim"..., 8000) = 8000
7006  writev(13, [{iov_base="\27\3\3\37Q\336\310}\v2b\335B\373\216F\266\200b\2\1\361\375\2608\214\213\3\10\355:n"..., iov_len=8022}], 1) = 8022
7006  mmap(NULL, 64624, PROT_READ, MAP_SHARED, 17, 0x3e4e) = -1 EINVAL (Invalid argument)
7006  lseek(17, 15950, SEEK_SET)        = 15950
7006  read(17, "argin-bottom:0;max-width:840px;p"..., 8000) = 8000
7006  writev(13, [{iov_base="\27\3\3\37Q\n\2076@\364\211\261OsS8\253\321&\201uK\203\5X\t\27\370\253\f\332M"..., iov_len=8022}], 1) = 8022
7006  mmap(NULL, 56624, PROT_READ, MAP_SHARED, 17, 0x5d8e) = -1 EINVAL (Invalid argument)

How large are the static files? Are they delivered directly by Apache?
File buckets below 256 byte of size are not sent via sendfile.

If would be helpful if you could break with gdb in can_sendfile_bucket and do

dump_bucket(b)


(requires the .gdbinit delivered with the source code to be loaded in gdb).

I've tried static files that range in size from 512 bytes to 16,000 bytes in size (html files)


(In reply to Ruediger Pluem from comment #2)
> How large are the static files? Are they delivered directly by Apache?
> File buckets below 256 byte of size are not sent via sendfile.
> 
> If would be helpful if you could break with gdb in can_sendfile_bucket and do
> 
> dump_bucket(b)
> 
> 
> (requires the .gdbinit delivered with the source code to be loaded in gdb).

Looks like b() is being optimized out, I guess I'll need to recompile everything without any opts at all;

Breakpoint 1, can_sendfile_bucket (b=<optimized out>) at core_filters.c:621
621	    if (APR_BUCKET_IS_FILE(b) && b->length >= AP_MIN_SENDFILE_BYTES) {
(gdb) dump_bucket(b)
value has been optimized out

I'll work on it tonight and get you the information.

(In reply to Ruediger Pluem from comment #2)
> How large are the static files? Are they delivered directly by Apache?
> File buckets below 256 byte of size are not sent via sendfile.
> 
> If would be helpful if you could break with gdb in can_sendfile_bucket and do
> 
> dump_bucket(b)
> 
> 
> (requires the .gdbinit delivered with the source code to be loaded in gdb).

Here's the requested information: 

Breakpoint 1, can_sendfile_bucket (b=0x558cded27068) at core_filters.c:621
621	    if (APR_BUCKET_IS_FILE(b) && b->length >= AP_MIN_SENDFILE_BYTES) {
(gdb) dump_bucket(b)
 bucket=HEAP     (558cded27068) length=1555   data=558cded26de8
     contents=[~~~~z~~~v~~~~~l~Z...] rc=1

(In reply to Gary S from comment #4)
> (In reply to Ruediger Pluem from comment #2)
> > How large are the static files? Are they delivered directly by Apache?
> > File buckets below 256 byte of size are not sent via sendfile.
> > 
> > If would be helpful if you could break with gdb in can_sendfile_bucket and do
> > 
> > dump_bucket(b)
> > 
> > 
> > (requires the .gdbinit delivered with the source code to be loaded in gdb).
> 
> Here's the requested information: 
> 
> Breakpoint 1, can_sendfile_bucket (b=0x558cded27068) at core_filters.c:621
> 621	    if (APR_BUCKET_IS_FILE(b) && b->length >= AP_MIN_SENDFILE_BYTES) {
> (gdb) dump_bucket(b)
>  bucket=HEAP     (558cded27068) length=1555   data=558cded26de8
>      contents=[~~~~z~~~v~~~~~l~Z...] rc=1

Thanks. The above is a heap bucket which cannot be sent via sendfile, but this does not mean that there is no issue. I think I just used a bad breakpoint. Can you please break in send_brigade_nonblocking and do

dump_brigade(bb)

? As a the response might be split over multiple brigades you need to do this until the file is delivered.

Another thing: Can you check (probably via strace) if that happens as well for a virtual host that is

- http/1.1 (no ssl)
- Has no PHP, content compression or alike setup, but just delivers static files?

Interesting discovery. a breakpoint on sendfile64/sendfile without SSL (in a virtualhost), sendfile is working: 

(gdb) c
Continuing.

Breakpoint 1, sendfile64 () at ../sysdeps/unix/syscall-template.S:81
81	T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
(gdb) c
Continuing.

#1  0x00007f80c891e5af in apr_socket_sendfile (sock=0x55d4ecb81a20, file=0x55d4ecd3c4e0, hdtr=0x7f80c8b33e40 <no_hdtr>, offset=<optimized out>, len=0x7ffd49aa0c08, flags=<optimized out>)
    at network_io/unix/sendrecv.c:333
#2  0x000055d4eb2b3185 in sendfile_nonblocking (s=0x55d4ecb81a20, bucket=0x55d4ecd20d48, ctx=0x55d4ecb82150, c=0x55d4ecb81c10) at core_filters.c:828
#3  0x000055d4eb2b296d in send_brigade_nonblocking (s=0x55d4ecb81a20, bb=0x55d4ecd3c5e0, ctx=0x55d4ecb82150, c=0x55d4ecb81c10) at core_filters.c:660
#4  0x000055d4eb2b243d in ap_core_output_filter (f=0x55d4ecb82098, new_bb=0x55d4ecd3c5e0) at core_filters.c:505
#5  0x000055d4eb28fb3c in ap_pass_brigade (next=0x55d4ecb82098, bb=0x55d4ecd3c5e0) at util_filter.c:590
#6  0x000055d4eb2e029f in ap_http_outerror_filter (f=0x55d4ecd2b298, b=0x55d4ecd3c5e0) at http_filters.c:1907
#7  0x000055d4eb28fb3c in ap_pass_brigade (next=0x55d4ecd2b298, bb=0x55d4ecd3c5e0) at util_filter.c:590
#8  0x000055d4eb2df2d0 in ap_http_header_filter (f=0x55d4ecd2b270, b=0x55d4ecd3c5e0) at http_filters.c:1541
#9  0x000055d4eb28fb3c in ap_pass_brigade (next=0x55d4ecd2b270, bb=0x55d4ecd3c5e0) at util_filter.c:590
#10 0x000055d4eb2996e0 in ap_content_length_filter (f=0x55d4ecd2b248, b=0x55d4ecd3c5e0) at protocol.c:1956


I confirmed this with an strace as well:

# strace -f $(ps auxfww | egrep [h]ttpd | awk '{print $2}' | sed 's/^/-p/g') 2>&1 -e sendfile64,sendfile
strace: Process 17680 attached
sendfile(13, 17, [0] => [77416], 83306) = 77416
sendfile(13, 17, [77416], 5890)         = -1 EAGAIN (Resource temporarily unavailable)
sendfile(13, 17, [77416], 5890)         = -1 EAGAIN (Resource temporarily unavailable)
sendfile(13, 17, [77416] => [83306], 5890) = 5890

Reproduction steps I'm using to demonstrate this behavior, Create a large index.html, ie: 

php -i > index.html 

Use curl locally / remotely (or a browser) 

curl -k https://virtualhost/index.html <-- self signed certificate host

This works, though:

curl http://virtualhost/index.htm

Do you still need the dump_brigade(bb) with the above inforation?

If I read this correctly, everything is working without TLS. 

The thing is that a TLS encrypted connection cannot make use of sendfile() on common operation systems.

There have been efforts by Netflix in the past to have an encrypted TLS sendfile() (<https://netflixtechblog.com/serving-100-gbps-from-an-open-connect-appliance-cdb51dda3b99>), but that requires OS specific support.

Ahh.. Makes sense now. I'll close this out, as I was unaware of sendfile() not working with TLS (maybe kernel TLS on Linux could solve this issue)

Closing this out. 

Thank you.