Bug 65158 - delivered files corrupted on Linux with architecture ARM64
Summary: delivered files corrupted on Linux with architecture ARM64
Status: NEW
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: Build (show other bugs)
Version: 2.4.46
Hardware: Other Linux
: P2 normal (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-02-24 20:52 UTC by hp4everything
Modified: 2021-03-09 13:34 UTC (History)
0 users



Attachments
APR mmap aligned to page (2.24 KB, patch)
2021-03-01 00:11 UTC, Yann Ylavic
Details | Diff
config.log for httpd and apr (24.65 KB, application/x-gzip)
2021-03-06 17:38 UTC, hp4everything
Details
Fix for ./buildconf (543 bytes, patch)
2021-03-07 18:45 UTC, Yann Ylavic
Details | Diff
strace output (82.06 KB, application/x-gzip)
2021-03-08 10:48 UTC, hp4everything
Details
strace.out and error_log (81.22 KB, application/x-gzip)
2021-03-08 16:11 UTC, hp4everything
Details
correction with dumpio on (81.36 KB, application/x-gzip)
2021-03-08 16:34 UTC, hp4everything
Details
with corrected ServerRoot (80.64 KB, application/x-gzip)
2021-03-08 17:23 UTC, hp4everything
Details
conf- and log-files (65.28 KB, application/x-gzip)
2021-03-09 10:46 UTC, hp4everything
Details

Note You need to log in before you can comment on or make changes to this bug.
Description hp4everything 2021-02-24 20:52:59 UTC
downloaded jpg- or pdf-files are corrupted (prepended by some garbage), when

- apache runs on Raspberry Pi4 (ARM64) with Linux aarch64 5.11 (Archlinuxarm)
- the files are located on a cifs-mount point within the document root

The access-log file reports correct delivery (200) with correct length
no error in error-log file

wget reports problem with received HTTP-message (fallback to HTTP/0.9)
firefox reports illegal file content

This problem does not happen on the same platform with a 32Bit-Linux
This problem does not happen with lighttpd-server
 
This problem occurs with the official package from Archlinuxarm-repository

This problem occurs also, when I build apache from source by following the build-description on the apache-website (downloaded: 2.4.46 release)

This problem only occurs when apache accesses the file on the cifs-mount.

It does not occur when accessing the file directly  on the mount by other applications PDF-viewer or image-viewer.
Comment 1 Yann Ylavic 2021-02-24 21:27:30 UTC
Does "EnableSendfile off" help?
Comment 2 Yann Ylavic 2021-02-24 21:29:55 UTC
(In reply to Yann Ylavic from comment #1)
> Does "EnableSendfile off" help?

And/or "EnableMMAP off"
Comment 3 hp4everything 2021-02-25 13:37:54 UTC
"EnableSendfile off"  didn't work.

But together with "EnableMMAP off"  it works!

some questions: 

my http.conf contained "#EnableMMAP off", so I thought that it is disabled by default. But that doesn't seem to be correct: comment lines don't indicate the default value? 

What does this solution mean: does the 64Bit-kernel for ARM architectures have a problem, which the 32Bit-kernel doesn't have?

Why does apache not detect this problem? And why does it send a corrupted HTTP-message according to the wget-error report? Sorry, I'm not a kernel-expert, but if apache is able to follow symlinks, why is it not possible to follow mount-points and disable the feature in this case? 

wouldn't it be better to set the above options as default, so that a web-administrator has to enable this feature intentionally after searching his webspace for mounts?
Comment 4 Yann Ylavic 2021-03-01 00:11:24 UTC
Created attachment 37750 [details]
APR mmap aligned to page

Could you please try to apply this patch to the APR library you are using to build httpd from sources, and then retest?
Comment 5 Yann Ylavic 2021-03-01 00:17:29 UTC
(In reply to Yann Ylavic from comment #4)
> and then retest?
(with the default "EnableMMAP on" of course)
Comment 6 hp4everything 2021-03-02 16:28:05 UTC
I hope I didn't make a mistake, but the patch doesn't seem to work:

- since my distro archlinux had installed apr by default I've downloaded it and built it from source(my apache from source was running with the pre-installed apr, but also version 1.7.0)

- apr including the patch was built without problems and I replaced the library libapr-1 manually in the /usr/lib directory

- after restart apache with "EnableMMAP on" the error also appears with the new library

same symptom: wget reports HTTP-protocol problems:

HTTP-Anforderung gesendet, auf Antwort wird gewartet … 200 Keine Header, vermutlich ist es HTTP/0.9.

(english: HTTP-request sent, waiting for response ... 200 No Header, probably version HTTP/0.9)

--------------------------------------------------
Here a dump of the garbage by which the transferred file is prepended (seems to be some file-info with strings "modified" and "Content-Length"

0000000   i   f   i   e   d   :  sp   T   h   u   ,  sp   1   9  sp   M
0000020   a   r  sp   2   0   0   9  sp   2   2   :   0   1   :   0   8
0000040  sp   G   M   T  cr  nl   E   T   a   g   :  sp   "   1   5   2
0000060   1   0   6   -   4   6   5   7   f   e   f   f   6   7   1   0
0000100   0   "  cr  nl   A   c   c   e   p   t   -   R   a   n   g   e
0000120   s   :  sp   b   y   t   e   s  cr  nl   C   o   n   t   e   n
0000140   t   -   L   e   n   g   t   h   :  sp   1   3   8   4   7   1
Comment 7 hp4everything 2021-03-02 17:28:13 UTC
Concerning my first message I've probably made a mistake: I've tested again on ARM-architecture withe 32Bit and the same problem occurs, so it seems to be independent of 64- or 32-Bit. Since there a are probably a lot of apaches out there  running on big-endian-architectures without problems, it seems to be specific for ARM or at least RaspberryPi-ARM-architecture.
Comment 8 Yann Ylavic 2021-03-02 21:41:48 UTC
(In reply to hp4everything from comment #6)
> 
> - since my distro archlinux had installed apr by default I've downloaded it
> and built it from source(my apache from source was running with the
> pre-installed apr, but also version 1.7.0)
> 
> - apr including the patch was built without problems and I replaced the
> library libapr-1 manually in the /usr/lib directory

I thought you were going to build httpd and APR from sources without replacing your system libraries, which may be an issue per se.

Could you make a separate/fesh httpd-2.4.46 + apr-1.7.0 installation in e.g. /usr/local/httpd, with something like this for httpd and apr(-util):
$ tar xf httpd-2.4.46.tar.gz
$ tar xf apr-1.7.0.tar.bz -C httpd-2.4.46/srclib/apr --strip-components=1
$ tar xf apr-util-1.6.1.tar.gz -C httpd-2.4.46/srclib/apr-util --strip-components=1
$ cd httpd-2.4.46
$ ./buildconf
$ ./configure "LDFLAGS=-Wl,-rpath,/usr/local/httpd/lib" --prefix=/usr/local/httpd --with-included-apr ...
$ make && sudo make install

You could then edit/replace "/usr/local/httpd/conf/httpd.conf" for your needs and then start with:
$ sudo /usr/local/httpd/conf/bin/httpd -f /usr/local/httpd/conf/httpd.conf -k start

Sorry, I don't have an ARM box nor a cifs mount (nor your specific configuration) at hand to test this by myself..
Comment 9 Yann Ylavic 2021-03-02 21:45:02 UTC
(In reply to Yann Ylavic from comment #8)
> $ ./configure "LDFLAGS=-Wl,-rpath,/usr/local/httpd/lib" --prefix=/usr/local/httpd --with-included-apr ...

This should be (-L added in LDFLAGS):
$ ./configure "LDFLAGS=-L/usr/local/httpd/lib -Wl,-rpath,/usr/local/httpd/lib" --prefix=/usr/local/httpd --with-included-apr ...
Comment 10 Yann Ylavic 2021-03-02 21:59:02 UTC
And I forgot to patch the APR sources extracted to "httpd-2.4.46/srclib/apr" with attachment 37750 [details] in my above procedure too..
Comment 11 hp4everything 2021-03-03 10:53:05 UTC
I tried it with your sequence of commands and the configure-step exited with

-----------------------------------------------------
checking size of void*... 8
checking size of char... 1
checking size of short... 2
checking size of int... 4
checking size of long... 8
checking size of long long... 8
checking whether int64_t and int use fmt %d... no
checking whether int64_t and long use fmt %ld... no
checking whether int64_t and long long use fmt %lld... no
configure: error: could not determine the string function for int64_t
configure failed for srclib/apr
-----------------------------------------------------

My machine may not have installed all development-packages required for building from source.

Or do I need an additional configure-parameter (path to include file?) ?

When I try to build a standalone apr I also get this error. The difference to yesterday is your 'buildconf'-command. If I omit this command the configure-step works fine: at least no error message, but will  the configuration be bad or inadequate for ARM?

The buildconf-induced problem seems to be related ti inttype-sizes, because configure can't compile a test-program. Here the log-file-error:

-----------------------------------------------------
| #define SIZEOF_INT 4
| #define SIZEOF_LONG 8
| #define SIZEOF_LONG_LONG 8
| /* end confdefs.h.  */
| #include "confdefs.h"
| 
|    #include <sys/types.h>
| #include <stdio.h>
| #ifdef HAVE_STDINT_H
| #include <stdint.h>
| #endif
| 
|    int main(int argc, const char *const *argv) {
| 
|     int64_t chk1, *ptr1;
|     long long chk2, *ptr2 = &chk1;
|     ptr1 = &chk2;
|     *ptr1 = *ptr2 = 0;
|     printf("%lld %lld", chk1, chk2);
| 
|       return 0; }
| 
configure:25890: result: no
configure:25900: error: could not determine the string function for int64_t
-----------------------------------------------------


Sorry for not being very familiar with this build-process.
Comment 12 hp4everything 2021-03-03 11:13:23 UTC
I'm not a package-developer for archlinux/archlinuxarm, but may be the guy responsible for the apache-packages also omitted the 'buildconf'-command. And then everything compiles fine, but the executable doesn't work correctly for certain architecture-dependent features?
Comment 13 Yann Ylavic 2021-03-06 15:58:01 UTC
Could you please attach the whole config.log file?
Comment 14 hp4everything 2021-03-06 17:38:33 UTC
Created attachment 37759 [details]
config.log for httpd and apr

compressed tar with :   tar -cvzf
Comment 15 Yann Ylavic 2021-03-06 22:37:01 UTC
Thanks, it seems that you hit https://github.com/apache/apr/pull/25
I just merged the patch in the APR repository, here: https://svn.apache.org/viewvc/apr/apr/trunk/build/apr_common.m4?r1=1887279&r2=1887278&pathrev=1887279&view=patch
It should fix your build..
Comment 16 hp4everything 2021-03-07 16:18:35 UTC
I tried to build with the "recommended" versions of the apache website and didn't pull any github-versions:

-rw-r--r--  1 root root   872238  6. Jul 2020  apr-1.7.0.tar.bz2
-rw-r--r--  1 root root       84  6. Jul 2020  apr-1.7.0.tar.bz2.sha256
-rw-r--r--  1 root root   428595  6. Jul 2020  apr-util-1.6.1.tar.bz2
-rw-r--r--  1 root root       89  6. Jul 2020  apr-util-1.6.1.tar.bz2.sha256
drwxr-sr-x 11 root root     4096  6. Mär 18:31 httpd-2.4.46
-rw-r--r--  1 root root  7187805  5. Aug 2020  httpd-2.4.46.tar.bz2
-rw-r--r--  1 root root       87  5. Aug 2020  httpd-2.4.46.tar.bz2.sha256

Could you please write a short note on what sources should be used and how to get these?

git clone ????? httpd apr apr-utils ??????

1 git command for all or separate git-commands for the srclib-subdirectory?
Comment 17 Yann Ylavic 2021-03-07 18:45:31 UTC
Created attachment 37760 [details]
Fix for ./buildconf
Comment 18 Yann Ylavic 2021-03-07 19:07:24 UTC
I didn"t mean that you use any git version or command, both httpd and apr don't use git as repository (github is just a mirror of the sources). Sorry for the confusion with the github link, it's just that the patch to fix your ./buildconf issue was proposed there, as some contributors sometimes do.

So since you already downloaded the httpd and apr sources (tar.gz) locally, you now need to download the patches from attachment 37750 [details] and attachment 37760 [details] (say respectfully as apr1.patch and apr2.patch) at the same place.

Now I'll repeat the procedure from comment 8 (hopefully with less confusion this time):

$ cd /path/to/sources
$ tar xf httpd-2.4.46.tar.gz
$ tar xf apr-1.7.0.tar.bz -C httpd-2.4.46/srclib/apr --strip-components=1
$ tar xf apr-util-1.6.1.tar.gz -C httpd-2.4.46/srclib/apr-util --strip-components=1
$ cd httpd-2.4.46
$ pushd srclib/apr
$ patch -p0 < ../../../apr1.patch
$ patch -p0 < ../../../apr2.patch
$ popd
$ ./buildconf
$ ./configure "LDFLAGS=-L/usr/local/httpd/lib -Wl,-rpath,/usr/local/httpd/lib" --prefix=/usr/local/httpd --with-included-apr ... plus your other usual configure options... 
$ make && sudo make install

There now should be a new installation of httpd+apr in the /usr/local/httpd directory, copy your configuration file to /usr/local/httpd/conf/myhttpd.conf, stop any other httpd instance, and start this one:
$ sudo /usr/local/httpd/bin/httpd -f /usr/local/httpd/conf/myhttpd.conf -k start

Up and listenning according to your configuratin, now ready to test..
Comment 19 Yann Ylavic 2021-03-07 19:13:23 UTC
The tar commands above should work with the bz2 archives too (since you seem to have the .tar.bz2 files and not the .tar.gz ones). Otherwise replace "tar xf .." by "tar xjf ..".
Comment 20 Yann Ylavic 2021-03-07 19:21:59 UTC
After all the source archives have been extracted, the source tree should look like:
- httpd-2.4.46
  `- srclib
     `- apr
        `- include
        `- atomic
        `- ...
     `- apr-util
        `- include
        `- buckets
        `- ...
  `- include
  `- modules
  `- server
  `- ...
Comment 21 hp4everything 2021-03-07 21:35:08 UTC
Thanks for your help.

The build steps executed without error. But wget-downloaded files are corrupted as before, when EnableMMap is not explicitly switched off.

Additional info:

I first had trouble to start apache:

AH00534: httpd: Configuration error: More than one MPM loaded


but that could be solved by disabling LoadModule mod_mpm_prefork in my conf-file.

Since this was not necessary with my previous build without your buildconf-command and also is not necessary in the archlinux-distributed apache, I'd like to know:

- does buildconf lead to a configuration with preloaded or static mpm-Module, which is not prefork?

Since apache other than prefork-mpm cannot work with the archlinux-distribution-php, because the php-library is not compiled threadsafe, archlinux-users have to use the prefork-mpm.
Comment 22 Yann Ylavic 2021-03-08 10:07:59 UTC
Thanks.

Please now start httpd with:
$ sudo strace -o strace.out /usr/local/httpd/bin/httpd -X -f /usr/local/httpd/conf/myhttpd.conf
(this will keep httpd attached to the terminal, with ^C to stop it)

Then run wget (from another terminal/machine) and attach the "strace.out" file here.
 
> Since this was not necessary with my previous build without your
> buildconf-command and also is not necessary in the archlinux-distributed
> apache, I'd like to know:
> 
> - does buildconf lead to a configuration with preloaded or static
> mpm-Module, which is not prefork?

You can use:
$ ./configure --with-mpm=prefork ...
to select the default MPM at build time.
Comment 23 hp4everything 2021-03-08 10:48:56 UTC
Created attachment 37761 [details]
strace output

please extract with "tar -xvzf strace.out.tar.gz"
Comment 24 hp4everything 2021-03-08 10:51:47 UTC
I'm confused with the result:

starting httpd without strace  =>  corrupted file transfer and HTTP-protocol-error

starting httpd with strace  =>  file transfer ok, HTTP-protocol ok


I did this 3 times with the same result:
-----------------------------------------------------------------------------
# wget http://192.168.85.17/haushalt/test.txt
--2021-03-08 11:32:33--  http://192.168.85.17/haushalt/test.txt
Verbindungsaufbau zu 192.168.85.17:80 … verbunden. (translat: connected)
HTTP-Anforderung gesendet, auf Antwort wird gewartet … 200 Keine Header, vermutlich ist es HTTP/0.9.   (translat: 200 no header, probably HTTP/0.9)
Länge: nicht spezifiziert (translat: length not specified )
Wird in »test.txt.1« gespeichert.

test.txt.1               [      <=>            ]     324  --.-KB/s    in 4,8s    

2021-03-08 11:32:38 (68,1 B/s) - »test.txt.1« gespeichert [324]

# wget http://192.168.85.17/haushalt/test.txt
--2021-03-08 11:33:30--  http://192.168.85.17/haushalt/test.txt
Verbindungsaufbau zu 192.168.85.17:80 … verbunden.
HTTP-Anforderung gesendet, auf Antwort wird gewartet … 200 OK
Länge: 17 [text/plain] (translat: length 17, which is ok )
Wird in »test.txt.2« gespeichert.

test.txt.2           100%[====================>]      17  --.-KB/s    in 0s
Comment 25 hp4everything 2021-03-08 10:55:02 UTC
please note: the corrupted transfer takes several seconds, maybe the HTTP-response is pending or terminated by timeout?
Comment 26 Yann Ylavic 2021-03-08 10:57:10 UTC
> I'm confused with the result:
> 
> starting httpd without strace  =>  corrupted file transfer and
> HTTP-protocol-error
> 
> starting httpd with strace  =>  file transfer ok, HTTP-protocol ok

Do you start httpd with the same command line, i.e. without strace:
$ sudo /usr/local/httpd/bin/httpd -X -f /usr/local/httpd/conf/myhttpd.conf

and with strace:
$ sudo strace -o strace.out /usr/local/httpd/bin/httpd -X -f /usr/local/httpd/conf/myhttpd.conf

?
Comment 27 hp4everything 2021-03-08 11:35:43 UTC
that's my command history, I hope I didn't make a mistake...

between 487 and 488 I started a wget from an other terminal, also after 491

  486  strace -o strace.out /usr/local/httpd/bin/httpd -X -f /usr/local/httpd/conf/myhttpd.conf
  487  /usr/local/httpd/bin/httpd -f /usr/local/httpd/conf/myhttpd.conf -k start
  488  /usr/local/httpd/bin/httpd -f /usr/local/httpd/conf/myhttpd.conf -k stop
  489  ls strace.out 
  490  rm strace.out 
  491  strace -o strace.out /usr/local/httpd/bin/httpd -X -f /usr/local/httpd/conf/myhttpd.conf
  492  vi strace.out 
  493  ls -la strace.out 
  494  tar -cvzf strace.out.tar.gz
  495  tar -cvzf strace.out.tar.gz strace.out
Comment 28 hp4everything 2021-03-08 11:40:06 UTC
does strace modify or influence the httpd-execution, e.g. by shifting the executable in memory to page boundaries?

I hope, it's not a simple mistake on my side....
Comment 29 hp4everything 2021-03-08 11:48:55 UTC
I've made an additional try with debug option -X but withut strace.

/usr/local/httpd/bin/httpd -X -f /usr/local/httpd/conf/myhttpd.conf -k start

That doesn't work, because the start command hangs till Ctrl-C

Maybe -X is not supposed to work without strace?
Comment 30 hp4everything 2021-03-08 11:57:57 UTC
to be precise:

the start-command with -X hangs, but the httpd-server is running and produces the error as opposed to the strace version.
Comment 31 Yann Ylavic 2021-03-08 12:33:40 UTC
Yes the -X option is to keep httpd as a single process attached to the terminal, so it will start and "hang" until you stop it with ^C (and not with "-k stop" like when it's started with "-k start"). That's why I talked about starting httpd and wget in two separate terminals, leaving httpd alone ("hanging") in one terminal until you stop it.

I'm a bit confused here too. This PR started with jpeg/pdf files (which I read as large files) corrupted with EnableMMap on. Now a tiny test.txt file of 17 bytes is also corrupted with EnableMMap on (only?) unless httpd is started by strace, right?

Let's restart the test proceduire if you wish.

1. Configure httpd
1.a. Stop any running httpd: "httpd ... -k stop" and ^C on any terminal where you ran "strace ... httpd -X". There shouldn't be any httpd running as shown by "ps -ef" or alike.
1.b. In /usr/local/httpd/conf/httpd.conf:
- modify LogLevel to trace7 
- add/uncomment "LoadModule dumpio_module modules/mod_dumpio.so"
- add "DumpIOInput on" and "DumpIOoutput on"
(note, if mod_dumpio.so is not in the /usr/local/httpd/modules directory, run "./configure ... --enable-dumpio=shared" in the sources directory and "sudo make install" again).
1.c. rm /usr/local/httpd/logs/error_log

2. Test with strace
2.a. In one terminal (terminal 1), run:
$ sudo strace -o strace.out /usr/local/httpd/bin/httpd -X -f /usr/local/httpd/conf/httpd.conf
2.b. In another terminal (terminal 2), run:
$ wget http://192.168.85.17/haushalt/test.txt
2.c. In terminal 1, hit ^C stop stop httpd
2.d. Is the reponse correct/corrupted?

3. Test without strace
3.a. In one terminal (terminal 1), run:
$ sudo /usr/local/httpd/bin/httpd -X -f /usr/local/httpd/conf/httpd.conf
3.b. In another terminal (terminal 2), run:
$ wget http://192.168.85.17/haushalt/test.txt
3.c. In terminal 1, hit ^C stop stop httpd
3.d. Is the reponse correct/corrupted?

4. Attach strace.out and /usr/local/httpd/logs/error_log (.tar.gz) to this PR
Comment 32 hp4everything 2021-03-08 16:11:26 UTC
Created attachment 37762 [details]
strace.out and error_log

unpack with "tar -xvzf"
Comment 33 hp4everything 2021-03-08 16:16:28 UTC
Sorry for changing the test-file. In the meantime I found that probably every type of file seems to be corrupted, even small text files.

I'll try to keep the "test-environment" fixed in the future.

Here the results corresponding to your test-sequence:


########### first terminal ################################################

vim  /usr/local/httpd/conf/myhttpd.conf
# grep LogLev  /usr/local/httpd/conf/myhttpd.conf
# LogLevel: Control the number of messages logged to the error_log.
#LogLevel warn
LogLevel trace7
# grep dumpio  /usr/local/httpd/conf/myhttpd.conf
LoadModule dumpio_module modules/mod_dumpio.so

# strace -o strace.out /usr/local/httpd/bin/httpd -X -f  /usr/local/httpd/conf/myhttpd.conf
[Mon Mar 08 17:01:40.402877 2021] [core:trace3] [pid 445607] core.c(3361): Setting LogLevel for all modules to trace7
^C


########### second terminal ################################################

# wget http://192.168.85.17/haushalt/test.txt
--2021-03-08 17:01:57--  http://192.168.85.17/haushalt/test.txt
Verbindungsaufbau zu 192.168.85.17:80 … verbunden.
HTTP-Anforderung gesendet, auf Antwort wird gewartet … 200 OK
Länge: 17 [text/plain]
Wird in »test.txt.13« gespeichert.

test.txt.13          100%[====================>]      17  --.-KB/s    in 0s      

2021-03-08 17:01:57 (952 KB/s) - »test.txt.13« gespeichert [17/17]


###########################################################################
########### first terminal ################################################

# /usr/local/httpd/bin/httpd -X -f  /usr/local/httpd/conf/myhttpd.conf
[Mon Mar 08 17:02:23.622230 2021] [core:trace3] [pid 445627] core.c(3361): Setting LogLevel for all modules to trace7

########### second terminal ################################################

# wget http://192.168.85.17/haushalt/test.txt
--2021-03-08 17:02:35--  http://192.168.85.17/haushalt/test.txt
Verbindungsaufbau zu 192.168.85.17:80 … verbunden.
HTTP-Anforderung gesendet, auf Antwort wird gewartet … 200 Keine Header, vermutlich ist es HTTP/0.9.
Länge: nicht spezifiziert
Wird in »test.txt.14« gespeichert.

test.txt.14              [      <=>            ]     324  --.-KB/s    in 4,8s    

2021-03-08 17:02:40 (68,1 B/s) - »test.txt.14« gespeichert [324]
Comment 34 hp4everything 2021-03-08 16:27:07 UTC
sorry, please forget the last results, probably I only loaded the dumpio-module, but forgot to activate it. I'll come back with new test results.
Comment 35 hp4everything 2021-03-08 16:34:46 UTC
Created attachment 37763 [details]
correction with dumpio on
Comment 36 hp4everything 2021-03-08 16:40:13 UTC
I've added the dumpio on commands.

When reading the apache-docs I've seen that the dumpio-module should be configured with "dumpio:trace7". Is this the same as "LogLevel trace7" ?

Here the terminal logs with the new timestamps:

---------------------------
strace -o strace.out /usr/local/httpd/bin/httpd -X -f  /usr/local/httpd/conf/myhttpd.conf
[Mon Mar 08 17:30:40.293244 2021] [core:trace3] [pid 446273] core.c(3361): Setting LogLevel for all modules to trace7
^C

# /usr/local/httpd/bin/httpd -X -f  /usr/local/httpd/conf/myhttpd.conf
[Mon Mar 08 17:31:28.929678 2021] [core:trace3] [pid 446297] core.c(3361): Setting LogLevel for all modules to trace7

---------------------------
wget http://192.168.85.17/haushalt/test.txt
--2021-03-08 17:30:52--  http://192.168.85.17/haushalt/test.txt
Verbindungsaufbau zu 192.168.85.17:80 … verbunden.
HTTP-Anforderung gesendet, auf Antwort wird gewartet … 200 OK
Länge: 17 [text/plain]
Wird in »test.txt.15« gespeichert.

test.txt.15          100%[====================>]      17  --.-KB/s    in 0s      

2021-03-08 17:30:52 (706 KB/s) - »test.txt.15« gespeichert [17/17]



# wget http://192.168.85.17/haushalt/test.txt
--2021-03-08 17:31:42--  http://192.168.85.17/haushalt/test.txt
Verbindungsaufbau zu 192.168.85.17:80 … verbunden.
HTTP-Anforderung gesendet, auf Antwort wird gewartet … 200 Keine Header, vermutlich ist es HTTP/0.9.
Länge: nicht spezifiziert
Wird in »test.txt.16« gespeichert.

test.txt.16              [      <=>            ]     324  --.-KB/s    in 4,8s    

2021-03-08 17:31:47 (68,1 B/s) - »test.txt.16« gespeichert [324]
Comment 37 Joe Orton 2021-03-08 16:55:33 UTC
It looks like uou are mixing modules compiled from the distribution and self-compiled in these traces:

execve("/usr/local/httpd/bin/httpd", ["/usr/local/httpd/bin/httpd", "-X", "-f", "/usr/local/httpd/conf/myhttpd.co"...], 0xffffe4f49e88 /* 23 vars */) = 0
openat(AT_FDCWD, "/etc/httpd/modules/mod_authn_file.so", O_RDONLY|O_CLOEXEC) = 4

which is certainly not a good idea, though I'm not sure it can cause the symptoms described. 

Please pass "-d /usr/local/httpd" to httpd as well to avoid this, and/or adjust the config file used.
Comment 38 hp4everything 2021-03-08 17:23:46 UTC
Created attachment 37764 [details]
with corrected ServerRoot
Comment 39 hp4everything 2021-03-08 17:24:22 UTC
ok, I checked it: same result
Comment 40 Yann Ylavic 2021-03-08 20:13:19 UTC
The logs from comment 35 seem to be missing many lines, notably the dumpio ones.
There must be multiple ErroLog and/or LogLevel directives configured in the whole /usr/local/httpd/conf tree.
I see that "/usr/local/httpd/conf/extra/010-std-80.conf" is included somehow, and that a VirtualHost is defined there, and possibly mod_php and other modules are loaded from there too.

Let's try to avoid external configuration files Include(d) from "myhttpd.conf" now, and have a simple VirtualHost directly in the main configuration file like:

Listen 80
<VirtualHost *:80>
    ServerName localhost

    LogLevel trace7
    DumpIOInput on
    DumpIOoutput on

    DocumentRoot "/usr/local/httpd/htdocs"
    <Directory "/usr/local/httpd/htdocs">
        Require all granted
    </Directory>
</VirtualHost>

and then copy "test.txt" in the directory "/usr/local/httpd/htdocs".

The loaded modules could be limited to something like:

LoadModule unixd_module modules/mod_unixd.so
LoadModule mpm_event_module modules/mod_mpm_event.so
LoadModule authn_core_module modules/mod_authn_core.so
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule dumpio_module modules/mod_dumpio.so

That is, a minimal configuration..
Comment 41 Yann Ylavic 2021-03-08 20:18:08 UTC
>     DocumentRoot "/usr/local/httpd/htdocs"
>     <Directory "/usr/local/httpd/htdocs">
>         Require all granted
>     </Directory>
> 
> and then copy "test.txt" in the directory "/usr/local/httpd/htdocs".

Obviously this needs to remain the cifs mounted directory, so replace "/usr/local/httpd/htdocs" above with whatever that is..
Comment 42 hp4everything 2021-03-09 10:45:09 UTC
Sorry for the confusion. When I followed your recommendation to simplify the configuration as much as possible, the problem disappeared.

So I reactivated parts of the original conf-tree step by step and eventually was able to get a very short myhttpd.conf which produces the error, which seems to be induced by a "forgotten LogLevel debug" statement down the conf-tree (see attached myhttpd.conf).

So the situation seems to me : 

- delete the "LogLevel debug" from myhttpd.conf => there is no problem at all
- activate the "LogLevel debug" in myhttpd.conf => problem with normal httpd-binary, but problem compensated by running strace

I'm not sure whether this is really an "ARM-architecture"-problem, since I don't have an apache on Intel.

Probably it's not even a problem at all, since nobody activates debug in productive environments. But the strace-dependency seems very strange to me and might point to some potential problem. 

I'll attach the conf- and log-files. The commands I've used in this very simple environment:  

/usr/local/httpd/bin/httpd -X -f /usr/local/httpd/conf/myhttpd.conf -k start
strace -o strace.out /usr/local/httpd/bin/httpd -X -f  /usr/local/httpd/conf/myhttpd.conf
Comment 43 hp4everything 2021-03-09 10:46:11 UTC
Created attachment 37766 [details]
conf- and log-files
Comment 44 Yann Ylavic 2021-03-09 12:33:22 UTC
Could you provide the error_log with "LogLevel trace7" please?
Comment 45 hp4everything 2021-03-09 13:34:25 UTC
I tried

LogLevel trace7

and 

LogLevel debug
LogLevel trace7

but then everything is ok.

"LogLevel debug" has to present alone or at least the last for the problem to occur.


Is there any way to specify debug and trace7 simultaneously?

Or does the error_log with dumpio, but without the error occuring help you?