Bug 62924 - tomcat-users.xml has open file descriptors and opens tomcat-users.xml all the time
Summary: tomcat-users.xml has open file descriptors and opens tomcat-users.xml all the...
Status: RESOLVED FIXED
Alias: None
Product: Tomcat 9
Classification: Unclassified
Component: Manager (show other bugs)
Version: 9.0.13
Hardware: Other Linux
: P2 major (vote)
Target Milestone: -----
Assignee: Tomcat Developers Mailing List
URL:
Keywords:
: 62958 (view as bug list)
Depends on:
Blocks:
 
Reported: 2018-11-18 15:34 UTC by Erich Siffert
Modified: 2018-11-27 20:11 UTC (History)
1 user (show)



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Erich Siffert 2018-11-18 15:34:37 UTC
Hi everybody
Since tomcat 9.0.13 and the ability to monitor "tomcat-users.xml" was introduced we encounter quite strange behaviour:
we run into "too many files open" exception after a certain time when tomcat is running.
nothing changed on the apps which run on this instance, except the version of tomcat (from 9.0.12 to 9.0.13)

I found now the following strange thing:
when doing "lsof" on the PID runnning tomcat, I see:

[gtt@tgttcn0:~/tomcat/logs:1061> lsof  -p 16829 | grep tomcat-users | wc -l
260

a couple of seconds later:

[gtt@tgttcn0:~/tomcat/logs:1062> lsof  -p 16829 | grep tomcat-users | wc -l
262

and about 20 seconds later:

[gtt@tgttcn0:~/tomcat/logs:1062> lsof  -p 16829 | grep tomcat-users | wc -l
264

and nothing is really going on on the tomcat at the moment.


It looks like something is opening tomcat-users.xml all the time and does not close it anymore....


Here the same "lsof" command on one of our servers running tomcat 9.0.12:

[gtt@pgttcn1:~:1008> lsof -p 26572 | grep tomcat-users | wc -l
0


thanks for having a look into this
Erich
Comment 1 Erich Siffert 2018-11-18 15:42:57 UTC
here a full listing of the "lsof" command to better illustrate the issue....

[gtt@tgttcn0:~/tomcat/logs:1064> lsof  -p 16829 | grep tomcat-users
java    16829  gtt   68r      REG              259,5       241 268435594 /home/gtt/apache-tomcat-9.0.13/conf/tomcat-users.xml
java    16829  gtt   71r      REG              259,5       241 268435594 /home/gtt/apache-tomcat-9.0.13/conf/tomcat-users.xml
java    16829  gtt   75r      REG              259,5       241 268435594 /home/gtt/apache-tomcat-9.0.13/conf/tomcat-users.xml
java    16829  gtt   77r      REG              259,5       241 268435594 /home/gtt/apache-tomcat-9.0.13/conf/tomcat-users.xml
java    16829  gtt   78r      REG              259,5       241 268435594 /home/gtt/apache-tomcat-9.0.13/conf/tomcat-users.xml
java    16829  gtt   79r      REG              259,5       241 268435594 /home/gtt/apache-tomcat-9.0.13/conf/tomcat-users.xml
java    16829  gtt   80r      REG              259,5       241 268435594 /home/gtt/apache-tomcat-9.0.13/conf/tomcat-users.xml
java    16829  gtt   81r      REG              259,5       241 268435594 /home/gtt/apache-tomcat-9.0.13/conf/tomcat-users.xml
Comment 2 Erich Siffert 2018-11-18 18:09:27 UTC
ok, I digged into this a little bit deeper -- I figured out that there is a new flag "watchSource=false" (default is true) which can be set:

    <Resource name="UserDatabase" auth="Container"
              type="org.apache.catalina.UserDatabase"
              description="User database that can be updated and saved"
              factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
              pathname="conf/tomcat-users.xml"
	watchSource="false"			  />

when I set the flag to false, everything seems to be fine!

I suspect the issue is somewhere here:

MemoryUserDatabase.java

in method "public void backgroundProcess()" which reads the file:
[...]
        URI uri = ConfigFileLoader.getURI(getPathname());
        try {
            URL url = uri.toURL();
            URLConnection uConn = url.openConnection();
[...]

but it looks like it remains open....
Comment 3 Remy Maucherat 2018-11-19 11:46:22 UTC
There is nothing that can be closed here, and the URL is a regular file URL. However, calling getLastModified on it indeed leaks a descriptor. There's a problem somewhere and more investigation is needed, but this is odd.
Comment 4 Mark Thomas 2018-11-19 11:53:24 UTC
Thanks for the report.

Fixed in:
- trunk for 9.0.14 onwards
Comment 5 Mark Thomas 2018-11-27 20:11:00 UTC
*** Bug 62958 has been marked as a duplicate of this bug. ***