Bug 62924 - tomcat-users.xml has open file descriptors and opens tomcat-users.xml all the time
Summary: tomcat-users.xml has open file descriptors and opens tomcat-users.xml all the...
Status: RESOLVED FIXED
Alias: None
Product: Tomcat 9
Classification: Unclassified
Component: Manager (show other bugs)
Version: 9.0.13
Hardware: Other Linux
: P2 major (vote)
Target Milestone: -----
Assignee: Tomcat Developers Mailing List
URL:
Keywords:
: 62958 (view as bug list)
Depends on:
Blocks:
 
Reported: 2018-11-18 15:34 UTC by Erich Siffert
Modified: 2019-12-06 10:20 UTC (History)
2 users (show)



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Erich Siffert 2018-11-18 15:34:37 UTC
Hi everybody
Since tomcat 9.0.13 and the ability to monitor "tomcat-users.xml" was introduced we encounter quite strange behaviour:
we run into "too many files open" exception after a certain time when tomcat is running.
nothing changed on the apps which run on this instance, except the version of tomcat (from 9.0.12 to 9.0.13)

I found now the following strange thing:
when doing "lsof" on the PID runnning tomcat, I see:

[gtt@tgttcn0:~/tomcat/logs:1061> lsof  -p 16829 | grep tomcat-users | wc -l
260

a couple of seconds later:

[gtt@tgttcn0:~/tomcat/logs:1062> lsof  -p 16829 | grep tomcat-users | wc -l
262

and about 20 seconds later:

[gtt@tgttcn0:~/tomcat/logs:1062> lsof  -p 16829 | grep tomcat-users | wc -l
264

and nothing is really going on on the tomcat at the moment.


It looks like something is opening tomcat-users.xml all the time and does not close it anymore....


Here the same "lsof" command on one of our servers running tomcat 9.0.12:

[gtt@pgttcn1:~:1008> lsof -p 26572 | grep tomcat-users | wc -l
0


thanks for having a look into this
Erich
Comment 1 Erich Siffert 2018-11-18 15:42:57 UTC
here a full listing of the "lsof" command to better illustrate the issue....

[gtt@tgttcn0:~/tomcat/logs:1064> lsof  -p 16829 | grep tomcat-users
java    16829  gtt   68r      REG              259,5       241 268435594 /home/gtt/apache-tomcat-9.0.13/conf/tomcat-users.xml
java    16829  gtt   71r      REG              259,5       241 268435594 /home/gtt/apache-tomcat-9.0.13/conf/tomcat-users.xml
java    16829  gtt   75r      REG              259,5       241 268435594 /home/gtt/apache-tomcat-9.0.13/conf/tomcat-users.xml
java    16829  gtt   77r      REG              259,5       241 268435594 /home/gtt/apache-tomcat-9.0.13/conf/tomcat-users.xml
java    16829  gtt   78r      REG              259,5       241 268435594 /home/gtt/apache-tomcat-9.0.13/conf/tomcat-users.xml
java    16829  gtt   79r      REG              259,5       241 268435594 /home/gtt/apache-tomcat-9.0.13/conf/tomcat-users.xml
java    16829  gtt   80r      REG              259,5       241 268435594 /home/gtt/apache-tomcat-9.0.13/conf/tomcat-users.xml
java    16829  gtt   81r      REG              259,5       241 268435594 /home/gtt/apache-tomcat-9.0.13/conf/tomcat-users.xml
Comment 2 Erich Siffert 2018-11-18 18:09:27 UTC
ok, I digged into this a little bit deeper -- I figured out that there is a new flag "watchSource=false" (default is true) which can be set:

    <Resource name="UserDatabase" auth="Container"
              type="org.apache.catalina.UserDatabase"
              description="User database that can be updated and saved"
              factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
              pathname="conf/tomcat-users.xml"
	watchSource="false"			  />

when I set the flag to false, everything seems to be fine!

I suspect the issue is somewhere here:

MemoryUserDatabase.java

in method "public void backgroundProcess()" which reads the file:
[...]
        URI uri = ConfigFileLoader.getURI(getPathname());
        try {
            URL url = uri.toURL();
            URLConnection uConn = url.openConnection();
[...]

but it looks like it remains open....
Comment 3 Remy Maucherat 2018-11-19 11:46:22 UTC
There is nothing that can be closed here, and the URL is a regular file URL. However, calling getLastModified on it indeed leaks a descriptor. There's a problem somewhere and more investigation is needed, but this is odd.
Comment 4 Mark Thomas 2018-11-19 11:53:24 UTC
Thanks for the report.

Fixed in:
- trunk for 9.0.14 onwards
Comment 5 Mark Thomas 2018-11-27 20:11:00 UTC
*** Bug 62958 has been marked as a duplicate of this bug. ***
Comment 6 Eduardo Guadalupe Quintanilla 2018-12-13 16:32:13 UTC
I found a question about a possibly related bug in Tomcat https://stackoverflow.com/questions/53745188/tomcat-undeploys-wars-automatically-after-some-time.

Could the log include details of the exception to easily confirm the cause of the issue?

 https://github.com/apache/tomcat/blob/40500131c71ec8116eb35628ddf6abd515c012ba/java/org/apache/catalina/users/MemoryUserDatabase.java#L447
Comment 7 Mark Thomas 2018-12-13 17:04:48 UTC
(In reply to Eduardo Guadalupe Quintanilla from comment #6)
> I found a question about a possibly related bug in Tomcat
> https://stackoverflow.com/questions/53745188/tomcat-undeploys-wars-
> automatically-after-some-time.
> 
> Could the log include details of the exception to easily confirm the cause
> of the issue?
> 
>  https://github.com/apache/tomcat/blob/
> 40500131c71ec8116eb35628ddf6abd515c012ba/java/org/apache/catalina/users/
> MemoryUserDatabase.java#L447


That is a network issue and unrelated to this bug.