Bug 62024 - Suspected memory leak of org.apache.coyote.AbstractProtocol$ConnectionHandler object while using Websocket
Summary: Suspected memory leak of org.apache.coyote.AbstractProtocol$ConnectionHandler...
Status: RESOLVED FIXED
Alias: None
Product: Tomcat 8
Classification: Unclassified
Component: WebSocket (show other bugs)
Version: 8.5.24
Hardware: PC All
: P2 major (vote)
Target Milestone: ----
Assignee: Tomcat Developers Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-01-19 22:48 UTC by SergeP
Modified: 2018-02-05 22:56 UTC (History)
0 users



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description SergeP 2018-01-19 22:48:31 UTC
When we run our app for 2-3 hours we experience this leak related to our web sockets connections:

One instance of "org.apache.coyote.AbstractProtocol$ConnectionHandler" loaded by "java.net.URLClassLoader @ 0x720029098" occupies 2,153,196,128 (88.10%) bytes. The memory is accumulated in one instance of "java.util.concurrent.ConcurrentHashMap$Node[]" loaded by "<system class loader>".

Keywords
org.apache.coyote.AbstractProtocol$ConnectionHandler
java.util.concurrent.ConcurrentHashMap$Node[]
java.net.URLClassLoader @ 0x720029098

Please help
Comment 1 Mark Thomas 2018-01-24 16:30:34 UTC
There is nothing in this report that indicates a bug in Apache Tomcat and no information that might enable a developer to reproduce the behaviour you are seeing.

Bugzilla is not a support forum. Please use the Tomcat users mailing list to debug this issue. If that discussion concludes that there is a Tomcat bug, please re-open this issue and provide the necessary steps to reproduce the issue on a clean install of the latest release of any currently supported branch.

http://tomcat.apache.org/lists.html#tomcat-users
Comment 2 SergeP 2018-01-24 18:17:47 UTC
Mark

Looks like it did not save my environment - Windows Server 2012

Also these bugs might be related to this one:

*    [57546](https://bz.apache.org/bugzilla/show_bug.cgi?id=57546)
*    [57750](https://bz.apache.org/bugzilla/show_bug.cgi?id=57750)

The 57546 bug looks very similar to what we are experiencing. We tested on Linux and so far we do not see the same behavior. Also this is our web socket code:

package ad.ecs.async.websocket;

import java.io.IOException;
import java.util.Arrays;

import javax.websocket.CloseReason;
import javax.websocket.OnClose;
import javax.websocket.OnError;
import javax.websocket.OnMessage;
import javax.websocket.OnOpen;
import javax.websocket.Session;
import javax.websocket.server.ServerEndpoint;

import ad.common.Global;
import ad.ecs.async.AsyncEngine;
import ad.ecs.async.AsyncResponse;
import ad.ecs.async.AsyncType;
import ad.ecs.db.DatabaseEngine;
import ad.ecs.db.paradox.User;
import ad.ecs.security.engine.SecurityEngine;

/**
 * @author Serge Perepel
 * @since Aug 23, 2017 12:23:32 PM
 */
@ServerEndpoint(value = "/asyncMsg", encoders = AsyncResponseEncoder.class)
public class ECSAsync {
    
    @OnOpen
    public void open(Session session) throws IOException{
        session.getBasicRemote().sendText("Connection Established");
    }

    @OnMessage
    public String login(String sessionID, Session session) {
    	AsyncEngine.INSTANCE.wsConnect(session, sessionID);
    	org.hibernate.Session dbSession = DatabaseEngine.getSessionFactory().openSession();
    	try {
    		int userID = SecurityEngine.INSTANCE.getUserIDBasedOnSessionID(sessionID);
    		User user = (User) dbSession.get(User.class, userID);
    		if (user != null) {
    			if (user.getNextLogin() == 1) {
    				AsyncResponse response = new AsyncResponse();
    				response.setType(AsyncType.None);
    				response.setData(Arrays.asList("PASSWORD"));
    				response.setObjData("PASSWORD");
    				AsyncEngine.INSTANCE.addTransientResult(sessionID, response);
    			}
    		}
    	} finally {
    		dbSession.close();
    	}
    	return "ok"; 
    }
    
    @OnClose 
    public void close(Session session, CloseReason reason) {
        AsyncEngine.INSTANCE.wsDisconnect(session);
    }
    
    @OnError
    public void error(Session session, Throwable error) {
        Global.INSTANCE.getLogHelper().exception(error);
        //session.close(new CloseReason(closeCode, reasonPhrase));
    }
}

Front end opens connection and sends invalid sessionID which causes
the line `AsyncEngine.INSTANCE.wsConnect(session, sessionID);` to throw exception and after that disconnect happens on the front end. At this point front end opens new connection and process goes into the loop. I'm assuming that the connection handler suppose to get freed after a disconnect happen. But it seems to accumulate. You can try this code and just replace code in the login method to always throw the exception. On the front end onDisconnect try to open new connection and send a random message to the web socket. Hopefully this will reproduce the issue for you.
Comment 3 SergeP 2018-01-30 15:16:57 UTC
We added the code to reproduce the leak:

https://www.mail-archive.com/users@tomcat.apache.org/msg128214.html
Comment 4 Mark Thomas 2018-02-03 21:22:07 UTC
Thanks for the test case. It makes investigation so much easier. The leak stands out very clearly in a profiler.

I've tracked down a bug in the exception handling case and I have a patch for that. I still need to look at the normal close variant.
Comment 5 Mark Thomas 2018-02-04 22:19:28 UTC
Fixed in:
- trunk for 9.0.5 onwards
- 8.5.x for 8.5.28 onwards

This appears to have been a regression in the connector refactoring in 8.5.x. Testing with 8.0.x shows it handles it correctly. Also, I could only trigger an issue when using an exception. Clean closes were handled correctly.
Comment 6 SergeP 2018-02-05 14:20:31 UTC
Thank you very much
Comment 7 SergeP 2018-02-05 17:06:17 UTC
Mark- Our technicians are wondering if there a patch fix that can be applied to existing Windows versions of Tomcat
Comment 8 Christopher Schultz 2018-02-05 22:56:00 UTC
Short answer: the Tomcat team provides replacement versions, not patches. You will need to upgrade to 8.5.28 when it becomes available.

The Tomcat 8.5 RM announced the intention of rolling a release and calling for a vote in the next few days. You are welcome to join the tomcat-dev mailing list and vote (non-binding) on the release. If all goes well, I'd expect a release to be ready for you by next Monday.