Created attachment 36224 [details] Trivial patch that addresses this bug Some JMX clients such as Prometheus'[1] jmx_exporter[2] will bulk-load all attributes of an MBean for performance reasons. This means that one non-Serializable attribute is sufficient to make all information from the MBean inaccessible. I could readily reproduce a problem with the ThreadPool MBean, specifically with the socketProperties property which is not Serializable. Reproducer: # enable JMX's full logging cat >logging.properties <<<" handlers=java.util.logging.ConsoleHandler java.util.logging.ConsoleHandler.level=ALL io.prometheus.jmx.level=ALL io.prometheus.jmx.shaded.io.prometheus.jmx.level=ALL " # configure reading from a local Tomcat instance with JMX configured on port 3333 cat >config.yml <<<" hostPort: localhost:3333 whitelistObjectNames: - Catalina:type=ThreadPool,name=* rules: - pattern: ".*" " # launch jmx_exporter /usr/bin/java -Djava.util.logging.config.file=logging.properties -jar jmx_prometheus_httpserver.jar 5556 jmx_exporter.yml Visiting with a browser http://localhost:5556/metrics will trigger jmx_exporter and the following error is produced: FINE: scrape: 'Catalina:type=ThreadPool,name="http-apr-127.0.0.1-8080"_[currentThreadsBusy, paused, sSLCertificateKeyFile, sSLProtocol, keyAlias, truststoreType, sSLPassword, sendfileCount, sSLCACertificatePath, useCometTimeout, socketProperties, p ollTime, sSLHonorCipherOrder, currentThreadCount, trustManagerClassName, sSLCARevocationFile, usePolling, keystorePass, trustMaxCertLength, maxThreads, truststoreAlgorithm, tcpNoDelay, algorithm, keepAliveTimeout, maxKeepAliveRequests, localPort, t ruststoreProvider, acceptorThreadCount, soTimeout, daemon, minSpareThreads, sSLDisableSessionTickets, sessionCacheSize, acceptorThreadPriority, backlog, port, maxHeaderCount, sSLCARevocationPath, keystoreType, name, sslProtocol, soLinger, sessionTi meout, useComet, sSLVerifyDepth, clientAuth, modelerType, sSLCertificateFile, connectionCount, sendfileThreadCount, threadPriority, running, executorTerminationTimeoutMillis, allowUnsafeLegacyRenegotiation, sslEnabledProtocolsArray, jniSslContext, ciphers, truststorePass, sSLEnabled, sendfileSize, maxThreadsWithExecutor, crlFile, sSLVerifyClient, maxConnections, keystoreProvider, keepAliveCount, sSLCipherSuite, deferAccept, sSLDisableCompression, useSendfile, sSLCertificateChainFile, sSLCACe rtificateFile, ciphersUsed, keyPass, bindOnInit, sSLInsecureRenegotiation, truststoreFile, keystoreFile, useServerCipherSuitesOrder]': Fail: java.rmi.UnmarshalException: error unmarshalling return; nested exception is: java.io.WriteAbortedException: writing aborted; java.io.NotSerializableException: org.apache.tomcat.util.net.SocketProperties I have successfully tested a trivial patch that simply makes socketProperties protected instead of public - this does not seem to have negative consequences at least in the environment I tested it in (Tomcat 8.0.53 as shipped in SUSE Linux Enterprise Server 12 SP3). Patch is attached to this bug. I do not really know Tomcat (and not even JMX, to be honest) to know if this patch is really acceptable or if it's really oversimplified, as this is my first contribution! [1] https://prometheus.io/ [2] https://github.com/prometheus/jmx_exporter [3] https://github.com/prometheus/jmx_exporter/issues/265#issuecomment-383614412
8.0.x is no longer supported. If you made a similar change in 8.5.x I'm fairly sure you'd just hit the next unserializable attribute. Changing visibility is not really the right solution as it just hides the information and there is a lot of valuable data in that attribute. Making the attribute Serializable is one option but that requires that either the class is present on the JMX client's class path or RMI is enabled to load the class remotely. In this instance, making SocketProperties Serializable will just move the problem as a bunch of its attributes will then fail. Creating a separate MBean for SocketProperties has the same issue. Similar issues exist throughout the Tomcat MBeans. Looking at your config, am I correct in thinking you are just interested in the ThreadPool MBeans?
The root cause is that the Endpoint (ThreadPool) MBeans are generated via introspection so it is non-trivial to filter out unwanted attributes and/or operations. If an mbeans-descriptors.xml was used, the filtering would be trivial as the entries could just be removed. There are a lot of attributes so manually generating the content for an mbeans-descriptors.xml file would be time consuming. I've created a tool to extract this data using the introspection code as a starting point: https://github.com/markt-asf/tomcat-bugs/blob/master/src/java/org/apache/tomcat/MBeanDump.java That allowed me to easily create an mbeans-descriptors.xml file for the Endpoints which I then edited to remove the broken attributes and operations. I also added a SWocketProperties Mbean although that looks like it would benefit from some refactoring. Fixed in: - trunk for 9.0.13 onwards - 8.5.x for 8.5.35 onwards
Thanks Mark for the very fast response and root cause fix for this bug!