Bug 59604

Summary: Invalid url-pattern in servlet mapping on s390x
Product: Tomcat 8 Reporter: Dave <davec>
Component: CatalinaAssignee: Tomcat Developers Mailing List <dev>
Status: RESOLVED FIXED    
Severity: major    
Priority: P2    
Version: 8.0.33   
Target Milestone: ----   
Hardware: Other   
OS: other   
Attachments: log file
modified web.xml for ROOT
log file with only ROOT app
conf/web.xml (did not make any changes)
log file with nothing in webapps/
minimum /conf/web.xml
"bare bones" conf/web.xml
log file with bare bones conf/web.xml

Description Dave 2016-05-20 00:54:22 UTC
Created attachment 33854 [details]
log file

Tomcat 8.0.33 was deployed as-is on USS (Unix Systems Service)under z/OS V2R2.

Default apps failed to start due to "Invalid <url-pattern>" errors.

Somehow it thinks there are invalid characters in web.xml even though nothing was modified.
Comment 1 Dave 2016-05-20 01:04:25 UTC
Tomcat 7.0.67 could be deployed with no problem in the same environment.
Comment 2 Mark Thomas 2016-05-20 09:37:14 UTC
That looks like some form of encoding problem. Tomcat 7.0.x uses ISO-8859-1 for web.xml whereas Tomcat 8.0.x uses UTF-8.

Given that the contents of the files are correct for the given encoding this looks like a JVM issue on your OS.

As far as I am aware, none of the Tomcat committers have access to a z/OS system for testing. You are going to have to debug this one on your own unless you can provide one of us with access to a test system.

Interesting... conf/web.xml is still using ISO-8859-1. That should be UTF-8. Although I don't think it will help in your case it is worth changing to see if it does.
Comment 3 Konstantin Kolinko 2016-05-20 11:33:36 UTC
Note that it succeeded in parsing XML markup (the tag of '<url-pattern>'), but contents of the tag is parsed as garbage. This is rather odd. Those patterns are English as well.


Maybe those web.xml files were processed, overwrittem, broken by some tool?

Note that server.xml was processed correctly, otherwise the server wouldn't start. Though a difference is that server.xml uses attributes to configure values, while web.xml uses plain text content wrapped by elements.


Maybe try a different XML parser implementation. E.g. place a recent copy of Apache Xerces-J into "endorsed" directory. From your logs its location is:

-Djava.endorsed.dirs=/u/aes/apache-tomcat-8.0.33/endorsed



Some (unrelated) oddity in the logs:

VersionLoggerListener.log Command line argument: -Djava.class.path=.

VersionLoggerListener.log Command line argument: -Djava.class.path=/u/aes/apache-tomcat-8.0.33/bin/bootstrap.jar:/u/aes/apache-tomcat-8.0.33/bin/tomcat-juli.jar

Where the first incorrect value of "-Djava.class.path" argument comes from?



Note that such questions are normally asked on the users mailing list. You need help from fellow users to help find the root cause here.
Comment 4 Dave 2016-05-20 22:32:17 UTC
Both versions (8.0.33 and 7.0.67) were deployed as-is without any changes, using the same JAVA_HOME.  7.0.67 runs OK without any problem.

Note: 
z/OS shell is EBCDIC-based but it can support ASCII shell scripts (e.g., Tomcat's startup.sh, etc.).  Below is my "init" script to set up the automatic conversion. 

#!/bin/sh                                           
export CATALINA_HOME=/u/aes/apache-tomcat-8.0.33    
export JAVA_HOME=/usr/lpp/java/J8.0_64              
export _BPXK_AUTOCVT=ON                             
chtag -t -c ISO8859-1 $CATALINA_HOME/bin/*.sh
Comment 5 Mark Thomas 2016-05-22 18:17:44 UTC
Unless you are able to provide a committer with access to a suitable test system, you are going to have to debug this yourself.
Comment 6 Dave 2016-05-23 19:08:50 UTC
We might be able to provide access to z/OS shell.  Meanwhile, what kind of diagnostic data I can collect for debugging purpose?
Comment 7 Mark Thomas 2016-05-24 19:52:06 UTC
Anything that narrows down what is going on is useful.

My sugegstion is to try the following but keep in mind that you may have to modify this as you go as you find out more information.

Strip Tomcat down to a single webapp that exhibits the problem (e.g ROOT).

Try different charsets in the XML prolog for conf/web.xml. UTF-8 and ISO-8859-1 as a minimum.

Remove content from conf/web.xml until you have the smallest possible file that triggers the error.

Try writing a simple Java program to parse that file. Does that work?
Comment 8 Christopher Schultz 2016-05-24 20:54:08 UTC
(In reply to Dave from comment #6)
> We might be able to provide access to z/OS shell.  Meanwhile, what kind of
> diagnostic data I can collect for debugging purpose?

Can you perform an MD5 signature of the conf/web.xml file so we can be sure it's identical to the stock web.xml? Is it possible to get a byte-for-byte copy of the file off that system somewhere we can see it?

Do you have any other XML-related utilities on that system that can be run against the file to check for formatting, content, etc.? I'm thinking something like 'xmllint' which is popular on *NIX systems (I recognize that z/OS is different).

Maybe even an "od"-style byte dump copy-pasted into the comments, here?

(Actually, a better place for this thread would be the users mailing list since it's not entirely clear that there is a bug here, yet.)
Comment 9 Dave 2016-05-25 00:47:22 UTC
Created attachment 33884 [details]
modified web.xml for ROOT

I stripped out the comments and the following:
Comment 10 Dave 2016-05-25 00:48:41 UTC
Created attachment 33885 [details]
log file with only ROOT app
Comment 11 Dave 2016-05-25 00:56:32 UTC
... continue with my last comment:

I removed all the apps except ROOT and modified its web.xml by stripping off the comments and the following:

<display-name>Welcome to Tomcat</display-name>
  <description>
     Welcome to Tomcat
  </description>


I'm still getting the same parsing error even though there are no servlet mappings in this bare minimum web.xml.  It looks like an encoding issue, but I guess the problem is not with the file contents but how it's being read.
Comment 12 Mark Thomas 2016-05-25 06:20:37 UTC
(In reply to Dave from comment #11)

> I'm still getting the same parsing error even though there are no servlet
> mappings in this bare minimum web.xml.  It looks like an encoding issue, but
> I guess the problem is not with the file contents but how it's being read.

You need to do the same to the global web.xml in conf/web.xml
Comment 13 Dave 2016-05-26 01:23:34 UTC
Created attachment 33892 [details]
conf/web.xml (did not make any changes)
Comment 14 Dave 2016-05-26 01:24:11 UTC
Created attachment 33893 [details]
log file with nothing in webapps/
Comment 15 Dave 2016-05-26 01:24:52 UTC
If I remove everything in webapps/ then there are no errors in the startup log.
Comment 16 Mark Thomas 2016-05-26 08:51:09 UTC
Please try the following:

Clean Tomcat install.
Confirm problem exists.
Remove apps one by one until only ROOT is left.
Confirm problem still exists as each app is removed.
Remove ROOT/WEB-INF/web.xml
Confirm problem still exists.
Remove content from conf/web.xml until you have the minimal conf/web.xml that triggers the problem.

I'm expecting a minimal conf/web.xml with a single Servlet definition and associated Servlet mapping to trigger this issue.

Experiment with different encodings for the XML prolog for conf/web.xml. Test UTF-8 and ISO-8859-1 as a minimum.

Report your findings.
Comment 17 Dave 2016-05-26 20:23:51 UTC
The problem still exists after each step:

1. as each app is removed
2. with only ROOT app
3. after removing ROOT/WEB-INF/web.xml
4. with a minimum conf/web.xml

I will upload the minimum conf/web.xml and catalina.out
Comment 18 Dave 2016-05-26 20:25:00 UTC
Created attachment 33895 [details]
minimum /conf/web.xml
Comment 19 Dave 2016-05-26 20:32:02 UTC
The problem still exits even with a "bare bones" conf/web.xml
Comment 20 Dave 2016-05-26 20:32:50 UTC
Created attachment 33896 [details]
"bare bones" conf/web.xml
Comment 21 Dave 2016-05-26 20:34:33 UTC
Created attachment 33897 [details]
log file with bare bones conf/web.xml
Comment 22 Dave 2016-05-26 20:35:18 UTC
By the way, switching between UTF-8 and ISO-8859-1 made no difference.
Comment 23 Mark Thomas 2016-05-31 13:26:19 UTC
Comment on attachment 33897 [details]
log file with bare bones conf/web.xml

Interesting. The exception is happening in the WebSocket ServletContainerInitializer. It looks to be complaining that the URL pattern is an empty string. However, the URL pattern is hard-coded to "/*".

There is no more debugging logging we can enable that will shed any light on this. Some custom patches to add additional logging are going to be required. While I can provide suitable patches, shell access to a test environment would enable this to be investigated much faster.
Comment 24 Dave 2016-05-31 18:54:29 UTC
OK.  Mark, I will send the z/OS shell access info to you.
Comment 25 Mark Thomas 2016-06-01 11:28:56 UTC
Thanks for the shell access. It really speed up debugging this.

The bug has been fixed in:
- trunk for 9.0.0.M7 onwards
- 8.5.x for 8.5.3 onwards
- 8.0.x for 8.0.36 onwards

7.0.x and earlier were not affected.
Comment 26 Dave 2016-06-01 19:43:45 UTC
Thanks for the speedy fix.