This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 235723 - starting GF on port which is already used results into neverending GF start
Summary: starting GF on port which is already used results into neverending GF start
Status: RESOLVED FIXED
Alias: None
Product: serverplugins
Classification: Unclassified
Component: GlassFish (show other bugs)
Version: 7.4
Hardware: PC Linux
: P2 normal (vote)
Assignee: TomasKraus
URL:
Keywords: 7.4_HR_FIX
Depends on:
Blocks:
 
Reported: 2013-09-10 02:22 UTC by David Konecny
Modified: 2013-09-20 13:13 UTC (History)
2 users (show)

See Also:
Issue Type: DEFECT
Exception Reporter:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description David Konecny 2013-09-10 02:22:17 UTC
I had a Tomcat started on 8080 port and in a serarate IDE instance I started GF which correctly resulted into:

INFO:   GlassFish Server Open Source Edition  4.0  (89) startup time : Felix (16,235ms), startup services(22,579ms), total(38,814ms)
SEVERE:   Shutting down server due to startup exception
java.net.BindException: Address already in use
	at sun.nio.ch.Net.bind0(Native Method)
	at sun.nio.ch.Net.bind(Net.java:444)
	at sun.nio.ch.Net.bind(Net.java:436)
	[...]
INFO:   Server shutdown initiated
INFO:   Registered com.sun.enterprise.glassfish.bootstrap.osgi.EmbeddedOSGiGlassFishImpl@2c79c6d8 as OSGi service registration: org.apache.felix.framework.ServiceRegistrationImpl@34a500c8.
INFO:   Initiating Jersey application, version Jersey: 2.0 2013-05-03 14:50:15...
INFO:   Unregistered com.sun.enterprise.glassfish.bootstrap.osgi.EmbeddedOSGiGlassFishImpl@2c79c6d8 from service registry.

but start task seems to never finish. If I kill the task then GF server stays in "hourglass" mode in Services tab; and Start, Stop, Restart actions are disabled.
Comment 1 David Konecny 2013-09-10 02:26:21 UTC
Restarting IDE resulted into

Starting GlassFish Server
Could not start GlassFish Server: Administrator port is occupied by V3 cannot process this command at this time, please wait
/tmp/EnterpriseApplication2/nbproject/build-impl.xml:307: Deployment error: Could not start GlassFish Server: Administrator port is occupied by V3 cannot process this command at this time, please wait
See the server log for details.
	at org.netbeans.modules.j2ee.ant.Deploy.execute(Deploy.java:137)
	at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
	[...]

because GF process was still running adn I had to kill it first.
Comment 2 TomasKraus 2013-09-10 11:09:36 UTC
Refresh should fix this. But maybe it's broken.
Comment 3 TomasKraus 2013-09-10 14:26:48 UTC
Refresh is working fine, problem is in StartTask timeout which is 5 minutes. Also server process remains running even when port is occupied.
After 5 minutes "hourglass" mode will be removed but server process is still there.

I can't lower StartTask timeout because of servers with huge applications where startup may take long. Because server port is occupied and process is still running there is no simple way to determine that something went wrong.

I'll activate code that I made in advance for next release. This allows user to terminate StartTask thread and kill running server process. It seems to work fine. There is just some delay before things settle down so you won't see start context menu active immediately after kill.
Comment 4 TomasKraus 2013-09-10 14:35:56 UTC
Checked into web-main:
----------------------
changeset:   263380:e3176e7d9dd9
summary:     #235723 - Added support to kill startup task and running
             server process
Comment 5 TomasKraus 2013-09-10 15:06:49 UTC
Pushed into web-main. Please let me know if it's ok for you now.
Comment 6 Quality Engineering 2013-09-11 01:41:25 UTC
Integrated into 'main-silver', will be available in build *201309110001* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)

Changeset: http://hg.netbeans.org/main-silver/rev/e3176e7d9dd9
User: Tomas Kraus <TomasKraus@netbeans.org>
Log: #235723 - Added support to kill startup task and running server process
Comment 7 Stepan Zebra 2013-09-11 11:24:21 UTC
(In reply to Quality Engineering from comment #6)
> Integrated into 'main-silver', will be available in build *201309110001* on
> http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
> 
> Changeset: http://hg.netbeans.org/main-silver/rev/e3176e7d9dd9
> User: Tomas Kraus <TomasKraus@netbeans.org>
> Log: #235723 - Added support to kill startup task and running server process

verified by QE
Comment 8 David Konecny 2013-09-11 21:06:39 UTC
I'm not sure what's the best solution here Tomas. Few unsorted comments:

* I might be wrong but I have feeling that in the past "port already used" aborted server startup immediately

* "Kill" seems to me quite "strong" action. Why not just enable "Stop" action and under the hood implement it as either Stop or Kill. This way "Stop" action in the GF output console would get enabled too I guess.

* after 5 minutes timeout there was a message box but GF node still has Start, Stop disabled. Refresh does not change it. The hourglass badge is gone though. But process in output window still runs because tab is bold. I think that timeout should simply kill the process.

Considering GF is inhouse project I would assume it would come with a better tooling story and we would quickly get a response what's happening during GF startup and if it failed or what.
Comment 9 TomasKraus 2013-09-11 23:16:11 UTC
There is check for admin port (usually 4848) in start task but HTTP listeners ports are not being checked. I can add such a tests. It may prevent this to happen.

I added kill feature because server process keeps running after HTTP listener failure. Also there was request to add kill feature in one bug long time ago so I already had this piece of code ready. It was just not available in node context menu. Think it makes sense to keep this feature for situations when server process started from NetBeans hangs.

I do not want to add process kill even after 5 minutes timeout, because we had issues with long server startup with old versions where process had been killed after timeout. Current start task will warn user about server process startup problems but plugin will keep monitoring it and mark it as running even later after comming up. Having kill in context menu allows user to do it manually.

All tooling is based on server asadmin interface which is available on admin port (usually 4848). But this port does not come up after HTTP listener bind issue. 
As you can see in the log, server starts shutdown but it never finishes and process ends up in zome kind of 'zombie' state. Unfortunately this is in collision with long server start up time and manual kill is the only solution that came to my mind.

Unfortunatel refresh won't help even after timeout because server zombie process
is still running.

So the only remaining thing I can do is to add http listeners availability check together with existing admin port check. Reopening to implement this.
Comment 10 Petr Hejl 2013-09-12 07:22:02 UTC
(In reply to TomasKraus from comment #9)
> There is check for admin port (usually 4848) in start task but HTTP
> listeners ports are not being checked. I can add such a tests. It may
> prevent this to happen.
I guess that would be just enough to solve this issue.

> 
> I added kill feature because server process keeps running after HTTP
> listener failure.
Feature must not be added after feature freeze. Considering we are even in code freeze this is bad idea.

> Also there was request to add kill feature in one bug long
> time ago so I already had this piece of code ready. It was just not
> available in node context menu. Think it makes sense to keep this feature
> for situations when server process started from NetBeans hangs.
Yep though not for 7.4.

> 
> I do not want to add process kill even after 5 minutes timeout, because we
> had issues with long server startup with old versions where process had been
> killed after timeout. Current start task will warn user about server process
> startup problems but plugin will keep monitoring it and mark it as running
> even later after comming up. Having kill in context menu allows user to do
> it manually.
> 
> All tooling is based on server asadmin interface which is available on admin
> port (usually 4848). But this port does not come up after HTTP listener bind
> issue. 
> As you can see in the log, server starts shutdown but it never finishes and
> process ends up in zome kind of 'zombie' state. Unfortunately this is in
> collision with long server start up time and manual kill is the only
> solution that came to my mind.
> 
> Unfortunatel refresh won't help even after timeout because server zombie
> process
> is still running.
> 
> So the only remaining thing I can do is to add http listeners availability
> check together with existing admin port check. Reopening to implement this.
Also remove the feature for now or do not transplant it. To be consistent with Tomcat the action should be labeled "Terminate".

Also note the Process.destroy() might keep children processes running if any.
Comment 11 TomasKraus 2013-09-12 09:37:01 UTC
Yes, I won't put kill changeset into 7.4. Port check should be enough, agree.
Comment 12 TomasKraus 2013-09-16 14:09:35 UTC
Checked into web-main:
----------------------
changeset:   263606:8a673279b136
tag:         tip
user:        Tomas Kraus <TomasKraus@netbeans.org>
date:        Mon Sep 16 16:06:55 2013 +0200
summary:     #235723 - Added simple HTTP port availability check before start.
             Fix to be backported into 7.4

This is simple fix that can go into 7.4. I made it in a separate changeset.

Next changeset will move those port checks into GF Tooling library to allow verification of all ports stored in domain.xml. But it will be more complex fix only for trunk.
Comment 13 TomasKraus 2013-09-16 16:53:40 UTC
Marking as fixed to finish 7.4 backport. Will reopen later for trunk fix (or maybe will create one more bug to cover it).
Comment 14 Stepan Zebra 2013-09-20 12:12:42 UTC
(In reply to TomasKraus from comment #12)
> Checked into web-main:
> ----------------------
> changeset:   263606:8a673279b136
> tag:         tip
> user:        Tomas Kraus <TomasKraus@netbeans.org>
> date:        Mon Sep 16 16:06:55 2013 +0200
> summary:     #235723 - Added simple HTTP port availability check before
> start.
>              Fix to be backported into 7.4
> 
> This is simple fix that can go into 7.4. I made it in a separate changeset.
> 
> Next changeset will move those port checks into GF Tooling library to allow
> verification of all ports stored in domain.xml. But it will be more complex
> fix only for trunk.

verified by QE
Comment 15 TomasKraus 2013-09-20 12:35:32 UTC
7.4 release branch changeset:
-----------------------------
changeset:   272641:84cc6f9677e5
branch:      release74
summary:     #235723 - Added simple HTTP port availability check before start