This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 189774 - exception and timeout using deploy on save to weblogic
Summary: exception and timeout using deploy on save to weblogic
Status: RESOLVED FIXED
Alias: None
Product: serverplugins
Classification: Unclassified
Component: WebLogic (show other bugs)
Version: 7.0
Hardware: PC All
: P2 normal (vote)
Assignee: Petr Hejl
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2010-08-23 13:36 UTC by Kenneth Ganfield
Modified: 2010-09-01 03:47 UTC (History)
4 users (show)

See Also:
Issue Type: DEFECT
Exception Reporter:


Attachments
For the record, attaching the deployment exception in the server log (18.19 KB, text/plain)
2010-08-23 21:40 UTC, Petr Jiricka
Details
IDE thread dump (17.63 KB, text/plain)
2010-08-24 20:33 UTC, Petr Jiricka
Details
Patch in progress (5.79 KB, text/plain)
2010-08-24 21:35 UTC, Petr Jiricka
Details
The patch I have so far (7.61 KB, patch)
2010-08-26 14:11 UTC, Petr Hejl
Details | Diff
possible fix/workaround (14.74 KB, patch)
2010-08-26 21:21 UTC, Petr Hejl
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Kenneth Ganfield 2010-08-23 13:36:00 UTC
Product Version: NetBeans IDE Dev (Build 201008230001)
Weblogic 10.3.3

1. create web app using entity>db and selecting jaxb in wizard
2. create jsf>entity
3. run
4. create webservice

When you create the webservice, the status bar appears and says "Deploying XXX" for several minutes until it timesout. There is no way to stop the deployment. The Output displays the following problem:


<Aug 23, 2010 3:21:40 PM CEST> <Error> <HTTP> <BEA-101216> <Servlet: "wservice.CustomerService" failed to preload on startup in Web application: "web".
com.sun.xml.ws.model.RuntimeModelerException: The web service defined by the class wservice.CustomerService does not contain any valid WebMethods.
        at com.sun.xml.ws.model.RuntimeModeler.buildRuntimeModel(RuntimeModeler.java:263)
        at com.sun.xml.ws.server.EndpointFactory.createSEIModel(EndpointFactory.java:363)
        at com.sun.xml.ws.server.EndpointFactory.createEndpoint(EndpointFactory.java:202)
        at com.sun.xml.ws.api.server.WSEndpoint.create(WSEndpoint.java:514)
        at com.sun.xml.ws.api.server.WSEndpoint.create(WSEndpoint.java:557)
        Truncated. see log file for complete stacktrace

The workaround seems to be to deactivate deploy on save for projects deployed to weblogic. deploy on save is currently active by default when you create the project
Comment 1 Kenneth Ganfield 2010-08-23 13:45:21 UTC
after 10 minutes the deployment times out with the following info:

<Aug 23, 2010 3:21:35 PM CEST> <Info> <J2EE Deployment SPI> <BEA-260121> <Initiating redeploy operation for application, WebApplication3 [archive: null], to AdminServer .> 
INFO [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Complete indexing of 25 binary roots took: 1805 ms
INFO [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Resolving dependencies took: 0 ms
INFO [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Complete indexing of 31 binary roots took: 1378 ms
INFO [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Complete indexing of 0 source roots took: 0 ms (New or modified files: 0, Deleted files: 0) [Adding listeners took: 0 ms]
SEVERE [org.openide.util.Exceptions]
org.netbeans.modules.j2ee.deployment.impl.ServerException: Deployment timeout has exceeded.
	at org.netbeans.modules.j2ee.deployment.impl.TargetServer.trackDeployProgressObject(TargetServer.java:926)
	at org.netbeans.modules.j2ee.deployment.impl.TargetServer.reloadArtifacts(TargetServer.java:860)
	at org.netbeans.modules.j2ee.deployment.impl.TargetServer.notifyArtifactsUpdated(TargetServer.java:837)
	at org.netbeans.modules.j2ee.deployment.impl.DeployOnSaveManager$DeployTask.notifyServer(DeployOnSaveManager.java:423)
	at org.netbeans.modules.j2ee.deployment.impl.DeployOnSaveManager$DeployTask.run(DeployOnSaveManager.java:357)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[catch] at java.lang.Thread.run(Thread.java:637)
INFO [org.netbeans.modules.j2ee.deployment.impl.TargetServer]: On save deployment failed
Comment 2 Petr Jiricka 2010-08-23 14:05:23 UTC
So when you say the deployment times out after several minutes, does the output also show the message after several minutes? Or does it appear sooner?

It sounds like the plugin is not able to recover from unsuccessful redeployment. The correct behavior would be to gracefully and immediately fail when the invalid web service is autodeployed in the background, and then when the error in the code is corrected, redeploy it again.
Comment 3 Petr Jiricka 2010-08-23 17:06:00 UTC
Weird. At first I was able to reproduce this, but now I am getting graceful behavior - the progress bar is stopped just a second after the error message appears in the log file. 

Looking at the code, it sounds like it should be possible to decrease the timeout by adding the following in <userdir>/config/J2EE/InstalledServers/.nbattrs, under the <fileobject> element corresponding to the WebLogic instance:
<attr name="deploymentTimeout" stringvalue="10"/>

As a workaround for this, I am thinking, would it be a good idea to always create a web service with a sample operation? Cc'ing Milan.
Comment 4 Petr Jiricka 2010-08-23 21:18:31 UTC
Now I can reproduce (when not run under debugger). The problem is that the WebLogic deployment implementation (weblogic.deploy.api.spi.deploy.WebLogicDeploymentManagerImpl, 
weblogic.deploy.api.spi.status.ProgressObjectImpl, 
weblogic.deploy.api.spi.status.DeploymentStatusImpl) sometimes returns deployment status StateType.RELEASED (e.g. in o.n.m.j2ee.deployment.impl.ProgressObjectUtil#trackProgressObject), and our deployment state handling code does not deal with that correctly. Neither GlassFish nor Tomcat use this state, so this was not a problem so far. StateType.RELEASED is incorrectly treated by our code as as "the action is still in progress", while in fact it already finished. We may need to review all the places where deployment status is handled.
Comment 5 Petr Jiricka 2010-08-23 21:40:54 UTC
Created attachment 101616 [details]
For the record, attaching the deployment exception in the server log
Comment 6 Milan Kuchtiak 2010-08-24 08:20:14 UTC
> As a workaround for this, I am thinking, would it be a good idea to always
create a web service with a sample operation? Cc'ing Milan.

Good idea. I'll fix that.
Comment 7 Milan Kuchtiak 2010-08-24 09:55:05 UTC
> As a workaround for this, I am thinking, would it be a good idea to always
create a web service with a sample operation? Cc'ing Milan.

See the fix:
http://hg.netbeans.org/main/rev/e0765fd7a9af
Comment 8 Milan Kuchtiak 2010-08-24 13:16:18 UTC
Sorry, the changeset is:
http://hg.netbeans.org/web-main/rev/e0765fd7a9af
Comment 9 Petr Jiricka 2010-08-24 14:02:36 UTC
FYI, simpler steps to reproduce:

1. Create EE 5 Web application with WebLogic target server
2. Run
3. Create web service
Comment 10 Petr Jiricka 2010-08-24 16:13:13 UTC
Ken is still hitting problems and weird deployment errors; I wonder if they could also be related to issue 178597.
Comment 11 Petr Jiricka 2010-08-24 20:33:15 UTC
Created attachment 101647 [details]
IDE thread dump

Milan's fix did not help. Now the web service is deployed successfully after creation (no exceptions in the server log), but the deployment progress bar is still stuck. Attaching a thread dump of the IDE while the progress bar is running.
Comment 12 Petr Jiricka 2010-08-24 21:35:16 UTC
Created attachment 101648 [details]
Patch in progress
Comment 13 Quality Engineering 2010-08-25 03:26:06 UTC
Integrated into 'main-golden', will be available in build *201008250001* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/e0765fd7a9af
User: mkuchtiak@netbeans.org
Log: #189774: created sample method in WS template
Comment 14 Petr Jiricka 2010-08-25 15:35:26 UTC
Hotfix (not proper fix): http://hg.netbeans.org/web-main/rev/c1a706a07871
Comment 15 Martin Schovanek 2010-08-25 16:57:50 UTC
Verified at trunk. Cannot reproduce the 'Deployment timeout' exception with the latest fix.
Comment 16 Petr Jiricka 2010-08-25 17:22:28 UTC
When trying out the original scenario with JSF from Entity, I am still getting exceptions such as this (in the server log):

Caused By: Exception [EclipseLink-28017] (Eclipse Persistence Services - 2.0.2.v20100323-r6872): org.eclipse.persistence.exceptions.EntityManagerSetupException
Exception Description: Unable to predeploy PersistenceUnit [WebApplication159PU] in invalid state [DeployFailed]
	at org.eclipse.persistence.exceptions.EntityManagerSetupException.cannotPredeploy(EntityManagerSetupException.java:202)
	at org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.predeploy(EntityManagerSetupImpl.java:823)
	at org.eclipse.persistence.jpa.PersistenceProvider.createContainerEntityManagerFactory(PersistenceProvider.java:219)

So it still does not work completely. The workaround for this is to shut down the server and deploy from scratch (undeploy and deploy without restarting the server does not work).
Comment 17 Quality Engineering 2010-08-26 03:22:49 UTC
Integrated into 'main-golden', will be available in build *201008260001* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/c1a706a07871
User: pjiricka@netbeans.org
Log: #189774 - hotfix to get rid of deployment failures and neverending progress bar; this fix disables redeployment and always does full deploy (after save), and it also adds treatment of the StateType.RELEASED deployment state, so it is considered successful.
Comment 18 Petr Hejl 2010-08-26 08:51:04 UTC
The hotfix looks ok. I'm not sure however I'll keep the RELEASED as successfull in ProgressObjectUtil once the issue is fixed.
Comment 19 Petr Hejl 2010-08-26 14:08:04 UTC
The issue is that DeploymentManager (the JSR88 one) is closed too early. However there is couple of other broken things in the WL implementation. 

- When I attach the listener on ProgressObject to release the manager when the job is done the release is performed immediately never delivering COMPLETED or FAILED event (at least sometimes).
- When I attach the listener on ProgressObject to release the manager when done and schedule the release to different thread (properly synchronizing it) there is warning about WLContext being closed from different thread.
- The listener registration on ProgressObject is most likely broken because when the listener is added at the same time the event is dispatched it leads to ConcurrentModificationException thrown by iterator in dispatching code:

java.util.ConcurrentModificationException
	at java.util.HashMap$HashIterator.nextEntry(HashMap.java:810)
	at java.util.HashMap$KeyIterator.next(HashMap.java:845)
	at weblogic.deploy.api.spi.status.ProgressObjectImpl.broadcastReport(ProgressObjectImpl.java:392)
	at weblogic.deploy.api.spi.status.ProgressObjectImpl.reportEvent(ProgressObjectImpl.java:347)
	at weblogic.deploy.api.spi.deploy.internal.ServerConnectionImpl$TaskPoller.poll(ServerConnectionImpl.java:1279)
	at weblogic.deploy.api.spi.deploy.internal.ServerConnectionImpl$TaskPoller.run(ServerConnectionImpl.java:1244)

Too bad :/
Comment 20 Petr Hejl 2010-08-26 14:11:17 UTC
Created attachment 101697 [details]
The patch I have so far
Comment 21 Petr Hejl 2010-08-26 21:21:30 UTC
Created attachment 101704 [details]
possible fix/workaround

Now I really think the main problem is ConcurrentModificationException. The only way to work around it is to actively poll the state. This of course may miss some updates of state, however it is not really important as we really care about the final state and final message. At least in this particular case. Perhaps we could use JMX redeploy bean in future however it is deprecated.

The plugin code is getting bit messy. Luckily these workarounds are isolated and hopefully we will have some time to do the things in a cleaner way.

The good thing is that the redeployment is now faster (cached DM) - at least for me :)

Attaching patch for now. I'll do some more testing tomorrow.
Comment 22 Petr Hejl 2010-08-27 07:42:31 UTC
Patch pushed to web-main 51a7509262b8. The issue in WL was reported via email.

Please verify DoS works correctly now.
Comment 23 Quality Engineering 2010-08-28 03:45:26 UTC
Integrated into 'main-golden', will be available in build *201008280001* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/51a7509262b8
User: Petr Hejl <phejl@netbeans.org>
Log: #189774 exception and timeout using deploy on save to weblogic
Comment 24 David Konecny 2010-09-01 03:47:27 UTC
Issue 190053 - potential side effect of this fix?