Bug 34633 - ReplaceRegExp periodically errors out with "Couldn't rename temporary file"
ReplaceRegExp periodically errors out with "Couldn't rename temporary file"
Status: REOPENED
Product: Ant
Classification: Unclassified
Component: Optional Tasks
1.7.0
All All
: P3 normal with 8 votes (vote)
: 1.8.0
Assigned To: Ant Notifications List
:
Depends on:
Blocks:
  Show dependency tree
 
Reported: 2005-04-26 20:20 UTC by nlitt
Modified: 2013-02-14 16:16 UTC (History)
2 users (show)



Attachments
Improve Error Message for ReplaceRegExp Rename Failure (652 bytes, patch)
2008-04-02 11:46 UTC, Eric Olsen
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description nlitt 2005-04-26 20:20:23 UTC
Recently I encountered an specific laptop on which the replaceregexp task 
intermittently fails with errors like: 
 
 - Couldn't rename temporary file C:\<builddir>\rep1253479585.tmp
 - Couldn't rename temporary file C:\<builddir>\replace690408058.txt

I see that this error message is generated by line 431 of the ReplaceRegExp 
task. The task creates a temp file, manipulates it, then overwrites the 
original file with the temp file. The "Couldn't rename temp file" error 
signifies a failure to mv the tempfile to the original.  

The user is running as root, so it's not a permissions issue. And replaceregexp 
works most of the time, but certain calls always seem to crash with this error. 
So a single target might call replaceregexp successfully four or five times, 
then error out.  

Googling the error, I found one other person who has encountered it and was 
also stumped:
http://64.233.161.104/search?q=cache:SyAlUcJaA7cJ:www.mail-archive.com/user%
40ant.apache.org/msg01075.html+Ant+%22Couldn%27t+rename+temporary+file%22&hl=en

Searching through ASF Bugzilla, I could not find any bug that appears to be 
related.
Comment 1 Heinz van Pee 2005-06-13 10:24:38 UTC
I have the same problem with Apache Ant version 1.6.1 compiled on February 12 2004.
Ant is executed via Eclipse
Im Running Windows XP with service pack 2.
Comment 2 Heinz van Pee 2005-06-14 09:59:52 UTC
I also encountered this effect when running ant 1.6.5 from console.
Comment 3 Heinz van Pee 2005-06-14 10:01:22 UTC
Forgot to mention the java version, using j2sdk1.4.2_04
Comment 4 Heinz van Pee 2005-06-14 11:38:49 UTC
i have updated to j2sdk1.4.2_08  
and disabled indexing for fast search, 
maybe update to 1.4.2_08 is enough, all run since then proceeded without errors.
Comment 5 Heinz van Pee 2005-06-14 11:44:49 UTC
(In reply to comment #4)
i was too fast, its still there occassionally.
Comment 6 Heinz van Pee 2005-06-14 12:20:49 UTC
It seems that if this behaviour occured once it occures on every run thereafter.
Comment 7 Heinz van Pee 2005-06-14 13:19:15 UTC
It seems to help if you delete all temporary files.
Comment 8 Mark Crocker 2006-05-04 21:49:39 UTC
There's a possibility that this is a file naming bug related to long vs. short
file names in Windows.  The error I got was:

  An error occurred processing file:
'C:\dev\LongFilenameExample\build\text\Example.java.html':
C:\dev\LongFilenameExample\build.xml:138: Couldn't rename temporary file
C:\DOCUME~1\mcrocker\LOCALS~1\Temp\replace1599386392.txt

Notice that the source file and build file names use the full file names, but
the temporary file uses the tilde vFAT naming style.

This error occurs on Windows XP Pro 2002 SP2, with ANT 1.6.5 under Java 
1.4.2_04 and 1.5.0_02 from the command line or inside of Eclipse 3.1.
Comment 9 csaona 2007-07-25 03:36:41 UTC
I'm also experiencing this problem with Ant 1.7.0 and Java 1.5.0_11 under
Windows XP SP2
Comment 10 Peter Reilly 2007-08-21 11:55:31 UTC
Marking as windows xp, but it applies to all windows versions.
Comment 11 Karol Bucek 2007-09-20 02:00:01 UTC
I think I have encountered it under Linux. I am using Fedora 7, ant (1.6.5) is
running under the hood while using maven (2.0.7) with appfuse project. My stack
trace is:
[INFO] ------------------------------------------------------------------------
[ERROR] FATAL ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Couldn't rename temporary file /tmp/replace688779408.txt
[INFO] ------------------------------------------------------------------------
[INFO] Trace
Couldn't rename temporary file /tmp/replace688779408.txt
        at
org.apache.tools.ant.taskdefs.optional.ReplaceRegExp.doReplace(ReplaceRegExp.java:431)
        at
org.apache.tools.ant.taskdefs.optional.ReplaceRegExp.execute(ReplaceRegExp.java:491)
        at
org.appfuse.mojo.installer.InstallSourceMojo.removeWarpathPlugin(InstallSourceMojo.java:609)
        at
org.appfuse.mojo.installer.InstallSourceMojo.execute(InstallSourceMojo.java:207)
        at
org.apache.maven.plugin.DefaultPluginManager.executeMojo(DefaultPluginManager.java:443)
        at
org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoals(DefaultLifecycleExecutor.java:539)
        at
org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeStandaloneGoal(DefaultLifecycleExecutor.java:493)
        at
org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoal(DefaultLifecycleExecutor.java:463)
        at
org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoalAndHandleFailures(DefaultLifecycleExecutor.java:311)
        at
org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeTaskSegments(DefaultLifecycleExecutor.java:278)
        at
org.apache.maven.lifecycle.DefaultLifecycleExecutor.execute(DefaultLifecycleExecutor.java:143)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:334)
        at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:125)
        at org.apache.maven.cli.MavenCli.main(MavenCli.java:280)

It seems I am not alone with this issue:

http://www.mail-archive.com/users@appfuse.dev.java.net/msg06914.html
http://www.nabble.com/appfuse-and-SOAP-t4035996s2369.html

Comment 12 Martyn Hiemstra 2007-12-13 03:29:57 UTC
I can now confirm that this a serious Java bug. 

I discovered this by doing the following:

I downloaded the ant source and I discovered that the original file is copied to
a temp file and then that temp file is renamed to the original file. So in
short, the original file is copied, edited and then overriden. So I decided to
do some experimenting. I moved my source code from 'My Documents' directory
which is located under the Document and Settings directory and moved it into
c:\Project. Now the replaceregexp task works.

Apparently Java doesnt know to handle files like this:
C:\Documents and Settings\User\My
documents\Projects\MyApp\projects\dao\src\conf\hibernate.properties

I tried the following:
if (!new File("C:\\Documents and Settings\\User\\My
documents\\Projects\\MyApp\\projects\\dao\\src\\conf\\hibernate.properties").delete())
    System.out.println( "Could not delete" );
else
    System.out.println( "Could delete" );

As output I get the message 'Could delete' and the file is deleted. This makes
me beleive that there is something totally messed up in Java. Deleteing a file
in one instance doesnt work but manually creating a File object pointing to that
file and then deleting it will work.

This isnt a Ant issue but a Java issue.
Comment 13 Peter Reilly 2007-12-13 03:48:43 UTC
I would suggest that it may be a windows issue.
Things like this can happen with virus checkers. If a virus
checker is configured to check a file when a file is created/read,
and the file is short lived - for example a temp file created
with there replaceregexp, there exists a race condition where the
file cannot be deleted (while the virus checker is checking the file).
[This is a windows "feature" apparently added with WindowsNT - previously
one could delete files while other "processes" were reading them - corrupting
the file system]

Sometimes a virus checker is configured to only do "heavy" checking on
certain directories - (My documents for example) so you may be seeing this
for some directories and not others.
Comment 14 Martyn Hiemstra 2007-12-14 08:02:36 UTC
> I would suggest that it may be a windows issue.
> Things like this can happen with virus checkers. 

To test this, I disabled my virus scanner and retested it. Still the same error.
So it isnt a virus scanner that is causing the problem. It was a good suggestion
though.
Comment 15 David Jackman 2007-12-14 08:30:05 UTC
(In reply to comment #14)
> > I would suggest that it may be a windows issue.
> > Things like this can happen with virus checkers. 
> 
> To test this, I disabled my virus scanner and retested it. Still the same error.
> So it isnt a virus scanner that is causing the problem. It was a good suggestion
> though.

This sounds like a problem I was experiencing with files in a different
situation.  Virus scanning software was also the expected culprit, but wasn't. 
Instead, I tracked it down to the TSVNCache.exe process of TortoiseSVN.  I was
able to work around the problem by going to the Icon Overlays page of the
TortoiseSVN settings and under "Status Cache" choosing "Shell" instead of "Default".
Comment 16 Eric Olsen 2008-04-02 08:11:34 UTC
(In reply to comment #8)

This is not an issue with Windows long vs. short names.  I ran into the same error you did in the Documents and Settings Temp area with Ant 1.7.0, Windows Server 2003, and Java 1.4.2_13 (don't ask).  So, I switched the user's environment TEMP to point to C:\temp and got:

BUILD FAILED
K:\O_build_stable\RSA_Build\Build\build.xml:268: The following error occurred while executing this line: K:\O_build_stable\RSA_Build\Build\subprojects\buildsummary.xml:350: Couldn't rename temporary file C:\Temp\replace8787748.txt
Comment 17 Eric Olsen 2008-04-02 11:46:33 UTC
Created attachment 21771 [details]
Improve Error Message for ReplaceRegExp Rename Failure
Comment 18 Eric Olsen 2008-04-02 11:46:56 UTC
The problem (at least the one that I encountered) is definitely not a Java bug or even a 'real' bug in Ant.  Instead it is one of misdirection due to a bad error message.  It turns out that everyone seems to be focused on the temporary file (e.g., C:\<builddir>\replace690408058.txt) when the problem is with the original file due to the way the Ant FileUtils rename works in order to work across file systems:

  1 - delete original file if there
  2 - copy updated file (temp file) to original file name and location
  3 - delete updated file (temp file)

The diff patch that I have attached to org/apache/tools/ant/taskdefs/optional/ReplaceRegExp.java will help sort out the bad error message confusion.  In one of my tests the original output was:

  BUILD FAILED
  C:\AntTestDir\build.xml:32: Couldn't
  rename temporary file C:\DOCUME~1\eolsen\LOCALS~1\Temp\replace2143556994.txt

The new (correct) output is:

  BUILD FAILED
  C:\Documents and Settings\eolsen\My Documents\AntTestDir\build.xml:32: Failed to
  delete W:\test2.txt while trying to rename C:\DOCUME~1\eolsen\LOCALS~1\Temp\replace1613181649.txt

The real issue as you can now see from the improved error message was a failure to delete the original file (which was on a filesystem to which I only had read permissions).

To misquote Matt Inger:  "Read code, not error messages.  Error messages can lie."
Comment 19 Eric Olsen 2008-04-02 11:50:02 UTC
Updating versions and systems since the patch applies to all platforms.
Comment 20 Eric Olsen 2008-04-02 11:54:25 UTC
One more note -- the real problem was that ReplaceRegExp.java gobbled up the good error message from org.apache.tools.ant.util.FileUtils.rename() instead of passing it on up the chain to the build failure where we could see it.
Comment 21 Stefan Bodewig 2008-10-09 08:59:41 UTC
the fix for bug 45960 may have improved the situation.  For other cases (like when Ant simply cannot write to the file), svn revision 703192 will now include the original error information (as suggested by Eric).
Comment 22 Daniel Holmes 2011-04-13 11:28:26 UTC
I am still seeing this issue in Ant 1.8.1.  Would there be a way to possibly handle the exception with a retry of the failed command (in this case looks like the original file delete).  This seems to happen often enough in my windows build execution since I started using the ReplaceRegExp task.  Once every 50 or so uses, which based on my CI build, fails every fourth or fifth build execution.

See ant -v full stact trace below for the underlying cause in the ant FileUtils

BUILD FAILED
c:\hudson\workspace\main_windows\build.xml:298: The following error occurred while executing this line:
c:\hudson\workspace\main_windows\build.xml:140: The following error occurred while executing this line:
c:\hudson\workspace\main_windows\modules\build-module-common.xml:399: The following error occurred while executing this line:
c:\hudson\workspace\main_windows\modules\build-module-common.xml:532: Couldn't rename temporary file C:\WINDOWS\TEMP\replace7543532359339722099.txt
	at org.apache.tools.ant.taskdefs.optional.ReplaceRegExp.doReplace(ReplaceRegExp.java:480)
	at org.apache.tools.ant.taskdefs.optional.ReplaceRegExp.execute(ReplaceRegExp.java:536)
	at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
	at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
	at org.apache.tools.ant.Task.perform(Task.java:348)
	at org.apache.tools.ant.Target.execute(Target.java:390)
	at org.apache.tools.ant.Target.performTasks(Target.java:411)
... (Extra stack trace clipped) ...
Caused by: java.io.IOException: Failed to delete c:\hudson\workspace\main_windows\modules\dfm-cli\target\reports\javac\javac_stdout.txt while trying to rename C:\WINDOWS\Temp\replace7543532359339722099.txt
	at org.apache.tools.ant.util.FileUtils.rename(FileUtils.java:1239)
	at org.apache.tools.ant.taskdefs.optional.ReplaceRegExp.doReplace(ReplaceRegExp.java:474)
	... 72 more
Comment 23 gw1500se 2012-02-22 15:32:49 UTC
I know this is an old thread but I have it today. It is also not a Windows specific problem as I am running Mandriva 2010.2, ant 1.7.1 and I don't know what version of replaceregexp I have. This error seems to be like a rash on the internet with no one coming up with a solution or even a workaround. Some have said it goes way on its own but mine hasn't yet and I cannot find any way to get rid of it even temporarily. My project is at a dead standstill at this point. Will rebooting help any?
Comment 24 Stefan Bodewig 2012-02-22 16:43:49 UTC
(In reply to comment #23)
> I know this is an old thread but I have it today. It is also not a Windows
> specific problem as I am running Mandriva 2010.2, ant 1.7.1 and I don't know
> what version of replaceregexp I have.

Since replaceregexp is a built-in task it should be the same version as Ant itself.

Before tyring to dig any deeper, you should give a more recent version of Ant a try (1.8.2 is current with 1.8.3 being close).  Even if bug 45960 says it is about Windows, it also applies to other platforms in that a failed delete operation is retried once (on Windows there is an additional GC between the attempts to delete the file).
Comment 25 gw1500se 2012-02-22 18:40:18 UTC
1.8.2 is not yet a stable release for Mandriva. I am reluctant to try the cooker release if there is a workaround. I can survive for a while if there is some way to manually clear this problem even if temporarily.

As for the version, I had lots of trouble getting replaceregexp since it kept telling me it was an ant-contrib optional function. I had to install it separately which is why I thought it might be a different version. Whatever I did, when I installed the Mandriva ant rpm, replaceregex was not included.
Comment 26 Stefan Bodewig 2012-02-23 05:04:42 UTC
(In reply to comment #25)
> 1.8.2 is not yet a stable release for Mandriva. I am reluctant to try the
> cooker release if there is a workaround. I can survive for a while if there is
> some way to manually clear this problem even if temporarily.
> 
> As for the version, I had lots of trouble getting replaceregexp since it kept
> telling me it was an ant-contrib optional function. I had to install it
> separately which is why I thought it might be a different version. Whatever I
> did, when I installed the Mandriva ant rpm, replaceregex was not included.

Well, we don't have any influence on the RPMs (or any other distribution you don't download from one of the ASF's mirrors), I don't have any idea whether the Mandriva distribution has made any modifications.

In Ant 1.7.x the replaceregexp task has been in a jar called ant-nodeps.jar, with Ant 1.8.x we've merged those into ant.jar.

You should be able to download a binary release from Ant's download page, untar the tarball anywhere, set ANT_HOME to point to that location and start using it without messing with your OS installation.

I'm not aware of any workaround if the problem is that Java is holding on to a lock to the file it has just written even after it has been closed.
Comment 27 gw1500se 2012-02-23 15:28:57 UTC
I took your advice and installed 1.8. No joy, same error. However, I wonder if I needed to do some kind of clean up.
Comment 28 Stefan Bodewig 2012-02-23 15:52:46 UTC
At least 1.8.x should tell you which file it failed to delete.
Comment 29 gw1500se 2012-02-23 18:08:25 UTC
I've always had that, from replaceregexp:

Couldn't rename temporary file /tmp/replace5358820582224705130.txt

However, I've just discovered what I think is an important piece of information and, as an aside, may be related to what I thought might be a different bug. I have been running this build via a perl script. What I have found is everything works as expected when the script is run from the command line. When it fails with this error, it is running in background which means no console (tty).

The possible related bug is that I use the echo property in a couple of places. When I try to capture the ant output, the echos do not show up when run in background. They do from command line. I was able to work around that problem by using -logfile and reading that into my script but I shouldn't have had to do that.

The bottom line is that ant behaves differently when run in background and/or with no console. It does not seem to matter if I use 'nohup'. Unless it is trying to write directly to a tty rather than STDOUT and STDERR, this behavior should not happen. I suppose it is possible that there is some environment variable missing that it needs but there is no indication in the error message if or what that might be.
Comment 30 ittai 2012-03-30 11:40:52 UTC
I'm also seeing this on a CentOS installation.
I can check on sunday what version exactly the ant is and I think I can update it without a problem.
This is a real problem for me.
Any workarounds? (BTW, the file isn't held by java in my case)
Comment 31 raghavendra.1729 2012-04-02 12:27:53 UTC
Hi
I'm too facing the same issue, I tried with ANT version ant_1.6.5 on RHEL 5.4.
Any quick workaround can help me lot.
Thanks
Raghav
Comment 32 David S 2012-12-10 22:20:33 UTC
I just faced the same problem, but got it fixed very easily.

Turns out that apparently, it is related to the path/filename of the file to which the regexp replacement is being done.

So, instead of trying to do the regexp directly to thefile (located in a path like C:\build\nightly\files\product_20121210_1600\file.txt), I just tried the following:

1)copy the file to C:\Temp
2)attempt the regexp to that file (C:\Temp\file.txt)
3)copy the file from C:\Temp to C:\build\nightly\files\product_20121210_1600\file.txt

and that's it. Worked like a charm.

I was using ANT 1.7.1, then moved to 1.8.4 to see if it solved the problem (which didn't). However, I'm assuming it should work with 1.7.1 as well.
Comment 33 David S 2012-12-10 23:52:12 UTC
In fact, by double checking, my previous comment is wrong. When I tried to copy back the file to the original path, I got an access denied error. And after verifying, indeed the original file was always having read-only permissions. That was my problem (not the file path/ file name).

Check your file permissions, you might be facing the same thing I was (rookie's problem!)
Comment 34 Pitecantrop 2013-02-13 14:11:06 UTC
Another workaround is redefining temporary dir location. 
Сan't find some rule, but overriding the temporary dir to the drive where you run ant should help.

ant -Djava.io.tmpdir=D:\temp ...