This may already be a known issue. When multiple java process log to the same log file and rotate the same log file the process that gets to rotate the file first gets the change to write to the new and most current log file and all other java processes continue to log to the renamed, rotated file. Scenario relevance : this probably should happen in any clustered environtment where cluster members log to the same log file. The issue seems to be happenning because the file writers in the other java processes are still pointing to the old file that was renamed by the other java process in a cluster. So if there is an additional rollOver method, in the current RollOver classes, that is cluster aware that checks the file writers in the appenders and refreshes them before writing to a log file and also after rolling them over it would be a great help to clustered environtments. Otherwise developers may have to extend the appenders and add this functionality and may have to update code everytime when Apache does a new log4j release. I have attached one such sample zip file with changed src code for reference. Or if there is some other solution to this problem can someone pls update the online documentation. Thanks !
You are correct that it is a known issue. I don't have the time at the moment to check which bug I should mark this as a duplicate. The unavoidable issue (at least for anything derived from WriterAppender) is that java.io provides no mechanism to synchronize file writing between multiple JVM's or even multiple FileOutputStreams within a single JVM's. Most of the time when people configure multiple appender to write to the same file, everything seems to work. However, they are depending on implementation details. Things do go spectacularly wrong with the RFA's as you have noticed. There is no way to reliably address this using java.io. The right thing to do would be to write a RFA replacement using java.nio which has been on my wish list for years now.
Or you just avoid the contention of trying to share one log file across multiple processes, e.g. by using process id (and host name as needed) in the log file name. [The process id is readily available in Java 5 and higher.]
Alternatively, you could use logback, log4j's successor, which supports having multiple JVMs writing to the same file. See prudent mode [1] for details. [1] http://logback.qos.ch/manual/appenders.html#prudent
Thank you all for your response ! I tried the tip to use logback and tested this in a windows env and it seems to work fine, thanks, when same file is accessed from multiple jvms and from multiple threads. General questions though ..since the app is already in prod and we are looking to take the route with minimal change to fix this until the next release is scheduled... * Is log4j goging to be retired soon ? Is there a place in the website where I can see End Of Life notifications for log 4j api ? * If we had to override a few public and protected methods from the current rolling file appenders how safe is it from a change management point of view ? Ball park, are there any proposals to update RFA's say within the next 6 months ? Thanks ! (In reply to comment #3) > Alternatively, you could use logback, log4j's successor, which supports having > multiple JVMs writing to the same file. See prudent mode [1] for details. > [1] http://logback.qos.ch/manual/appenders.html#prudent
* SLF4J ships with a migration tool [1] which significatly eases the pain of migrating to SLF4J from log4j. Once you migrate to SL4J, you can either continue to use log4j or switch to logback. Since logback natively implements the SLF4J API, you don't need change the code in *your* application. BTW, that is the whole point of SLF4J. * Although log4j has been vegetating for several years, I don't see any indication that log4j will be officially retired any time soon. * The logging API of logback is determined by SLF4J which is very stable. However, although logback internals are relatively stable, that stability is not guaranteed and radical changes may occur. More specifically, there are proposals [2] to change the way FileAppender and co. write their output to log files. When this change is adopted, it will affect your implementation of RFA. Nevertheless, you should be able to adapt to the changes with few lines of code on your side. It should be mechanical change... [1] http://www.slf4j.org/migrator.html [2] http://jira.qos.ch/browse/LBCORE-128
I have not reviewed the logback implementation, but there is nothing that indicates that implementing the "prudent" mode in log4j would not be possible. Apache log4j and QoS.ch's logback have a shared heritage, but have divergent licenses, communities and governance. Statements from one fork about the other should be considered in that light.
(In reply to comment #6) > I have not reviewed the logback implementation, but there is nothing that > indicates that implementing the "prudent" mode in log4j would not be possible. Prudent mode could be implemented in log4j. It would just take a few days of work. > Apache log4j and QoS.ch's logback have a shared heritage, but have divergent > licenses, communities and governance. Statements from one fork about the other > should be considered in that light. Arnold, speaking of governance, when do you plan to put your seat as chairman of the Apache Logging project for elections? Five years without elections is not exactly the Apache way, is it? You previously stated that since the advent of java.util.logging there was no longer any need for log4j. It is my understanding that your vision for the future of log4j is to see it vegetate and die. Or am I misquoting you in some way?
I've reviewed logback's implementation of "prudent" mode. Essentially, it precedes each stream write with java.nio calls to lock the stream and position the stream to the end of the file, then it performs the write and then releases the lock. As it depends on java.nio, it is not available on JDK's prior to 1.4. It is a reasonable hybrid use of java.io and .nio. I'll take a shot implementing it and try to make it so it doesn't add a JDK 1.4 run-time dependency unless you use it.
By my reading, the effectiveness of prudent mode is dependent on the JVM implementation and OS. Can't make a hard guarantee that things will work as desired or consistently on different JVM implementations or OS's. http://java.sun.com/javase/6/webnotes/compatibility.html, item 3, suggests that it may be more effective on JDK 1.6 than 1.4 or 1.5. http://books.google.com/books?id=BSz14O3uQwsC&lpg=PA68&ots=ZOO7a9ysfH&dq=java%20FileChannel%20multiple%20JVM&pg=PA68#v=onepage&q=java%20FileChannel%20multiple%20JVM&f=false (O'Reilly Java NIO, page 68) "The semantics of concurrent file access by multiple processes is highly dependent on the underlying operating system and/or filesystem. Concurrent access to the same file by FileChannel objects running in different JVMs will, generally, be consistent with concurrent access between non-Java processes."
I've done a quick implementation of prudent mode. However, I think your success with logback's RFA's could be due to switch from Unix to Windows, instead of switching to prudent mode on logback. Reviewing your log you mentioned that you tested logback's RFA in a Windows environment, but your initial bug report indicates Solaris and the behavior you describe is how RFA's misbehave on Unix platforms when multiple RFA's are acting. File renames and deletes are radically different between Windows and Unix. Basically, in Unix directory entries are changed, but any existing handle is still valid for operations. On Windows, an attempt to rename an open file will fail and the rollover does not occur. Prudent mode would likely improve observed behavior of multiple JVM's writing to the same file, however I do not believe it would offer any benefit to the observed behavior on rollovers under Unix-like OS's. I did a quick check of the logback code and was not able to find anything that indicated that looked like cross-app locking around file renames if that were possible.
Committed purient mode option in rev 909868.
Reverted in rev 910151. The independent authorship of the patch is disputed, see log4j-dev mailing list on 2010-02-14. Since a release push for log4j 1.2.16 is underway, it seems prudent to pull the patch for the time being.
Iam sorry .you are correct I performed my test on windows env the other day with log back ..today I compiled and ran the same code in Solaris 5.10 env with jdk150_11 and got the same results. I used TimeBasedRolling with prudent mode( as there were some constraints with using size based rolling with prudent mode in logback) and tried to roll same file from two multithreaded java instances pointing to the same log file and the rolling and logging worked well as far as I can tell found no issues with Solaris. Btw ..thank you for the fix ..I will look forward to it on the next release of log4j. (In reply to comment #12) > Reverted in rev 910151. > The independent authorship of the patch is disputed, see log4j-dev mailing list > on 2010-02-14. Since a release push for log4j 1.2.16 is underway, it seems > prudent to pull the patch for the time being.
Hi iam seeing that log4j version 16 got released sometime in April ..so did this patch make it in by any chance ? Thanks ! (In reply to comment #13) > Iam sorry .you are correct I performed my test on windows env the other day > with log back ..today I compiled and ran the same code in Solaris 5.10 env with > jdk150_11 and got the same results. > I used TimeBasedRolling with prudent mode( as there were some constraints with > using size based rolling with prudent mode in logback) and tried to roll same > file from two multithreaded java instances pointing to the same log file and > the rolling and logging worked well as far as I can tell found no issues with > Solaris. > Btw ..thank you for the fix ..I will look forward to it on the next release of > log4j. > (In reply to comment #12) > > Reverted in rev 910151. > > The independent authorship of the patch is disputed, see log4j-dev mailing list > > on 2010-02-14. Since a release push for log4j 1.2.16 is underway, it seems > > prudent to pull the patch for the time being.
Hi I have written a Java Class to log some telephonic based error event in Log4j logs. I am able to write that logs to the file however other java class which not a part of my Java Application project are also pushing there logs in my log fie. Any idea how do I fix this issue?