| Summary: | tar task is very slow when a <zipfileset> is used as the source resource | ||
|---|---|---|---|
| Product: | Ant | Reporter: | Steve Loughran <stevel> |
| Component: | Core tasks | Assignee: | Ant Notifications List <notifications> |
| Status: | NEW --- | ||
| Severity: | normal | CC: | mike, noelgrandin |
| Priority: | P2 | ||
| Version: | 1.8.2 | ||
| Target Milestone: | --- | ||
| Hardware: | Other | ||
| OS: | other | ||
no, it is an implementation artifact. We open and close the zip archive for each entry (because right now we wouldn't know when to close it otherwise). Having the same problem when using
<zip destfile="proguard_input.zip" >
<archives>
<zips>
<filelist refid="peralex.libs" />
</zips>
</archives>
</zip>
This makes it pretty much unusable when operating from a network drive.
This is probably also why using <copy> with a nested <zipfileset> is awfully slow for any decent size archive making it rather useless. +1.. something's really wrong here, on Ant v1.9.0 we're seeing <zip> of our 666 MB product takes 1 minute, 930 MB <tar> (using two <zipfileset>) takes.. 30 minutes! I had similar problem, using <zipfileset> consumes more than 30 minutes to create a tar. This is how I solved, tar creation takes approximately 40s.
<unzip src="product.zip" dest="extractedProduct"/>
<tar tarfile="product.tar.gz" longfile="gnu" compression="gzip">
<tarfileset dir="extractedProduct" includes="**/PS" filemode="755" />
<tarfileset dir="extractedProduct" excludes="PS" />
<fileset dir="${rootdir}" includes="resources/"/>
</tar>
<delete dir="extractedProduct" quiet="true"/>
This is a major limitation of Ant. With other build systems (Maven, Gradle...) making an uber-jar that contains all dependencies is very fast. With Ant it is painfully slow and we have to manually unzip/zip them to get decent performance. I have opened a pull request[1] with a potential way to solve this. [1] https://github.com/apache/ant/pull/76 > With other build systems (Maven, Gradle...) making an uber-jar that contains all dependencies is very fast.
Vincent, you mention uber-jar whereas this issue started off with the tar task and the proposed patch is currently only applied in the tar task. Are you using the jar task with a zipfileset and that one too is showing a slow performance? I haven't checked the code for that yet but I won't be surprised if that task too is impacted. Do you have a sample build file showing your usage?
Jaikiran, see Noel's example here which probably tries to build an ueberjar. Likely background: https://stackoverflow.com/q/35577351/4524982 Hi, Thanks for replies! Yes I was building an uber-jar using jar > restrict > archives > zips > fileset, resulting in a 85 Mb jar file with ~2000 files. The build time took more than 30 minutes. I changed it using unzip + jar and the build time is now about 1 minute, see this commit for details: https://trac.openstreetmap.org/changeset/34703/subversion/applications/editors/josm/plugins/javafx/build.xml I debugged Ant quickly with VisualVM and found out the processing of all different files resulted in hundreds of thousands of calls to ZipResource.getInputStream(). The symptom looks the same as Stefan first comment ("We open and close the zip archive for each entry"). I will try your PR (thanks for creating it so fast!) Thanks Stefan and Vincent for those examples. I had a look at those and the PR that I have open won't address this usecase and will only address the tar task. This other usecase(s) will need a similar fix and I'll look into how to address it. I encountered the same problem today.
Trying to pack the following into a JAR using <archive> took 5 minutes:
<path id="runtime_classpath">
<pathelement path="${javax:javaee-api:jar}"/>
<pathelement path="${com.solacesystems:sol-jms:jar}"/>
<pathelement path="${com.fasterxml.jackson.core:jackson-databind:jar}"/>
<pathelement path="${com.fasterxml.jackson.core:jackson-core:jar}"/>
<pathelement path="${com.fasterxml.jackson.core:jackson-annotations:jar}"/>
<pathelement path="${commons-logging:commons-logging:jar}"/>
<pathelement path="${commons-lang:commons-lang:jar}"/>
<pathelement path="${log4j:log4j:jar}"/>
<pathelement path="${org.slf4j:slf4j-api:jar}"/>
<pathelement path="${org.slf4j:slf4j-log4j12:jar}"/>
</path>
<jar destfile="${out.dir}/${env}/${project.build.finalName}.jar">
<fileset
dir="${project.build.directory}\${project.build.finalName}\WEB-INF\classes"
includes="**/*.class, **/*.json, **/*.properties"
/>
<archives>
<zips>
<path refid="runtime_classpath" />
</zips>
</archives>
<manifest>
<attribute name="Main-Class" value="${mainClass}"/>
</manifest>
</jar>
I changed to use Unzip to a temp folder, and add the temp folder as a fileset instead. That took 1 minute.
The archive tag is doing something really inefficiently...
|
I dont know where the blame is, but I'm trying to create a zip and tar. So I <zip> everything up (a few seconds; its a big source tree) and then jump to <tar>. <tar destFile="${full.tar}" longfile="gnu"> <zipfileset src="${full.zip}"/> </tar> Which takes a few minutes with the CPU and 100% and memory consumption at the limit a JVM can handle. Much, much slower than using a file resource. Is it the fact that the zip files are compressed?