Issue 83540

Summary: Add support for multi processing for native W32 dmake
Product: Build Tools Reporter: quetschke
Component: dmakeAssignee: hjs <hans-joachim.lankenau>
Status: CLOSED FIXED QA Contact: issues@tools <issues>
Severity: Trivial    
Priority: P3 CC: hans-joachim.lankenau, issues, stevehay
Version: current   
Target Milestone: ---   
Hardware: All   
OS: Windows, all   
Issue Type: DEFECT Latest Confirmation in: ---
Developer Difficulty: ---
Attachments:
Description Flags
Patch for dmake
none
Patch for dmake
none
Testcase
none
Patch for native W32 dmake
none
Patch for dmake
none
dmake error from "build --all -- -P6" none

Description quetschke 2007-11-11 22:52:43 UTC
Allow parallel building with native W32 dmake. Currently the -P# (MAXPROCESS)
switch is not working.
Comment 1 quetschke 2007-11-11 22:57:33 UTC
The following preliminary patch enables the multi process feature for MinGW
builds, the build changes for MS compilers are not ready yet.

With the patch the following makefile builds a1 and a2 in parallel:

D:\w1\cws_src680_dmake412>dmake\dmake.exe -P3 -rf makefile.mk
Start a1
Start a2
End a1
End a2
Build all

- - - start makefile.mk - - -
SHELL:=cmd.exe 
SHELLFLAGS:=/S /C

all : a1 a2
	@echo Build all

a1 :
	@echo Start a1
	@sleep 4
	@echo End a1

a2 :
	@echo Start a2
	@sleep 4
	@echo End a2
- - - end makefile.mk - - -
Comment 2 quetschke 2007-11-11 23:00:30 UTC
Created attachment 49584 [details]
Patch for dmake
Comment 3 quetschke 2007-11-11 23:58:44 UTC
The following patch makes dmake buildable with .NET2003 again. I verified that
my minimal parallel build testcase also works with that version. I hope this
also compiles with MSVC 6.0, but I cannot test that.
Comment 4 quetschke 2007-11-12 00:01:14 UTC
Created attachment 49585 [details]
Patch for dmake
Comment 5 quetschke 2007-11-12 00:12:39 UTC
Both patches committed to CWS dmake412.
Comment 6 quetschke 2007-11-25 17:59:09 UTC
Everything is committed and the simple test case works in parallel mode for
native W32 dmakes -> FIXED.

It would be nice if our testsuite could test MinGW / VC build dmakes, but
currently it can not.
Comment 7 quetschke 2007-11-25 17:59:45 UTC
Please verify.
Comment 8 quetschke 2007-11-26 00:30:01 UTC
Hrmpf, there seems to be a subtle difference that manifests itself only with
a 4NT OOo build. The following attached makefile works fine with dmake411
and 4nt, but fails with dmake412. In a regular cmd.exe shell:

D:\w1\tin_ws2>dmake412.exe -rf dm_test.mk
"Compiler Version 13.10.3077"
dmake412.exe: Executing shell macro: +type testfile $(CFLAGSVERSION_CMD)
awk: (FILENAME=- FNR=1) warning: error writing standard output (Bad file descriptor)
dmake412.exe:  Error code 129, while making 'Shell escape'

This works fine if cmd.exe is used instead of 4nt :(
Comment 9 quetschke 2007-11-26 00:30:47 UTC
Created attachment 49887 [details]
Testcase
Comment 10 quetschke 2007-11-26 03:36:23 UTC
Created attachment 49889 [details]
Patch for native W32 dmake
Comment 11 quetschke 2007-11-28 04:07:01 UTC
Created attachment 49955 [details]
Patch for dmake
Comment 12 quetschke 2007-11-28 14:38:46 UTC
Building OOo is a nice testcase ;)

@ause: As is said on IRC I am still a little bit suspicious about the code in
unix/rmprq.c, but the bug was not using it and to use the code from the single
process only msdos/rmprq.c.

A 4nt build with -P2 ran successful until hitting an expected break in m236 in
canvas/source/directx.

Please verify.
Comment 13 hjs 2007-12-04 17:51:10 UTC
Created attachment 50095 [details]
dmake error from "build --all -- -P6"
Comment 14 hjs 2007-12-04 17:52:39 UTC
from tim e to time the build breaks and i get pages of the error attached. will
try top get some context
Comment 15 hjs 2007-12-04 19:11:41 UTC
wow, 30000 lines of errors before actually stopping ;)
tends to break in extras, but no more information from the log
Comment 16 quetschke 2008-01-08 21:00:26 UTC
@ause: Can you try again? I don't think I fixed it, but I added some diagnostics
and also caught the loop when quitting.
Comment 17 quetschke 2008-02-06 19:37:49 UTC
Some more patches and clean-up changes went in, and ause confirmed that it seems
to work now.

Setting to FIXED, please verify.
Comment 18 hjs 2008-02-25 16:45:44 UTC
it still does :)
Comment 19 hjs 2008-06-05 15:06:05 UTC
.