Bug 7163

Summary: Parallel make test
Product: Spamassassin Reporter: Joe Quinn <jquinn+SAbug>
Component: Building & PackagingAssignee: SpamAssassin Developer Mailing List <dev>
Status: NEW ---    
Severity: enhancement CC: apache, jquinn+SAbug, kmcgrail, toddr
Priority: P2    
Version: SVN Trunk (Latest Devel Version)   
Target Milestone: 4.0.0   
Hardware: All   
OS: All   
Whiteboard:
Attachments: Log of make test

Description Joe Quinn 2015-04-03 12:38:00 UTC
Running "make test" takes a while, and it would be nice to make it support parallel testing to make it go faster for easier rapid development.

The test suite is executed as a single job, so -j will not work to speed it up as you would usually expect. Fortunately, Test::Harness looks for a few environment variables and one of them lets us set jobs. Unfortunately, I get the following error when trying to use it on the current trunk, among others:

    Apr  3 08:23:42.600 [20246] warn: locker: error accessing ./log/user_state/auto-whitelist.lock: No such file or directory at ../blib/lib/Mail/SpamAssassin/Locker/UnixNFSSafe.pm line 106. 

There are likely more issues, will include a full log in a bit. In addition to being able to do faster testing, we can gain some extra opportunities for concurrency and maybe make scanning faster as well.

Further Reading:
http://search.cpan.org/~leont/Test-Harness-3.35/lib/Test/Harness.pm
http://stackoverflow.com/questions/9147196/makefile-pass-jobs-param-to-sub-makefiles
Comment 1 Joe Quinn 2015-04-03 13:42:03 UTC
Created attachment 5292 [details]
Log of make test

Full test log attached. The command I ran was:

HARNESS_OPTIONS=j2 make test

From quickly scanning the file, I see a variety of issues that seem to largely revolve around file manipulation. The test did run faster by 150 seconds, even for being on a single-processor VM.
Comment 2 Kevin A. McGrail 2018-02-05 07:14:53 UTC
In addition to running the plans on test in parallel, it would seem that running separate t/ files in parallel might be much quicker.
Comment 3 Todd Rinaldo 2018-02-06 01:45:03 UTC
I'm planning on looking at this some time soon.
Comment 4 Kevin A. McGrail 2018-02-06 02:23:23 UTC
(In reply to Todd Rinaldo from comment #3)
> I'm planning on looking at this some time soon.

Thanks.  Right now it takes 16 mins on my rig for a non-root make test.  So every little snippet means a quarter hour gone.  If I can close that gap down, it will help.  

The harness options for parallel in a single test doesn't look that promising but the concept that perhaps SATest.pm is a framework to run more than one t/ file at a time is interesting to me.

Both would be ideal but perhaps overthinking it!

KAM
Comment 5 Todd Rinaldo 2018-02-06 06:27:39 UTC
Assuming the test can't be broken up Test::Parallel works just fine. 

I think this requires you don't care about windows.

Todd