Bug 7163 - Parallel make test
Summary: Parallel make test
Status: NEW
Alias: None
Product: Spamassassin
Classification: Unclassified
Component: Building & Packaging (show other bugs)
Version: SVN Trunk (Latest Devel Version)
Hardware: All All
: P2 enhancement
Target Milestone: 4.0.0
Assignee: SpamAssassin Developer Mailing List
Depends on:
Reported: 2015-04-03 12:38 UTC by Joe Quinn
Modified: 2019-06-15 20:12 UTC (History)
4 users (show)

Attachment Type Modified Status Actions Submitter/CLA Status
Log of make test text/plain None Joe Quinn [HasCLA]

Note You need to log in before you can comment on or make changes to this bug.
Description Joe Quinn 2015-04-03 12:38:00 UTC
Running "make test" takes a while, and it would be nice to make it support parallel testing to make it go faster for easier rapid development.

The test suite is executed as a single job, so -j will not work to speed it up as you would usually expect. Fortunately, Test::Harness looks for a few environment variables and one of them lets us set jobs. Unfortunately, I get the following error when trying to use it on the current trunk, among others:

    Apr  3 08:23:42.600 [20246] warn: locker: error accessing ./log/user_state/auto-whitelist.lock: No such file or directory at ../blib/lib/Mail/SpamAssassin/Locker/UnixNFSSafe.pm line 106. 

There are likely more issues, will include a full log in a bit. In addition to being able to do faster testing, we can gain some extra opportunities for concurrency and maybe make scanning faster as well.

Further Reading:
Comment 1 Joe Quinn 2015-04-03 13:42:03 UTC
Created attachment 5292 [details]
Log of make test

Full test log attached. The command I ran was:

HARNESS_OPTIONS=j2 make test

From quickly scanning the file, I see a variety of issues that seem to largely revolve around file manipulation. The test did run faster by 150 seconds, even for being on a single-processor VM.
Comment 2 Kevin A. McGrail 2018-02-05 07:14:53 UTC
In addition to running the plans on test in parallel, it would seem that running separate t/ files in parallel might be much quicker.
Comment 3 Todd Rinaldo 2018-02-06 01:45:03 UTC
I'm planning on looking at this some time soon.
Comment 4 Kevin A. McGrail 2018-02-06 02:23:23 UTC
(In reply to Todd Rinaldo from comment #3)
> I'm planning on looking at this some time soon.

Thanks.  Right now it takes 16 mins on my rig for a non-root make test.  So every little snippet means a quarter hour gone.  If I can close that gap down, it will help.  

The harness options for parallel in a single test doesn't look that promising but the concept that perhaps SATest.pm is a framework to run more than one t/ file at a time is interesting to me.

Both would be ideal but perhaps overthinking it!

Comment 5 Todd Rinaldo 2018-02-06 06:27:39 UTC
Assuming the test can't be broken up Test::Parallel works just fine. 

I think this requires you don't care about windows.