% svn ci -m "bug 8227 - Add tests for proper DNS resolver setup during build/install" lib/Mail/SpamAssassin/Util/DependencyInfo.pm Makefile.PL MANIFEST t/dnsplatform.t
Sending        MANIFEST
Sending        Makefile.PL
Sending        lib/Mail/SpamAssassin/Util/DependencyInfo.pm
Adding         t/dnsplatform.t
Transmitting file data ....done
Committing transaction...
Committed revision 1916223.

The bits added to DependencyInfo.pm breaks install on some networks where SpamAssassin would work by adding "dns_server 127.0.0.1" option.

(In reply to Giovanni Bechis from comment #2)
> The bits added to DependencyInfo.pm breaks install on some networks where
> SpamAssassin would work by adding "dns_server 127.0.0.1" option.

Oh, that's a problem for the fix I did for bug 8225 too, because the calls to Net::DNS::Resolver aren't using the dns_server conf options.

I also didn't think about t/config.dist default is run_net_tests=n which this check ignores, breaking the install on systems that will not get test failures.

Ok, I'll have to revert that part of the commit. The new t/dnsplatform.t test is still useful, though. If it fails it signals that the failures in the other network tests that fail can be ignored, and that dns_server config options may be necessary to run on that system.

(In reply to Giovanni Bechis from comment #2)
> The bits added to DependencyInfo.pm breaks install on some networks where
> SpamAssassin would work by adding "dns_server 127.0.0.1" option.

What I was trying to do was overkill anyway. Instead killing the entire build, and not even suppressing all network tests, since most of them won't trigger big DNS replies or whatever is going wrong with multihomed addresses, I can do this:

Put the test of getting the txttcp large packet in a function in t/SATest.pm and have only the tests that actually fail when it is a problem call it (not all network tests have to care about it). If it fails, they can skip all tests with a warning message that dns_server may have to be set when running SpamAssassin on that system.

Add the check of multihomed to t/spamc_H.t so it can skip the tests with a similar warning if something is wrong with the local DNS resolver that affects getting those records.

That will prevent the egregious failure reports on those CPAN test machines that have bad DNS configuration without breaking any builds, and will warn people who need to use dns_server options in their configuration.

How does that seem?

I have more information from a test build I uploaded to CPAN to see what their test machines make of it. The results are a bit confusing. What I uploaded is what is in trunk now, which will not generate any report from a test machine that fails the DNS test in Makefile.PL. Despite that, there are a number of failures reported anyway, although from the lower numbers I think that most test machines with problems are failing fast and silently.

There are three failures, on the same FreeBSD 10.1 test machine with three runs each using different versions of perl, getting identical failures. They fail t/dnsplatform.t, getting 0 answers to the query of multihomed.dnsbltest.spamassassin.org A even though the identical test in Makefile.PL has to have passed for the build to continue on to run the tests. They also all have the same failures in t/dnsbl.t and t/spamc_H.t

I can't do any testing on a FreeBSD 10.1 VM because I can't find any FreeBSD archives that go back that far, other than the basic install images.

Another three failures are on a Windows 10 test machine, again three runs with different versions of perl. Those all have failures in t/dnsbl.t, t/hashbl.t, and t/spf.t, but the specific tests that fail are different. I suspect that there are replies being lost randomly. I haven't yet seen any other tests of this upload on Windows machines, and I can't reproduce the problem on my Windows 10 VM.

Given that this test is not definitively predictive of the failures, and especially because I can't see why we should care about failures in FreeBSD 10.1, I'll revert all the changes I committed except to keep to new t/dnsplatform.t which is still useful to alert one to a badly configured DNS setup that should either be fixed or require use of the dns_server option.

% svn ci -m "bug 8227 - Revert addition of DNS test run durimg build"  Makefile.PL lib/Mail/SpamAssassin/Util/DependencyInfo.pm
Sending        Makefile.PL
Sending        lib/Mail/SpamAssassin/Util/DependencyInfo.pm
Transmitting file data ..done
Committing transaction...
Committed revision 1916250.

Now the only thing that has ben added for this issue is the new t/dnsplatform.t test. I'm closing this as resolved.

(In reply to Sidney Markowitz from comment #5)
> I can't see why we should care about failures in FreeBSD
> 10.1, 

IMHO We shouldn't.
 
I say that as someone who wrangles a fleet of FreeBSD boxes in my day job. It would be irresponsible to encourage anyone to keep a box that far from modernity functional. If SA builds, that's great, but it would be a waste of time to bad ends for us to do anything to make it work.