I've suggested it before and I'll suggest it again: if the volume of sa-update downloads is larger than we're comfortable supporting with available infrastructure, then the config files should try using the official mirror servers. This is simple, transparent, automatically distributed, retrieves from a cache server topologically near the requesting host, and is (in my experience) reliable. It also makes updates robust in the face of temporary unavailablility of the mirror servers.

I'm not offering this as an alternative to code changes that rate-limit sa-update download attempts, but as anothar approach to use along with such limits to manage load.

(In reply to comment #1)
> I've suggested it before and I'll suggest it again: if the volume of
> sa-update downloads is larger than we're comfortable supporting with
> available infrastructure, then the config files should try using the
> official mirror servers. This is simple, transparent, automatically
> distributed, retrieves from a cache server topologically near the requesting
> host, and is (in my experience) reliable. It also makes updates robust in
> the face of temporary unavailablility of the mirror servers.
> 
> I'm not offering this as an alternative to code changes that rate-limit
> sa-update download attempts, but as anothar approach to use along with such
> limits to manage load.

I believe this issue is that these are unnecessary checks. For example bug 6655.

If someone is checking for updates, that just hits DNS and is minor.  But we have systems for no known reason checking repeatedly for updates.  

While I agree with you John, the reality is that replacing and keeping the existing infrastructure running for the entire project is time consuming.  I want to try and focus on publishing code rather than changing/improving infrastructure as much as possible.  

Regards,
KAM

(In reply to comment #2)
> (In reply to comment #1)
> > I've suggested it before and I'll suggest it again: if the volume of
> > sa-update downloads is larger than we're comfortable supporting with
> > available infrastructure, then the config files should try using the
> > official mirror servers.

Argh! I don't know how "official mirror servers" crept into that comment.

What I've suggested before and was suggesting again here was to first try CORAL-ified URLs.

> > This is simple, transparent, automatically
> > distributed, retrieves from a cache server topologically near the requesting
> > host, and is (in my experience) reliable. It also makes updates robust in
> > the face of temporary unavailablility of the mirror servers.
> > 
> > I'm not offering this as an alternative to code changes that rate-limit
> > sa-update download attempts, but as anothar approach to use along with such
> > limits to manage load.
> 
> I believe this issue is that these are unnecessary checks. For example bug
> 6655.
> 
> If someone is checking for updates, that just hits DNS and is minor.  But we
> have systems for no known reason checking repeatedly for updates.  
> 
> While I agree with you John, the reality is that replacing and keeping the
> existing infrastructure running for the entire project is time consuming.  I
> want to try and focus on publishing code rather than changing/improving
> infrastructure as much as possible.  

Trying CORAL first is no change whatsoever to infrastructure, I apologize that my misstatement about "official mirror servers" gave that inpression. It is either a minor change to the base sa-update URLs file, or a (likely) minor code change in sa-update, to first try downloading:

  http://buildbot.spamassassin.org.nyud.net:8080/updatestage/1422798.tar.gz

before trying to download:

  http://buildbot.spamassassin.org/updatestage/1422798.tar.gz

simply append ".nyud.net:8080" to the base host FQDN and you try to retrieve the file from a transparent automatic distributed cache service.

> Trying CORAL first is no change whatsoever to infrastructure, I apologize that > my misstatement about "official mirror servers" gave that inpression. It is 
> either a minor change to the base sa-update URLs file, or a (likely) minor code > change in sa-update, to first try downloading:
>
>  http://buildbot.spamassassin.org.nyud.net:8080/updatestage/1422798.tar.gz
>
>before trying to download:
>
>  http://buildbot.spamassassin.org/updatestage/1422798.tar.gz
>
> simply append ".nyud.net:8080" to the base host FQDN and you try to retrieve 
> the file from a transparent automatic distributed cache service.

We used todo this and changed it for some reason.  You might want to search for some history to see why that was to make sure it wasn't any sort of issue, rather than just no longer needing the cache.

(In reply to comment #4)
> > simply append ".nyud.net:8080" to the base host FQDN and you try to retrieve 
> > the file from a transparent automatic distributed cache service.
> 
> We used todo this and changed it for some reason.  You might want to search
> for some history to see why that was to make sure it wasn't any sort of
> issue, rather than just no longer needing the cache.

ISTR the reason was that some people found it (or believed it to be) unreliable.

If sa-update is robust in the face of a corrupt download from one mirror (i.e. it tries another if one fails) this shouldn't be a problem. A corrupt download is a potential problem and a common failure mode that is not specific to any particular source.

Coral has been reliable in my experience.

>  http://buildbot.spamassassin.org.nyud.net:8080/updatestage/1422798.tar.gz

Hmm.. A non-standard port. That will not go through our firewall unless I
add a rule for it. It is not too hard to do that as long as I know that I
need to. Another place for some documentation.

Coral has (also) been in standard port for ages.

http://buildbot.spamassassin.org.nyud.net/updatestage/1422798.tar.gz