Bug 8110 - DecodeShortURLs should offer per-shortener setting to override UA
Summary: DecodeShortURLs should offer per-shortener setting to override UA
Status: NEW
Alias: None
Product: Spamassassin
Classification: Unclassified
Component: Plugins (show other bugs)
Version: 4.0.0
Hardware: PC Linux
: P4 normal
Target Milestone: Undefined
Assignee: SpamAssassin Developer Mailing List
URL:
Whiteboard:
Keywords:
: 8114 (view as bug list)
Depends on:
Blocks:
 
Reported: 2023-01-22 02:34 UTC by Christer Mjellem Strand
Modified: 2023-01-25 01:59 UTC (History)
2 users (show)


Attachment Type Modified Status Actions Submitter/CLA Status
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Christer Mjellem Strand 2023-01-22 02:34:18 UTC 
The default UA in DecodeShortURLs tries to emulate a browser, which for many redirectors is necessary. However, for certain redirectors this causes an undesired change in behavior.

Compare (using URL found in spam, originally leading to a phishing page):

curl -A "Mozilla/5.0 (Windows NT 10.0; Win64 ;x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.67 Safari/537.36)" -I https://t.co/QXabAdmraO

This uses the default UA, and results in a 200, with a meta refresh in the body. No trouble for a browser, but trouble for us.

But when using a non-browser (or at least specifically a "curl/n.nn.n") UA:

curl -I https://t.co/QXabAdmraO

You get a 301 with a location header, which works fine.

To get around this, there could be a config setting to override UA on a per-shortener basis, the same way url_shortener_get lets you override method. For maximum flexibility, this could even be a string input, so you could set an entirely custom UA for an individual shortener.
Comment 1 Christer Mjellem Strand 2023-01-22 02:41:32 UTC 
See also somewhat related bug #8111. Addressing either should be sufficient to make t.co work, though if having to pick one, this would probably be the cleaner (and likely easier) approach. OTOH, the other would fix any such broken shorteners out of the gate, whereas this would require further config for each of them.
Comment 2 Sidney Markowitz 2023-01-25 01:59:45 UTC 
*** Bug 8114 has been marked as a duplicate of this bug. ***