Bug 16779

Summary: scrape taglib functionality
Product: Taglibs Reporter: Struts User <strutsuser>
Component: Scrape TaglibAssignee: Tomcat Developers Mailing List <dev>
Status: RESOLVED FIXED    
Severity: enhancement    
Priority: P3    
Version: 1.0   
Target Milestone: ---   
Hardware: All   
OS: other   

Description Struts User 2003-02-04 21:02:48 UTC
can someone advise on how i might send user-agent and referer headers on 
scrapes... 

by default, according to logs, the referer and user-agent headers are not sent, 
they show as "-" "-" in http-access.log, some sites don't serve content to 
blank user agents. 

i tried using proxomitron (a freeware proxy utility 
http://www.proxomitron.org/), to specify agent using scrapes provision for 
proxy and proxyport, but still blank referer and user-agent are sent??? this is 
baffling, because when i setup my browser to use proxomitron's proxy, the 
specified referer and agent appear in logs.

 
also, how might i lower the default 10-minute default before a rescrape?
Comment 1 Rich Catlett 2003-02-19 05:14:08 UTC
I added a Header tag that must be nested within the page tag that allows http
request headers to be set.