Bug 16779 - scrape taglib functionality
Summary: scrape taglib functionality
Status: RESOLVED FIXED
Alias: None
Product: Taglibs
Classification: Unclassified
Component: Scrape Taglib (show other bugs)
Version: 1.0
Hardware: All other
: P3 enhancement (vote)
Target Milestone: ---
Assignee: Tomcat Developers Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2003-02-04 21:02 UTC by Struts User
Modified: 2004-11-16 19:05 UTC (History)
0 users


Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Struts User 2003-02-04 21:02:48 UTC 
can someone advise on how i might send user-agent and referer headers on 
scrapes... 

by default, according to logs, the referer and user-agent headers are not sent, 
they show as "-" "-" in http-access.log, some sites don't serve content to 
blank user agents. 

i tried using proxomitron (a freeware proxy utility 
http://www.proxomitron.org/), to specify agent using scrapes provision for 
proxy and proxyport, but still blank referer and user-agent are sent??? this is 
baffling, because when i setup my browser to use proxomitron's proxy, the 
specified referer and agent appear in logs.

 
also, how might i lower the default 10-minute default before a rescrape?
Comment 1 Rich Catlett 2003-02-19 05:14:08 UTC 
I added a Header tag that must be nested within the page tag that allows http
request headers to be set.