Issue 249 - Downloads cannot complete
Summary: Downloads cannot complete
Status: CLOSED FIXED
Alias: None
Product: Infrastructure
Classification: Infrastructure
Component: Downloads (show other issues)
Version: current
Hardware: All All
: P3 Trivial (vote)
Target Milestone: ---
Assignee: Unknown
QA Contact: issues@www
URL:
Keywords:
: 250 251 (view as issue list)
Depends on:
Blocks:
 
Reported: 2000-12-19 21:40 UTC by issues@www
Modified: 2003-12-06 14:52 UTC (History)
1 user (show)

See Also:
Issue Type: DEFECT
Latest Confirmation in: ---
Developer Difficulty: ---


Attachments

Note You need to log in before you can comment on or make changes to this issue.
Description issues@www 2000-12-19 21:40:41 UTC
I currently cannot download any large files
through http, the transfer breaks, and wget does not 
continue, but restarts.  Perhaps there is an interfering 
transparent proxy between my host and yours.

Is there any good reason for not allowing downloads through ftp?
Comment 1 Unknown 2000-12-19 21:58:04 UTC
*** Issue 250 has been marked as a duplicate of this issue. ***
Comment 2 Unknown 2000-12-19 21:59:02 UTC
*** Issue 251 has been marked as a duplicate of this issue. ***
Comment 3 Unknown 2000-12-20 02:01:50 UTC
First -- the problem appears to be a client side issue.  If your connection is
spotty, as suggested by your message to openoffice-admin, then there's a limit
to what is possible.

However, you should be able to download larger files eventually using wget's
continue feature.  HTTP does have a mechanism for resuming downloads which are
terminated part way through, and wget supports that feature.  However, you need
to provide the appropriate command line option to convince wget to do so -- this
is outlined in wget's documentation, quite clearly.  FYI, the option is "-c"; I
tried this out and was able to resume a download from openoffice.org.

We use a service called akamai to distribute our files; they maintain servers
throughout the world, so download time from them should be at least slightly
better than it'd be from our servers in California.  We don't use FTP for file
transfers because it doesn't offer any advantages, and maitaining extra
unnecessary services would not be helpful.

If you are still not able to download any files using wget -c, please provide
more detailed information -- your IP, the URL of the page you're looking at, the
URL of the file you're trying to download, the time, and the user-agent used.

For now, I'm going to mark this closed, 'cause I had no problems using wget -c
with the akamai URLs.

thanks --

Ed
Comment 4 issues@www 2000-12-20 12:01:41 UTC
I have used wget -c from the start, so the advice does not help any.
My experience with wget is spotless for ftp, but shaky with http.

Here are some of the details that were requested:

IP: 143.107.45.56
uname -a: SunOS jaca 5.7 Generic_106541-04 sun4u sparc
wget: 1.5.3, locally compiled, exaustively used

wget command line:

wget -S -b -c --proxy=off -nd -nH \
http://a1508.g.akamai.net/7/1508/2064/OpenOffice613/anoncvs.openoffice.org/download/OpenOffice613/install613_solaris_sparc.tar.gz

Part of wget-log for today's run 
*** means ommitted lines):

--09:02:13-- 
http://a1508.g.akamai.net:80/7/1508/2064/OpenOffice613/anoncvs.openoffice.org/download/OpenOffice613/install613_solaris_sparc.tar.gz
           => `install613_solaris_sparc.tar.gz'
Connecting to a1508.g.akamai.net:80... connected!
HTTP request sent, awaiting response... 200 OK
2 Age: 8
3 Date: Wed, 20 Dec 2000 11:02:15 GMT
4 Content-Length: 47022918
5 Content-Type: application/x-gzip
6 Server: Apache/1.3.13-dev (Unix)
7 Last-Modified: Tue, 12 Dec 2000 13:51:58 GMT
8 ETag: "5bd30-2cd8346-3a362d7e"
9 Accept-Ranges: bytes
10 Content-Encoding: x-gzip
11 Via: 1.0 netcache2 (NetCache NetApp/5.0D6)
12 

    0K -> .......... .......... .......... .......... .......... [  0%]
***
  350K -> .......... .......... .......... .....                 [  0%]

09:04:12 (3.57 KB/s) - Read error at byte 395004/47022918 (Connection reset by
peer). Retrying.

--09:04:12-- 
http://a1508.g.akamai.net:80/7/1508/2064/OpenOffice613/anoncvs.openoffice.org/download/OpenOffice613/install613_solaris_sparc.tar.gz
  (try: 2) => `install613_solaris_sparc.tar.gz'
Connecting to a1508.g.akamai.net:80... connected!
HTTP request sent, awaiting response... 200 OK
2 Age: 3
3 Date: Wed, 20 Dec 2000 11:04:16 GMT
***

    0K -> .......... .......... .......... .......... .......... [  0%]
***
  400K -> .........                                              [  0%]

09:07:40 (2.04 KB/s) - Read error at byte 419620/47022918 (Connection reset by
peer). Retrying.
***
3 Date: Wed, 20 Dec 2000 11:07:44 GMT
***
    0K -> .......... .......... .......... .......... .......... [  0%]
***
  300K -> .......... .......... .......... ..........            [  0%]

09:10:35 (2.00 KB/s) - Read error at byte 348668/47022918 (Connection reset by
peer). Retrying.
***

09:15:28 (2.77 KB/s) - Read error at byte 347220/47022918 (Connection reset by
peer). Retrying.
***

09:17:41 (2.24 KB/s) - Read error at byte 287852/47022918 (Connection reset by
peer). Retrying.
***

09:20:44 (2.70 KB/s) - Read error at byte 494916/47022918 (Connection reset by
peer). Retrying.
***
    0K -> .......... .......... .......... .......... .......... [  0%]
   50K -> .......... .......... .......... .......... ........   [  0%]

09:23:47 (574.58 B/s) - Read error at byte 101060/47022918 (Connection reset by
peer). Retrying.


It looks like wget and the server are not communicating correctly.

am
Comment 5 Unknown 2000-12-20 16:40:24 UTC
Thanks for the additional detail.  When I try this, I'm still unable to
duplicate your problems -- but I can't generate connection reset by peer. 
Please do the same test, but without backgrounding wget -- let is download 50kb
or 100kb, cancel it, and then see if you can continue where you left off.

I'm not saying that's a reasonable workaround, but it would help pinpoint the
problem.  I'm also using wget 1.53; it seems possible wget doesn't handle
connection reset very well.  Aside from that, one other piece of information
which might be useful is the output from 'nslookup a1508.g.akamai.net'.

Comment 6 issues@www 2000-12-20 18:19:53 UTC
Ok, let me try:

run wget on the foreground, ^C at some point.

ls -l install613_solaris_sparc.tar.gz
-rw-r--r--   1 am       staff     111196 Dec 20 16:05
install613_solaris_sparc.tar.gz

 wget -c --proxy=off -nd -nH
http://a1508.g.akamai.net/7/1508/2064/OpenOffice613/anoncvs.openoffice.org/download/OpenOffice613/install613_solaris_sparc.tar.gz
--16:07:22-- 
http://a1508.g.akamai.net:80/7/1508/2064/OpenOffice613/anoncvs.openoffice.org/download/OpenOffice613/install613_solaris_sparc.tar.gz
           => `install613_solaris_sparc.tar.gz'
Connecting to a1508.g.akamai.net:80... connected!
HTTP request sent, awaiting response... 200 OK
Length: 47,022,918 [application/x-gzip]

    0K -> .......... .......... .......... .......... .......... [  0%]
   50K -> .......... ...^C

(still -c doesn't work)

nslookup a1508.g.akamai.net
Server:  bidu.ime.usp.br
Address:  143.107.45.12

Name:    a1508.g.akamai.net
Addresses:  208.148.96.125, 208.148.96.149

Comment 7 issues@www 2000-12-22 14:33:07 UTC
The problem is no more.
The cause for the strange miscommunication between wget and the server
was this:

   Via: 1.0 netcache2 (NetCache NetApp/5.0D6)

I contacted the provider that routes my connections
through this netcache, had them route my host 
directly, and now downloading is proceeding in 
the normal way (get a piece, diconnect, reconnect, continue, 
and so on.)

am
Comment 8 Unknown 2001-01-09 04:43:45 UTC
Per last issue comment entry by reporter, 
resolved / fixed.
Comment 9 michael.bemmer 2003-03-24 08:22:49 UTC
As agreed by Louis I will close these resolved fixed support-owned issues now.
If you have trouble with that, please re-open the issue.
Comment 10 michael.bemmer 2003-03-24 08:27:16 UTC
As agreed by Louis I will close these resolved fixed support-owned issues now.
If you have trouble with that, please re-open the issue.