Broken uploads
Message boards :
Questions and problems :
Broken uploads
Message board moderation
| Author | Message |
|---|---|
|
Send message Joined: 18 Apr 11 Posts: 6 
|
Hi! I'm experimenting this problem only with RNA World project http://www.rnaworld.de/rnaworld/. If, and only if, for whatever reason a result upload gives a transient error, it won't ever be able to resume: it will try again and again, just to end up in other transient errors. However, most results are uploaded smootly. Six have had problems so far, in a few days. I've sniffed the tcp stream to the boinc server, and to me nothing is wrong till the HTTP 500 error from the server (sorry for the lenght): POST /rnaworld_cgi/file_upload_handler HTTP/1.1
User-Agent: BOINC client (x86_64-pc-linux-gnu 6.10.58)
Host: www.rnaworld.de
Accept: */*
Accept-Encoding: deflate, gzip
Content-Type: application/x-www-form-urlencoded
Content-Length: 343
<data_server_request>
<core_client_major_version>6</core_client_major_version>
<core_client_minor_version>10</core_client_minor_version>
<core_client_release>58</core_client_release>
<get_file_size>cms_GA-p[HZ-Lin64s]_5_Herminiimonas-arsenicoxydans_CU207211.cir.EMBL_1302069423_34469_1_0</get_file_size>
</data_server_request>
HTTP/1.1 200 OK
Date: Fri, 15 Apr 2011 13:53:17 GMT
Server: Apache/2.2.13 (Linux/SUSE)
Content-Length: 98
Content-Type: text/plain
<data_server_reply>
<status>0</status>
<file_size>950156</file_size>
</data_server_reply>
POST /rnaworld_cgi/file_upload_handler HTTP/1.1
User-Agent: BOINC client (x86_64-pc-linux-gnu 6.10.58)
Host: www.rnaworld.de
Accept: */*
Accept-Encoding: deflate, gzip
Content-Type: application/x-www-form-urlencoded
Content-Length: 2676459
Expect: 100-continue
HTTP/1.1 100 Continue
<data_server_request>
<core_client_major_version>6</core_client_major_version>
<core_client_minor_version>10</core_client_minor_version>
<core_client_release>58</core_client_release>
<file_upload>
<file_info>
<name>cms_GA-p[HZ-Lin64s]_5_Herminiimonas-arsenicoxydans_CU207211.cir.EMBL_1302069423_34469_1_0</name>
<generated_locally/>
<upload_when_present/>
<max_nbytes>1000000000</max_nbytes>
<url>http://www.rnaworld.de/rnaworld_cgi/file_upload_handler</url>
<xml_signature>
a8e3ebd257e002dd6269cd3b7a69af1e04fbf847074a0a0d33018a7f393de208
7debe2b318afec68fb30fcb5f9575351780e5ab333fe93fd494d49d08138d88a
12e1bd0c03b9264a67a9b3d02de223c5d027d230ac81b251b370696ccb34e8aa
814dbc2d48e724e58f770d6d37c5298ba56ddb2d93070a482ebccd29d08b48cd
.
</xml_signature>
</file_info>
<nbytes>3625704</nbytes>
<md5_cksum>873418c994c289204e246e97ac20f501</md5_cksum>
<offset>950156</offset>
<data>
.U....:V.e [I'm SNIPPING here] ....i@..HTTP/1.1 500 Internal Server Error
Date: Fri, 15 Apr 2011 13:53:18 GMT
Server: Apache/2.2.13 (Linux/SUSE)
Vary: accept-language,accept-charset
Accept-Ranges: bytes
Connection: close
Content-Type: text/html; charset=iso-8859-1
Content-Language: en
X-Pad: avoid browser bug
<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<title>Server error!</title>
<link rev="made" href="mailto:%5bno%20address%20given%5d" />
<style type="text/css"><!--/*--><![CDATA[/*><!--*/
body { color: #000000; background-color: #FFFFFF; }
a:link { color: #0000CC; }
p, address {margin-left: 3em;}
span {font-size: smaller;}
/*]]>*/--></style>
</head>
<body>
<h1>Server error!</h1>
<p>
The server encountered an internal error and was
unable to complete your request. Either the server is
overloaded or there was an error in a CGI script.
</p>
<p>
If you think this is a server error, please contact
the <a href="mailto:%5bno%20address%20given%5d">webmaster</a>.
</p>
<h2>Error 500</h2>
<address>
<a href="/">www.rnaworld.de</a><br />
<span>Fri Apr 15 15:53:18 2011<br />
Apache/2.2.13 (Linux/SUSE)</span>
</address>
</body>
</html>I'd like to point out that even the "nbytes", the "file_size" and the "offset" fields look right. I wrote about this strange behaviour in the project forum, but no solution came out. I have to say that a nice cruncher helped me investigating, trying himself to stop and then restart one of his uploads, and everything apparently went right for him. We have the same OS and the same Boinc version, but maybe there was something different in the way his upload was stopped and resumed... I cannot write directly to the HTTP server sysadmin, because of "mailto:%5bno%20address%20given%5d". However this stuff actually pisses me off, moreover because I feel like RNA World is a valuable project which I'm really interested in. Is there anything I can try to identify the problem? What could be wrong in the CGI /rnaworld_cgi/file_upload_handler? Thanks in advance. Bye. ID: 37516 ·  |
|
Send message Joined: 18 Apr 11 Posts: 6
|
News. I've enabled logging for: <file_xfer_debug>1</file_xfer_debug> <http_debug>1</http_debug> <http_xfer_debug>1</http_xfer_debug> Dunno why I didn't do it before. Perhaps I stupidly though I couldn't get anything more than sniffing packets. Me idiot! mar 19 apr 2011 20:03:39 CEST RNA World [file_xfer_debug] FILE_XFER_SET::poll(): http op done; retval 0 mar 19 apr 2011 20:03:39 CEST RNA World [file_xfer_debug] parsing upload response: <data_server_reply> <status>0</status> <file_size>2784940</file_size></data_server_reply> mar 19 apr 2011 20:03:39 CEST RNA World [file_xfer_debug] parsing status: 0 mar 19 apr 2011 20:03:39 CEST RNA World [fxd] starting upload, upload_offset 2784940 mar 19 apr 2011 20:03:39 CEST [http_debug] [ID#3989] Info: Re-using existing connection! (#1) with host www.rnaworld.de mar 19 apr 2011 20:03:39 CEST [http_debug] [ID#3989] Info: Connected to www.rnaworld.de (78.46.16.148) port 80 (#1) mar 19 apr 2011 20:03:39 CEST [http_debug] [ID#3989] Sent header to server: POST /rnaworld_cgi/file_upload_handler HTTP/1.1 mar 19 apr 2011 20:03:39 CEST [http_debug] [ID#3989] Sent header to server: User-Agent: BOINC client (x86_64-pc-linux-gnu 6.10.58) mar 19 apr 2011 20:03:39 CEST [http_debug] [ID#3989] Sent header to server: Host: www.rnaworld.de mar 19 apr 2011 20:03:39 CEST [http_debug] [ID#3989] Sent header to server: Accept: */* mar 19 apr 2011 20:03:39 CEST [http_debug] [ID#3989] Sent header to server: Accept-Encoding: deflate, gzip mar 19 apr 2011 20:03:39 CEST [http_debug] [ID#3989] Sent header to server: Content-Type: application/x-www-form-urlencoded mar 19 apr 2011 20:03:39 CEST [http_debug] [ID#3989] Sent header to server: Content-Length: 2194891 mar 19 apr 2011 20:03:39 CEST [http_debug] [ID#3989] Sent header to server: Expect: 100-continue mar 19 apr 2011 20:03:39 CEST [http_debug] [ID#3989] Sent header to server: mar 19 apr 2011 20:03:39 CEST [http_debug] [ID#3989] Received header from server: HTTP/1.1 100 Continue mar 19 apr 2011 20:08:39 CEST [http_debug] [ID#3989] Received header from server: HTTP/1.1 500 Internal Server Error mar 19 apr 2011 20:08:39 CEST [http_debug] [ID#3989] Received header from server: Date: Tue, 19 Apr 2011 18:03:41 GMT mar 19 apr 2011 20:08:39 CEST [http_debug] [ID#3989] Received header from server: Server: Apache/2.2.13 (Linux/SUSE) mar 19 apr 2011 20:08:39 CEST [http_debug] [ID#3989] Received header from server: Vary: accept-language,accept-charset mar 19 apr 2011 20:08:39 CEST [http_debug] [ID#3989] Received header from server: Accept-Ranges: bytes mar 19 apr 2011 20:08:39 CEST [http_debug] [ID#3989] Received header from server: Connection: close mar 19 apr 2011 20:08:39 CEST [http_debug] [ID#3989] Received header from server: Content-Type: text/html; charset=iso-8859-1 mar 19 apr 2011 20:08:39 CEST [http_debug] [ID#3989] Received header from server: Content-Language: en mar 19 apr 2011 20:08:39 CEST [http_debug] [ID#3989] Received header from server: X-Pad: avoid browser bug mar 19 apr 2011 20:08:39 CEST [http_debug] [ID#3989] Received header from server: mar 19 apr 2011 20:08:39 CEST [http_xfer_debug] [ID#3989] HTTP: wrote 1071 bytes mar 19 apr 2011 20:08:39 CEST [http_debug] [ID#3989] Info: we are done reading and this is set to close, stop send mar 19 apr 2011 20:08:39 CEST [http_debug] [ID#3989] Info: Expire cleared mar 19 apr 2011 20:08:39 CEST [http_debug] [ID#3989] Info: Closing connection #1 mar 19 apr 2011 20:08:40 CEST RNA World [file_xfer_debug] FILE_XFER_SET::poll(): http op done; retval -184 mar 19 apr 2011 20:08:40 CEST RNA World [file_xfer_debug] file transfer status -184 mar 19 apr 2011 20:08:40 CEST RNA World Temporarily failed upload of cms_GA-p[LZ-Lin64s]_12_Lactococcus-lactis-subsp.-lactis-Il1403_AE005176.cir.EMBL_1302243723_36714_1_0: HTTP error mar 19 apr 2011 20:08:40 CEST RNA World Backing off 2 hr 57 min 32 sec on upload of cms_GA-p[LZ-Lin64s]_12_Lactococcus-lactis-subsp.-lactis-Il1403_AE005176.cir.EMBL_1302243723_36714_1_0 So here it is, the most interesting part: mar 19 apr 2011 20:08:39 CEST [http_debug] [ID#3989] Info: we are done reading and this is set to close, stop send mar 19 apr 2011 20:08:39 CEST [http_debug] [ID#3989] Info: Expire cleared mar 19 apr 2011 20:08:39 CEST [http_debug] [ID#3989] Info: Closing connection #1 mar 19 apr 2011 20:08:40 CEST RNA World [file_xfer_debug] FILE_XFER_SET::poll(): http op done; retval -184 mar 19 apr 2011 20:08:40 CEST RNA World [file_xfer_debug] file transfer status -184 Notice that the above WU is still waiting for my result: http://www.rnaworld.de/rnaworld/workuni ... id=5064684 And what's "file transfer status -184"? It seems an old problem: http://boinc.berkeley.edu/dev/forum_thread.php?id=4582 And here they say: it's a server problem http://boinc.berkeley.edu/dev/forum_thread.php?id=5541 Does anybody know anything about it? Bye. ID: 37528 · |
 Jord Send message Joined: 29 Aug 05 Posts: 15482 
|
ERR_HTTP_ERROR -184 Can you ask RNA what server revision they use? Can you also test with BOINC 6.12.22 (development version, available from here), to see if you can reproduce this problem? Only then can I forward it to the developers, since they're done with 6.10 and in the last days of the 6.12 release. With thanks. ID: 37532 · |
|
Send message Joined: 18 Apr 11 Posts: 6
|
Can you ask RNA what server revision they use? I'll surely do it tomorrow. For now, I have to say that one of my partially uploaded files has been deleted on the server. The next try, of course, the upload hasn't been resumed: it's been restarted from scratch. It has gone on for a while, but it has then stopped at the very exact point it did the first time. So it doesn't look like a "resuming issue" anymore. Still no clue. Bye. ID: 37534 · |
|
Send message Joined: 18 Apr 11 Posts: 6
|
I'm running now the latest 6.12.22 Boinc version lem@biggy:~$ boinc --version 6.12.22 x86_64-pc-linux-gnu Sadly, nothing has changed. In the meantime, I've looked a bit into curl & stuff, and I've found something that at first sight seemed possibly related to my problem: http://www.cherrypy.org/ticket/951 However IMHO this is not my case: the http_1_0 workaround doesn't work for me. To be sure, though I'm running Ubuntu Lucid, I've upgraded libcurl: lem@biggy:~$ curl -V curl 7.19.7 (x86_64-pc-linux-gnu) libcurl/7.21.3 OpenSSL/0.9.8o zlib/1.2.3.3 libidn/1.15 Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smtp smtps telnet tftp Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz Furthermore, since yesterday we know it shouldn't be a simple resuming issue: some WU results are uploaded just to a fixed point (different for every WU), and no more. Then the upload stops; it cannot be successfully resumed, and, if the partial file is deleted on the server, next time the upload will reach the same fixed point as before, and will then stop again. The first thing I thought of was something wrong in the "content lenght" declared to the server before the upload, but the lenght looks right. One more WU can't be fully uploaded now, and so they're seven till now (in a total of about 550). I don't have a clue. This is my log with 6.12.22: mer 20 apr 2011 22:05:02 CEST | RNA World | Scheduler request completed mer 20 apr 2011 22:05:37 CEST | RNA World | [fxd] starting upload, upload_offset -1 mer 20 apr 2011 22:05:37 CEST | RNA World | Started upload of cms_GA-p[MZ-Lin64s]_6_Methylococcus-capsulatus-str.-Bath_AE017282.cir.EMBL_1302888724_32529_1_0 mer 20 apr 2011 22:05:37 CEST | RNA World | [file_xfer] URL: http://www.rnaworld.de/rnaworld_cgi/file_upload_handler mer 20 apr 2011 22:05:37 CEST | | [http] [ID#27] Info: Connection #0 seems to be dead! mer 20 apr 2011 22:05:37 CEST | | [http] [ID#27] Info: Closing connection #0 mer 20 apr 2011 22:05:37 CEST | | [http] [ID#27] Info: About to connect() to www.rnaworld.de port 80 (#0) mer 20 apr 2011 22:05:37 CEST | | [http] [ID#27] Info: Trying 78.46.16.148... mer 20 apr 2011 22:05:37 CEST | | [http] [ID#27] Info: 0x158f1b0 is at send pipe head! mer 20 apr 2011 22:05:37 CEST | | [http] [ID#27] Info: Connected to www.rnaworld.de (78.46.16.148) port 80 (#0) mer 20 apr 2011 22:05:37 CEST | | [http] [ID#27] Sent header to server: POST /rnaworld_cgi/file_upload_handler HTTP/1.1 mer 20 apr 2011 22:05:37 CEST | | [http] [ID#27] Sent header to server: User-Agent: BOINC client (x86_64-pc-linux-gnu 6.12.22) mer 20 apr 2011 22:05:37 CEST | | [http] [ID#27] Sent header to server: Host: www.rnaworld.de mer 20 apr 2011 22:05:37 CEST | | [http] [ID#27] Sent header to server: Accept: */* mer 20 apr 2011 22:05:37 CEST | | [http] [ID#27] Sent header to server: Accept-Encoding: deflate, gzip mer 20 apr 2011 22:05:37 CEST | | [http] [ID#27] Sent header to server: Content-Type: application/x-www-form-urlencoded mer 20 apr 2011 22:05:37 CEST | | [http] [ID#27] Sent header to server: Content-Length: 349 mer 20 apr 2011 22:05:37 CEST | | [http] [ID#27] Sent header to server: mer 20 apr 2011 22:05:37 CEST | | [http] [ID#27] Received header from server: HTTP/1.1 200 OK mer 20 apr 2011 22:05:37 CEST | | [http] [ID#27] Received header from server: Date: Wed, 20 Apr 2011 20:05:40 GMT mer 20 apr 2011 22:05:37 CEST | | [http] [ID#27] Received header from server: Server: Apache/2.2.13 (Linux/SUSE) mer 20 apr 2011 22:05:37 CEST | | [http] [ID#27] Received header from server: Content-Length: 99 mer 20 apr 2011 22:05:37 CEST | | [http] [ID#27] Received header from server: Content-Type: text/plain mer 20 apr 2011 22:05:37 CEST | | [http] [ID#27] Received header from server: mer 20 apr 2011 22:05:37 CEST | | [http_xfer] [ID#27] HTTP: wrote 99 bytes mer 20 apr 2011 22:05:37 CEST | | [http] [ID#27] Info: Connection #0 to host www.rnaworld.de left intact mer 20 apr 2011 22:05:37 CEST | | [http] [ID#27] Info: Expire cleared mer 20 apr 2011 22:05:38 CEST | RNA World | [file_xfer] FILE_XFER_SET::poll(): http op done; retval 0 mer 20 apr 2011 22:05:38 CEST | RNA World | [file_xfer] parsing upload response: <data_server_reply> <status>0</status> <file_size>3522130</file_size></data_server_reply> mer 20 apr 2011 22:05:38 CEST | RNA World | [file_xfer] parsing status: 0 mer 20 apr 2011 22:05:38 CEST | RNA World | [fxd] starting upload, upload_offset 3522130 mer 20 apr 2011 22:05:38 CEST | | [http] [ID#27] Info: Re-using existing connection! (#0) with host www.rnaworld.de mer 20 apr 2011 22:05:38 CEST | | [http] [ID#27] Info: Connected to www.rnaworld.de (78.46.16.148) port 80 (#0) mer 20 apr 2011 22:05:38 CEST | | [http] [ID#27] Info: 0x158f1b0 is at send pipe head! mer 20 apr 2011 22:05:38 CEST | | [http] [ID#27] Sent header to server: POST /rnaworld_cgi/file_upload_handler HTTP/1.1 mer 20 apr 2011 22:05:38 CEST | | [http] [ID#27] Sent header to server: User-Agent: BOINC client (x86_64-pc-linux-gnu 6.12.22) mer 20 apr 2011 22:05:38 CEST | | [http] [ID#27] Sent header to server: Host: www.rnaworld.de mer 20 apr 2011 22:05:38 CEST | | [http] [ID#27] Sent header to server: Accept: */* mer 20 apr 2011 22:05:38 CEST | | [http] [ID#27] Sent header to server: Accept-Encoding: deflate, gzip mer 20 apr 2011 22:05:38 CEST | | [http] [ID#27] Sent header to server: Content-Type: application/x-www-form-urlencoded mer 20 apr 2011 22:05:38 CEST | | [http] [ID#27] Sent header to server: Content-Length: 4351083 mer 20 apr 2011 22:05:38 CEST | | [http] [ID#27] Sent header to server: Expect: 100-continue mer 20 apr 2011 22:05:38 CEST | | [http] [ID#27] Sent header to server: mer 20 apr 2011 22:05:38 CEST | | [http] [ID#27] Received header from server: HTTP/1.1 100 Continue [here my client apparently doesn't send anything, even if it should] mer 20 apr 2011 22:10:38 CEST | | [http] [ID#27] Received header from server: HTTP/1.1 500 Internal Server Error mer 20 apr 2011 22:10:38 CEST | | [http] [ID#27] Received header from server: Date: Wed, 20 Apr 2011 20:05:41 GMT mer 20 apr 2011 22:10:38 CEST | | [http] [ID#27] Received header from server: Server: Apache/2.2.13 (Linux/SUSE) mer 20 apr 2011 22:10:38 CEST | | [http] [ID#27] Received header from server: Vary: accept-language,accept-charset mer 20 apr 2011 22:10:38 CEST | | [http] [ID#27] Received header from server: Accept-Ranges: bytes mer 20 apr 2011 22:10:38 CEST | | [http] [ID#27] Received header from server: Connection: close mer 20 apr 2011 22:10:38 CEST | | [http] [ID#27] Received header from server: Content-Type: text/html; charset=iso-8859-1 mer 20 apr 2011 22:10:38 CEST | | [http] [ID#27] Received header from server: Content-Language: en mer 20 apr 2011 22:10:38 CEST | | [http] [ID#27] Received header from server: X-Pad: avoid browser bug mer 20 apr 2011 22:10:38 CEST | | [http] [ID#27] Received header from server: mer 20 apr 2011 22:10:38 CEST | | [http_xfer] [ID#27] HTTP: wrote 1071 bytes mer 20 apr 2011 22:10:39 CEST | | [http] [ID#27] Info: we are done reading and this is set to close, stop send mer 20 apr 2011 22:10:39 CEST | | [http] [ID#27] Info: Closing connection #0 mer 20 apr 2011 22:10:39 CEST | | [http] [ID#27] Info: Expire cleared mer 20 apr 2011 22:10:39 CEST | RNA World | [file_xfer] FILE_XFER_SET::poll(): http op done; retval -184 mer 20 apr 2011 22:10:39 CEST | RNA World | [file_xfer] file transfer status -184 mer 20 apr 2011 22:10:39 CEST | RNA World | Temporarily failed upload of cms_GA-p[MZ-Lin64s]_6_Methylococcus-capsulatus-str.-Bath_AE017282.cir.EMBL_1302888724_32529_1_0: HTTP error mer 20 apr 2011 22:10:39 CEST | RNA World | Backing off 10 hr 50 min 18 sec on upload of cms_GA-p[MZ-Lin64s]_6_Methylococcus-capsulatus-str.-Bath_AE017282.cir.EMBL_1302888724_32529_1_0 Thanks in advance for everything. Bye. ID: 37548 · |
|
Jord Send message Joined: 29 Aug 05 Posts: 15482
|
You only see this with RNAWorld, or also with other projects? I did find the server revision number, so don't need that info anymore. ID: 37550 · |
|
Send message Joined: 18 Apr 11 Posts: 6
|
You only see this with RNAWorld, or also with other projects? Only RNA World. Now I'm running RNA alone (but Einstein on my GPU), but till a few days ago also simap and four WCG projects: aids, conquercancer, cleanwater and cleanenergy2. This never happened with these. I've never had this problem before, with any other project, AFAICR. But I have to say that I don't know of any other RNA cruncher in my situation. I did find the server revision number, so don't need that info anymore. Fine, RNA message board is out of service at the moment. Bye. ID: 37551 · |
|
Jord Send message Joined: 29 Aug 05 Posts: 15482
|
Okay, I've had a conversation with the developers and we think this can be solved if Rechenkraft.net updates their BOINC server software. I will contact the administrator of that domain and ask on your behalf, no need for you to pursue this. But I cannot assure that they will update, let's just hope that they will. ID: 37554 · |
|
Send message Joined: 18 Apr 11 Posts: 6
|
Okay, I've had a conversation with the developers and we think this can be solved if Rechenkraft.net updates their BOINC server software. I will contact the administrator of that domain and ask on your behalf Thank you very much. It's the best I could hope for. :) no need for you to pursue this. Okie Dokie: I give it up... and I hope for the best. :) Thanks again, for everything. Goodbye. ID: 37555 · |
Copyright © 2024 University of California.
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License,
Version 1.2 or any later version published by the Free Software Foundation.