climateprediction.net home page
Temporarily failed upload

Temporarily failed upload

Questions and Answers : Getting started : Temporarily failed upload
Message board moderation

To post messages, you must log in.

AuthorMessage
Profile BOUCHAUDON

Send message
Joined: 23 Nov 05
Posts: 5
Credit: 368,421
RAC: 0
Message 46409 - Posted: 12 Jun 2013, 18:32:03 UTC

Hi,

The host uploader1.atm didn't run since more than one month. What's happen?
I have finished a task, and 3 uploads are stucked!

JM.

12/06/2013 20:06:36 | climateprediction.net | Started upload of hadam3p_eu_63v6_2007_1_007620010_2_11.zip
12/06/2013 20:06:38 | climateprediction.net | Temporarily failed upload of hadam3p_eu_63v6_2007_1_007620010_2_11.zip: transient HTTP error
12/06/2013 20:06:38 | climateprediction.net | Backing off 5 hr 7 min 22 sec on upload of hadam3p_eu_63v6_2007_1_007620010_2_11.zip

ID: 46409 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 46410 - Posted: 12 Jun 2013, 20:04:11 UTC - in response to Message 46409.  

That upload server IS running; it's just saying that it isn't.

There's a lot of these large models running at present, so the server may be busy, or it may have filled up again as discussed here. I'll ask again.



Backups: Here
ID: 46410 · Report as offensive     Reply Quote
Profile astroWX
Volunteer moderator

Send message
Joined: 5 Aug 04
Posts: 1496
Credit: 95,522,203
RAC: 0
Message 46411 - Posted: 12 Jun 2013, 20:46:00 UTC

My recent uploads (for HadCM3N) have been successful.

When an HTTP error continues for awhile, I find it necessary to reboot the box. It seems something in our boinc setup gets in the way and doesn't clear without rebooting. (Whatever works! [even if it makes no sense].)

"We have met the enemy and he is us." -- Pogo
Greetings from coastal Washington state, the scenic US Pacific Northwest.
ID: 46411 · Report as offensive     Reply Quote
Profile JIM

Send message
Joined: 31 Dec 07
Posts: 1152
Credit: 22,053,321
RAC: 4,417
Message 46412 - Posted: 13 Jun 2013, 4:41:03 UTC
Last modified: 13 Jun 2013, 4:41:39 UTC

My hadcm model uploaded successfully yesterday (12, June). So the system is working at least for some.
ID: 46412 · Report as offensive     Reply Quote
Profile BOUCHAUDON

Send message
Joined: 23 Nov 05
Posts: 5
Credit: 368,421
RAC: 0
Message 46425 - Posted: 15 Jun 2013, 9:07:12 UTC
Last modified: 15 Jun 2013, 9:10:08 UTC

I have always the problem. 3 intermediate uploads are stucked despite the reboot of my box. Curiusly, the last upload at the end of the job has been successfully uploaded but the WU is considered as in progress
http://climateapps2.oerc.ox.ac.uk/cpdnboinc/workunit.php?wuid=7798329

Is it a problem to try to upload intermediate result not in the right order?

I don't worry, I have until the 25 Nov 2013 (deadline) to find a solution ;-)
ID: 46425 · Report as offensive     Reply Quote
Profile Iain Inglis
Volunteer moderator

Send message
Joined: 16 Jan 10
Posts: 1079
Credit: 6,903,221
RAC: 6,722
Message 46426 - Posted: 15 Jun 2013, 22:37:54 UTC - in response to Message 46425.  

[BOUCHAUDON wrote:] Curiusly, the last upload at the end of the job has been successfully uploaded but the WU is considered as in progress
http://climateapps2.oerc.ox.ac.uk/cpdnboinc/workunit.php?wuid=7798329

This HADAM3P model has submitted 12 trickles (here) and has been granted all the expected credits. The model will be marked as complete when it is 'reported', which can only happen when all the uploads have finished.

Is it a problem to try to upload intermediate result not in the right order?

No, it is not a problem. Intermediate results are sometimes sent to different servers; in particular, the final result is often sent to a different server to the earlier results. The results may therefore be uploaded out of order if a server is unavailable for some reason.
ID: 46426 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 46427 - Posted: 16 Jun 2013, 8:54:46 UTC - in response to Message 46425.  

As with a lot of things on this project, the "deadline" isn't the one used by BOINC (which just results in an error message for that model), but the one imposed by the work schedule of the researchers running an experiment.
Which is: We need the results ASAP, so we can create the next run in the sequence.

Models which are tardy in being returned will definitely be added to the accumulating Grand Assemble of results for other research, but the one used by the researchers for their original work may well be from someone else's computer, which returned it sooner. It depends on the max # of error/total/success tasks settings for that model.

As for servers, different ideas have been tried over the years to spread the load. I think that the current one is: odd zips go to one server, and even zips go to a second server.
And zip file 13, which contains the data needed to create the next one in the sequence, goes to a third server.
For the PNW models, they all go directly to a server in Oregon, USA, except for zip 13, which goes to the afore mentioned server in Oxford.


If you take a long time to complete a model, (they can be completed in about 4.5 days on my machines), It's possible that a router somewhere in the network between you and Oxford has gone faulty, and been replaced by one with an older software version.
This can be solved by forcing your uploads to use an older version of HTTP. Use the following code for a cc_config.xml file in the BOINC data folder.

<cc_config>
<log_flags>
</log_flags>
<options>
<http_1_0>1</http_1_0>
</options>
</cc_config>

If you've never used this before, the details are here.



Backups: Here
ID: 46427 · Report as offensive     Reply Quote
Profile BOUCHAUDON

Send message
Joined: 23 Nov 05
Posts: 5
Credit: 368,421
RAC: 0
Message 46428 - Posted: 16 Jun 2013, 11:43:53 UTC - in response to Message 46427.  

Thank you Les Bayliss for your help,

I have create a cc_config.xml but it seems to have any effet on my problem. I have also add a log flag "http_debug" and the results are the same. You will find here after a try with the http_1_0 set and a try with http_1_1

16/06/2013 13:36:49 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle 'C:\Program Files\BOINC\ca-bundle.crt'
16/06/2013 13:36:49 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle set
16/06/2013 13:36:49 | climateprediction.net | Started upload of hadam3p_eu_63v6_2007_1_007620010_2_10.zip
16/06/2013 13:36:50 | climateprediction.net | [http] [ID#13] Info: About to connect() to uploader1.atm.ox.ac.uk port 80 (#0)
16/06/2013 13:36:50 | climateprediction.net | [http] [ID#13] Info: Trying 129.67.195.183...
16/06/2013 13:36:50 | climateprediction.net | [http] [ID#13] Info: connected
16/06/2013 13:36:50 | climateprediction.net | [http] [ID#13] Info: Connected to uploader1.atm.ox.ac.uk (129.67.195.183) port 80 (#0)
16/06/2013 13:36:50 | climateprediction.net | [http] [ID#13] Sent header to server: POST /cpdn_cgi/file_upload_handler HTTP/1.0
16/06/2013 13:36:50 | climateprediction.net | [http] [ID#13] Sent header to server: User-Agent: BOINC client (windows_intelx86 7.0.64)
16/06/2013 13:36:50 | climateprediction.net | [http] [ID#13] Sent header to server: Host: uploader1.atm.ox.ac.uk
16/06/2013 13:36:50 | climateprediction.net | [http] [ID#13] Sent header to server: Accept: */*
16/06/2013 13:36:50 | climateprediction.net | [http] [ID#13] Sent header to server: Accept-Encoding: deflate, gzip
16/06/2013 13:36:50 | climateprediction.net | [http] [ID#13] Sent header to server: Content-Type: application/x-www-form-urlencoded
16/06/2013 13:36:50 | climateprediction.net | [http] [ID#13] Sent header to server: Content-Length: 294
16/06/2013 13:36:50 | climateprediction.net | [http] [ID#13] Sent header to server:
16/06/2013 13:36:50 | climateprediction.net | [http] [ID#13] Received header from server: HTTP/1.1 500 Internal Server Error
16/06/2013 13:36:50 | climateprediction.net | [http] [ID#13] Received header from server: Date: Sun, 16 Jun 2013 11:36:52 GMT
16/06/2013 13:36:50 | climateprediction.net | [http] [ID#13] Received header from server: Server: Apache/2.2.22 (Ubuntu)
16/06/2013 13:36:50 | climateprediction.net | [http] [ID#13] Received header from server: Content-Length: 635
16/06/2013 13:36:50 | climateprediction.net | [http] [ID#13] Received header from server: Connection: close
16/06/2013 13:36:50 | climateprediction.net | [http] [ID#13] Received header from server: Content-Type: text/html; charset=iso-8859-1
16/06/2013 13:36:50 | climateprediction.net | [http] [ID#13] Received header from server:
16/06/2013 13:36:50 | climateprediction.net | [http] [ID#13] Info: Closing connection #0
16/06/2013 13:36:51 | climateprediction.net | Temporarily failed upload of hadam3p_eu_63v6_2007_1_007620010_2_10.zip: transient HTTP error
16/06/2013 13:36:51 | climateprediction.net | Backing off 5 hr 17 min 59 sec on upload of hadam3p_eu_63v6_2007_1_007620010_2_10.zip
16/06/2013 13:39:11 | | Re-reading cc_config.xml
16/06/2013 13:39:11 | | Not using a proxy
16/06/2013 13:39:11 | | log flags: file_xfer, sched_ops, task, http_debug
16/06/2013 13:39:16 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle 'C:\Program Files\BOINC\ca-bundle.crt'
16/06/2013 13:39:16 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle set
16/06/2013 13:39:16 | climateprediction.net | Started upload of hadam3p_eu_63v6_2007_1_007620010_2_10.zip
16/06/2013 13:39:16 | climateprediction.net | [http] [ID#14] Info: About to connect() to uploader1.atm.ox.ac.uk port 80 (#0)
16/06/2013 13:39:16 | climateprediction.net | [http] [ID#14] Info: Trying 129.67.195.183...
16/06/2013 13:39:17 | climateprediction.net | [http] [ID#14] Info: Connected to uploader1.atm.ox.ac.uk (129.67.195.183) port 80 (#0)
16/06/2013 13:39:17 | climateprediction.net | [http] [ID#14] Info: Connected to uploader1.atm.ox.ac.uk (129.67.195.183) port 80 (#0)
16/06/2013 13:39:17 | climateprediction.net | [http] [ID#14] Sent header to server: POST /cpdn_cgi/file_upload_handler HTTP/1.1
16/06/2013 13:39:17 | climateprediction.net | [http] [ID#14] Sent header to server: User-Agent: BOINC client (windows_intelx86 7.0.64)
16/06/2013 13:39:17 | climateprediction.net | [http] [ID#14] Sent header to server: Host: uploader1.atm.ox.ac.uk
16/06/2013 13:39:17 | climateprediction.net | [http] [ID#14] Sent header to server: Accept: */*
16/06/2013 13:39:17 | climateprediction.net | [http] [ID#14] Sent header to server: Accept-Encoding: deflate, gzip
16/06/2013 13:39:17 | climateprediction.net | [http] [ID#14] Sent header to server: Content-Type: application/x-www-form-urlencoded
16/06/2013 13:39:17 | climateprediction.net | [http] [ID#14] Sent header to server: Content-Length: 294
16/06/2013 13:39:17 | climateprediction.net | [http] [ID#14] Sent header to server:
16/06/2013 13:39:18 | climateprediction.net | [http] [ID#14] Received header from server: HTTP/1.1 500 Internal Server Error
16/06/2013 13:39:18 | climateprediction.net | [http] [ID#14] Received header from server: Date: Sun, 16 Jun 2013 11:39:19 GMT
16/06/2013 13:39:18 | climateprediction.net | [http] [ID#14] Received header from server: Server: Apache/2.2.22 (Ubuntu)
16/06/2013 13:39:18 | climateprediction.net | [http] [ID#14] Received header from server: Content-Length: 635
16/06/2013 13:39:18 | climateprediction.net | [http] [ID#14] Received header from server: Connection: close
16/06/2013 13:39:18 | climateprediction.net | [http] [ID#14] Received header from server: Content-Type: text/html; charset=iso-8859-1
16/06/2013 13:39:18 | climateprediction.net | [http] [ID#14] Received header from server:
16/06/2013 13:39:18 | climateprediction.net | [http] [ID#14] Info: Closing connection #0
16/06/2013 13:39:18 | climateprediction.net | Temporarily failed upload of hadam3p_eu_63v6_2007_1_007620010_2_10.zip: transient HTTP error
16/06/2013 13:39:18 | climateprediction.net | Backing off 4 hr 57 min 45 sec on upload of hadam3p_eu_63v6_2007_1_007620010_2_10.zip

ID: 46428 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 46431 - Posted: 16 Jun 2013, 13:26:48 UTC - in response to Message 46428.  

There's now another poster having the same problems, so I'm going to email the project.
Middle of the weekend though. :(


Backups: Here
ID: 46431 · Report as offensive     Reply Quote
Profile BOUCHAUDON

Send message
Joined: 23 Nov 05
Posts: 5
Credit: 368,421
RAC: 0
Message 46432 - Posted: 16 Jun 2013, 16:23:06 UTC - in response to Message 46431.  

Thanks Les Bayliss,

The problem occurs on May the 5th, it can wait a little bit more ;-).

JM.
ID: 46432 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 46437 - Posted: 17 Jun 2013, 20:05:12 UTC

Server problem. They're looking into it.


Backups: Here
ID: 46437 · Report as offensive     Reply Quote
Andy Bowery
Volunteer moderator
Project administrator

Send message
Joined: 11 Feb 11
Posts: 13
Credit: 53,909
RAC: 0
Message 46440 - Posted: 18 Jun 2013, 10:59:42 UTC

Hi,

This issue is now fixed, the issue was with the file_upload_handler script on the server uploader1.atm, the script has now been recompiled and is functioning correctly.

Regards,

Andy
ID: 46440 · Report as offensive     Reply Quote
Profile BOUCHAUDON

Send message
Joined: 23 Nov 05
Posts: 5
Credit: 368,421
RAC: 0
Message 46461 - Posted: 19 Jun 2013, 18:46:52 UTC

The 3 files have been correctly uploaded this evening. Thanks to all for your help. I close the topic.
ID: 46461 · Report as offensive     Reply Quote
elect

Send message
Joined: 28 Jul 11
Posts: 2
Credit: 61,196
RAC: 0
Message 50259 - Posted: 22 Sep 2014, 13:06:46 UTC
Last modified: 22 Sep 2014, 13:07:01 UTC

22/09/2014 14:47:40 | climateprediction.net | Backing off 03:21:48 on upload of hadcm3s_275c_1991_2_009017239_0_2.zip
22/09/2014 14:47:54 | | Project communication failed: attempting access to reference site
22/09/2014 14:47:56 | | Internet access OK - project servers may be temporarily down.



Is it really down or is something else?
ID: 50259 · Report as offensive     Reply Quote
Profile Iain Inglis
Volunteer moderator

Send message
Joined: 16 Jan 10
Posts: 1079
Credit: 6,903,221
RAC: 6,722
Message 50263 - Posted: 22 Sep 2014, 14:29:48 UTC - in response to Message 50259.  

22/09/2014 14:47:40 | climateprediction.net | Backing off 03:21:48 on upload of hadcm3s_275c_1991_2_009017239_0_2.zip
22/09/2014 14:47:54 | | Project communication failed: attempting access to reference site
22/09/2014 14:47:56 | | Internet access OK - project servers may be temporarily down.



Is it really down or is something else?

The ANOTHER UPLOAD PROBLEM thread discusses various types of upload failures, some transient others longer term.
ID: 50263 · Report as offensive     Reply Quote

Questions and Answers : Getting started : Temporarily failed upload

©2024 climateprediction.net