climateprediction.net home page
Upload Failure

Upload Failure

Message boards : Number crunching : Upload Failure
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 6 · 7 · 8 · 9 · 10 · Next

AuthorMessage
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 44389 - Posted: 13 Jun 2012, 3:52:19 UTC

My large collection have now all uploaded, so things have improved.


Backups: Here
ID: 44389 · Report as offensive     Reply Quote
Digby

Send message
Joined: 17 Feb 06
Posts: 89
Credit: 4,309,159
RAC: 0
Message 44390 - Posted: 13 Jun 2012, 7:52:37 UTC
Last modified: 13 Jun 2012, 8:10:32 UTC

Yes all my files have now uploaded. Yesterday I actually got error messages from the server before the upload started...I had planned to send in those server error messages but I got sidetracked....

So all files are uploaded but the Project has no tasks to work on...I hope that changes soon...
ID: 44390 · Report as offensive     Reply Quote
marpes

Send message
Joined: 11 Nov 04
Posts: 8
Credit: 15,267,364
RAC: 0
Message 44391 - Posted: 13 Jun 2012, 13:02:19 UTC

Upload all of .zip files is OK.
Some PCs have tasks for 3-4 days only. Hopefully, the new job soon.
ID: 44391 · Report as offensive     Reply Quote
glaesum

Send message
Joined: 24 Feb 06
Posts: 47
Credit: 782,082
RAC: 0
Message 44392 - Posted: 13 Jun 2012, 14:11:59 UTC - in response to Message 44391.  

hi Greg, all's well. In the small hours I was just about to shut down network activity overnight when I found everything had finally uploaded ok. I think it was several people reporting success in the previous hours that got me slightly concerned while it looked like the servers all had green flags.

let's hope the long period of recurrent server problems is eventually overcome.
now I just need to clear the cache of SIMAPs before returning to my HADAM eu models! :)
ID: 44392 · Report as offensive     Reply Quote
nairb

Send message
Joined: 3 Sep 04
Posts: 105
Credit: 5,646,090
RAC: 102,785
Message 44410 - Posted: 15 Jun 2012, 14:37:34 UTC

Its all Green...... but uploads not playing again with:-

15/06/2012 15:33:17 | climateprediction.net | Temporarily failed upload of hadam3p_eu_d9gm_2008_1_007978472_0_6.zip: transient upload error

Maybe it will become UNtransient in a while.

Nairb
<img border="0" src="http://boinc.mundayweb.com/one/stats.php?userID=343" />
ID: 44410 · Report as offensive     Reply Quote
Profile old_user651284

Send message
Joined: 28 Mar 11
Posts: 35
Credit: 82,588
RAC: 0
Message 44411 - Posted: 15 Jun 2012, 15:15:18 UTC
Last modified: 18 Jun 2012, 9:17:51 UTC

EDIT: This issue has now been resolved, and problems encountered as of 10.00 am BST on 18 June 2012 are likely to be due to the servers struggling to catch up with a weekend of uploads.

Sorry everyone, but the data centre in which we host our servers has had a mystery network problem since Thursday 14 June 15:45 GMT.

The problem manifests as intermittent loss of network within the data centre, which means that the project servers cannot communicate with oneanother.

The error below is because the upload server you are trying to use cannot see the network file storage that it want to write your result to.

I have taken cpdn-upload2.oerc and cpdn-restarts.oerc offine until this is resolved.

I am dependent upon other people to look at this issue, and it is now gone 4 pm on the last day of University term, so I don't hold out any hope at all of a fix until Monday.

Sorry.

Jonathan
ID: 44411 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4352
Credit: 16,574,226
RAC: 5,569
Message 44428 - Posted: 18 Jun 2012, 10:29:45 UTC

Last of 6 uploads almost finished now, so looks like all is sorted.

Dave
ID: 44428 · Report as offensive     Reply Quote
nairb

Send message
Joined: 3 Sep 04
Posts: 105
Credit: 5,646,090
RAC: 102,785
Message 44433 - Posted: 18 Jun 2012, 22:43:52 UTC

Works for me too.
<img border="0" src="http://boinc.mundayweb.com/one/stats.php?userID=343" />
ID: 44433 · Report as offensive     Reply Quote
Steve Camilleri

Send message
Joined: 27 Nov 05
Posts: 4
Credit: 375,040
RAC: 959
Message 44434 - Posted: 19 Jun 2012, 19:09:55 UTC

still seems to be down for me:
19/06/2012 21:08:19 | climateprediction.net | project resumed by user
19/06/2012 21:08:23 | climateprediction.net | Started upload of hadam3p_eu_4j8v_1999_1_007309876_1_13.zip
19/06/2012 21:08:25 | climateprediction.net | Temporarily failed upload of hadam3p_eu_4j8v_1999_1_007309876_1_13.zip: can't resolve hostname
19/06/2012 21:08:25 | climateprediction.net | Backing off 3 hr 4 min 35 sec on upload of hadam3p_eu_4j8v_1999_1_007309876_1_13.zip
19/06/2012 21:08:28 | | Project communication failed: attempting access to reference site
19/06/2012 21:08:31 | | Internet access OK - project servers may be temporarily down.
ID: 44434 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 44435 - Posted: 19 Jun 2012, 21:32:53 UTC - in response to Message 44434.  

The servers have been working OK for some time now, so your problem must be due to a different cause.

How many files are stuck?
Have you shut down both parts of BOINC, and then restarted them?
Have you tried the above, and restarted the computer while BOINC was stopped?


Backups: Here
ID: 44435 · Report as offensive     Reply Quote
Profile Thyme Lawn
Volunteer moderator

Send message
Joined: 5 Aug 04
Posts: 1283
Credit: 15,824,334
RAC: 0
Message 44437 - Posted: 20 Jun 2012, 7:06:35 UTC - in response to Message 44434.  

19/06/2012 21:08:25 | climateprediction.net | Temporarily failed upload of hadam3p_eu_4j8v_1999_1_007309876_1_13.zip: can't resolve hostname

That task (the second one listed for WU hadam3p_eu_4j8v_1999_1_007309876) was issued on 27th June 2011. I wonder if the upload file is being sent to a system on the oucs.ox.ac.uk network rather than the oerc.ox.ac.uk network. A redirect was in place to handle that, but if that's not working it would explain the hostname resolution failure.

To find out exactly what's happening create the file cc_config.xml (using a plain text editor, e.g. notepad rather than wordpad) in your BOINC data directory containing the following:

<cc_config>
<log_flags>
<http_debug>1</http_debug>
</log_flags>
</cc_config>

In BOINC Manager's advanced view click Advanced - Read config file, force an upload retry on the Transfers tab, and post the resulting HTTP debug messages. Once you've got the messages you shpuld change the <http_debug> value in the file from 1 to 0 and reload the config file to disable the debug messages; once you've done that you can delete the cc_config.xml.
"The ultimate test of a moral society is the kind of world that it leaves to its children." - Dietrich Bonhoeffer
ID: 44437 · Report as offensive     Reply Quote
Steve Camilleri

Send message
Joined: 27 Nov 05
Posts: 4
Credit: 375,040
RAC: 959
Message 44460 - Posted: 25 Jun 2012, 23:54:04 UTC - in response to Message 44437.  

Thanks Thyme Lawn seems like you nailed it. Event Log with debug on below.
Question is, do I dump the WU or can it be sent somewhere useful?

26/06/2012 01:47:06 | | Re-reading cc_config.xml
26/06/2012 01:47:06 | | log flags: file_xfer, sched_ops, task, http_debug
26/06/2012 01:47:15 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle 'C:\Program Files\BOINC\ca-bundle.crt'
26/06/2012 01:47:15 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle set
26/06/2012 01:47:15 | climateprediction.net | Started upload of hadam3p_eu_4j8v_1999_1_007309876_1_13.zip
26/06/2012 01:47:16 | climateprediction.net | [http] [ID#6] Info: Connection #0 seems to be dead!
26/06/2012 01:47:16 | climateprediction.net | [http] [ID#6] Info: Closing connection #0
26/06/2012 01:47:16 | climateprediction.net | [http] [ID#6] Info: Connection #1 seems to be dead!
26/06/2012 01:47:16 | climateprediction.net | [http] [ID#6] Info: Closing connection #1
26/06/2012 01:47:16 | climateprediction.net | [http] [ID#6] Info: Could not resolve host: climateapps1.oucs.ox.ac.uk; Host not found
26/06/2012 01:47:16 | climateprediction.net | [http] [ID#6] Info: Closing connection #0
26/06/2012 01:47:16 | climateprediction.net | [http] HTTP error: Couldn't resolve host name
26/06/2012 01:47:17 | climateprediction.net | Temporarily failed upload of hadam3p_eu_4j8v_1999_1_007309876_1_13.zip: can't resolve hostname
26/06/2012 01:47:17 | climateprediction.net | Backing off 5 hr 37 min 15 sec on upload of hadam3p_eu_4j8v_1999_1_007309876_1_13.zip
26/06/2012 01:47:32 | | Project communication failed: attempting access to reference site
26/06/2012 01:47:32 | | [http] HTTP_OP::init_get(): http://www.google.com/
26/06/2012 01:47:32 | | [http] HTTP_OP::libcurl_exec(): ca-bundle set
26/06/2012 01:47:33 | | [http] [ID#0] Info: About to connect() to www.google.com port 80 (#0)
26/06/2012 01:47:33 | | [http] [ID#0] Info: Trying 173.194.35.49...
26/06/2012 01:47:33 | | [http] [ID#0] Info: Connected to www.google.com (173.194.35.49) port 80 (#0)
ID: 44460 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 44461 - Posted: 26 Jun 2012, 0:31:22 UTC - in response to Message 44460.  

I'll pass on a note to the project, but it's still night time there at the moment.


Backups: Here
ID: 44461 · Report as offensive     Reply Quote
Profile old_user651284

Send message
Joined: 28 Mar 11
Posts: 35
Credit: 82,588
RAC: 0
Message 44465 - Posted: 26 Jun 2012, 14:18:27 UTC - in response to Message 44460.  
Last modified: 26 Jun 2012, 14:21:16 UTC

We are no longer allowed to use the DNS names containing .OUCS
We have moved over to using .OERC.

We were graciously allowed to use the domain name for 9 months during the transition period. My requests for a transition period of a year were vetoed by managers in my department, for their own inexplicable reasons.

If you are trying to upload to climateapps1.oucs.ox.ac.uk then you now need

cpdn-restarts.oerc.ox.ac.uk

129.67.195.121

If you could convince your machine (perhaps through /etc/hosts on a linux machine) to use this address instead, then you can upload it.

Otherwise, I am afraid you will have to trash it.

Jonathan
ID: 44465 · Report as offensive     Reply Quote
Profile Thyme Lawn
Volunteer moderator

Send message
Joined: 5 Aug 04
Posts: 1283
Credit: 15,824,334
RAC: 0
Message 44466 - Posted: 26 Jun 2012, 20:28:42 UTC - in response to Message 44465.  

Alternatively you could stop BOINC, edit the client_state.xml file to change every occurrence of climateapps1.oucs.ox.ac.uk to cpdn-restarts.oerc.ox.ac.uk and restart BOINC.

The project has disabled upload signatures (forced by changes in BOINC 7.*) so you shouldn't have to worry about leaving anything inside <signed_xml></signed_xml> blocks unchanged.
"The ultimate test of a moral society is the kind of world that it leaves to its children." - Dietrich Bonhoeffer
ID: 44466 · Report as offensive     Reply Quote
Digby

Send message
Joined: 17 Feb 06
Posts: 89
Credit: 4,309,159
RAC: 0
Message 44467 - Posted: 27 Jun 2012, 10:15:02 UTC
Last modified: 27 Jun 2012, 10:16:34 UTC

Ok I am a bit confused...

For a long time I had 8 models crunching. I was happy to do the 400hr models but for separate reasons on three occasions in the past 12 months I had a Windoze 7 blue screen of death and this unfortunately trashed some of those models. I was also happy to do the 100hr models because there was a reduced chance of corrupting models.

Recently there have been problems uploading results and lengthy delays to get this resolved. Once this upload was achieved there was a paucity of models to download but eventually after a week I got two models.

I have kept BOINC internet access open all the time to try and get new models to maximise my processor, but to no avail. This morning the 2 models I have tried to upload three times (I have to stop my internet access while this happens) but they errored:

<code>27/06/2012 09:39:43 climateprediction.net Started upload of hadam3p_eu_99fs_1959_1_007732288_2_1.zip
27/06/2012 09:39:52 climateprediction.net Started upload of hadam3p_eu_ct11_1999_1_007995766_1_1.zip
27/06/2012 09:49:09 climateprediction.net [error] Error reported by file upload server: can't write file /storage/incoming/uploader/hadam3p_eu_99fs_1959_1_007732288_2_1.zip: Input/output error
27/06/2012 09:49:09 climateprediction.net Temporarily failed upload of hadam3p_eu_99fs_1959_1_007732288_2_1.zip: transient upload error
27/06/2012 09:49:09 climateprediction.net Backing off 1 min 0 sec on upload of hadam3p_eu_99fs_1959_1_007732288_2_1.zip
27/06/2012 09:49:24 climateprediction.net [error] Error reported by file upload server: can't write file /storage/incoming/uploader/hadam3p_eu_ct11_1999_1_007995766_1_1.zip: Input/output error
27/06/2012 09:49:24 climateprediction.net Temporarily failed upload of hadam3p_eu_ct11_1999_1_007995766_1_1.zip: transient upload error
27/06/2012 09:49:24 climateprediction.net Backing off 1 min 0 sec on upload of hadam3p_eu_ct11_1999_1_007995766_1_1.zip
27/06/2012 09:50:10 climateprediction.net Started upload of hadam3p_eu_99fs_1959_1_007732288_2_1.zip
27/06/2012 09:50:24 climateprediction.net Started upload of hadam3p_eu_ct11_1999_1_007995766_1_1.zip
27/06/2012 09:59:30 climateprediction.net [error] Error reported by file upload server: can't write file /storage/incoming/uploader/hadam3p_eu_99fs_1959_1_007732288_2_1.zip: Input/output error
27/06/2012 09:59:30 climateprediction.net Temporarily failed upload of hadam3p_eu_99fs_1959_1_007732288_2_1.zip: transient upload error
27/06/2012 09:59:30 climateprediction.net Backing off 1 min 0 sec on upload of hadam3p_eu_99fs_1959_1_007732288_2_1.zip
27/06/2012 09:59:50 climateprediction.net [error] Error reported by file upload server: can't write file /storage/incoming/uploader/hadam3p_eu_ct11_1999_1_007995766_1_1.zip: Input/output error
27/06/2012 09:59:50 climateprediction.net Temporarily failed upload of hadam3p_eu_ct11_1999_1_007995766_1_1.zip: transient upload error
27/06/2012 09:59:50 climateprediction.net Backing off 1 min 0 sec on upload of hadam3p_eu_ct11_1999_1_007995766_1_1.zip
27/06/2012 10:00:50 climateprediction.net Started upload of hadam3p_eu_99fs_1959_1_007732288_2_1.zip
27/06/2012 10:00:50 climateprediction.net Started upload of hadam3p_eu_ct11_1999_1_007995766_1_1.zip
27/06/2012 10:10:29 climateprediction.net [error] Error reported by file upload server: can't write file /storage/incoming/uploader/hadam3p_eu_ct11_1999_1_007995766_1_1.zip: Input/output error
27/06/2012 10:10:29 climateprediction.net Temporarily failed upload of hadam3p_eu_ct11_1999_1_007995766_1_1.zip: transient upload error
27/06/2012 10:10:29 climateprediction.net Backing off 1 min 0 sec on upload of hadam3p_eu_ct11_1999_1_007995766_1_1.zip
27/06/2012 10:10:31 climateprediction.net [error] Error reported by file upload server: can't write file /storage/incoming/uploader/hadam3p_eu_99fs_1959_1_007732288_2_1.zip: Input/output error
27/06/2012 10:10:31 climateprediction.net Temporarily failed upload of hadam3p_eu_99fs_1959_1_007732288_2_1.zip: transient upload error
27/06/2012 10:10:31 climateprediction.net Backing off 1 min 0 sec on upload of hadam3p_eu_99fs_1959_1_007732288_2_1.zip </code>

Ques:
1) Is this temporary or do I have to make changes within the BOINC setup?
2) Is it my imagination or is the climate change project a bit 'rocky' at the moment?

If the answer to ques 2 is yes then that would be a shame. Is it short of money? Is there not a grant to do this research?

Climate Change is a real threat and we need to have analytical ammunition to fight the corporates and politicians who want to carry on with 'business as usual' and trash the planet.

Cheers

BTW apologies for this broad post.
ID: 44467 · Report as offensive     Reply Quote
MarkJ
Avatar

Send message
Joined: 28 Mar 09
Posts: 126
Credit: 9,825,980
RAC: 0
Message 44468 - Posted: 27 Jun 2012, 10:20:25 UTC
Last modified: 27 Jun 2012, 10:22:30 UTC

I am getting the same error, except I am doing pnw models. I manages to get to 100% and then fails.

27/06/2012 6:37:29 PM | climateprediction.net | [error] Error reported by file upload server: can't write file /storage/cpdn-restarts/incoming/uploader/hadam3p_pnw_b9c1_1977_1_008004972_0_13.zip: Input/output error
27/06/2012 6:37:29 PM | climateprediction.net | Temporarily failed upload of hadam3p_pnw_b9c1_1977_1_008004972_0_13.zip: transient upload error


Maybe its run out of room again?
BOINC blog
ID: 44468 · Report as offensive     Reply Quote
MarkJ
Avatar

Send message
Joined: 28 Mar 09
Posts: 126
Credit: 9,825,980
RAC: 0
Message 44469 - Posted: 27 Jun 2012, 10:51:59 UTC


27/06/2012 8:47:03 PM | climateprediction.net | [error] Error reported by file upload server: can't write file /storage/cpdn-restarts/incoming/uploader/hadam3p_pnw_bcry_1998_1_008005674_0_13.zip: Input/output error
27/06/2012 8:47:03 PM | climateprediction.net | Temporarily failed upload of hadam3p_pnw_bcry_1998_1_008005674_0_13.zip: transient upload error
27/06/2012 8:47:03 PM | climateprediction.net | Backing off 3 min 47 sec on upload of hadam3p_pnw_bcry_1998_1_008005674_0_13.zip
27/06/2012 8:47:11 PM | climateprediction.net | [error] Error reported by file upload server: can't write file /storage/cpdn-restarts/incoming/uploader/hadam3p_pnw_b9c1_1977_1_008004972_0_13.zip: Input/output error
27/06/2012 8:47:11 PM | climateprediction.net | Temporarily failed upload of hadam3p_pnw_b9c1_1977_1_008004972_0_13.zip: transient upload error


Mine are uploading for 10-15 minutes and then failing at the 100% mark, then going into retry where they waste another 15 minutes trying to upload again and so on and so on. A total waste of bandwidth.
BOINC blog
ID: 44469 · Report as offensive     Reply Quote
glaesum

Send message
Joined: 24 Feb 06
Posts: 47
Credit: 782,082
RAC: 0
Message 44470 - Posted: 27 Jun 2012, 13:15:55 UTC - in response to Message 44469.  

same here on my old machine with a European model. (just noticed boinc not running on the other machine so no news from that machine until it runs for a bit).
ID: 44470 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 44471 - Posted: 27 Jun 2012, 15:46:28 UTC

New problem people. Patience.


Backups: Here
ID: 44471 · Report as offensive     Reply Quote
Previous · 1 . . . 6 · 7 · 8 · 9 · 10 · Next

Message boards : Number crunching : Upload Failure

©2024 climateprediction.net