climateprediction.net home page
A different upload problem -- out of disk space on rapid-watch.badc.rl.ac.uk

A different upload problem -- out of disk space on rapid-watch.badc.rl.ac.uk

Message boards : Number crunching : A different upload problem -- out of disk space on rapid-watch.badc.rl.ac.uk
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · Next

AuthorMessage
Richard Haselgrove

Send message
Joined: 1 Jan 07
Posts: 943
Credit: 34,180,510
RAC: 6,451
Message 47356 - Posted: 20 Oct 2013, 10:13:18 UTC - in response to Message 47355.  

I've had uploads complete since you first reported this problem.

What happens if you paste

http://rapid-watch.badc.rl.ac.uk/cpdn_cgi/file_upload_handler

into a browser address bar?

I get

<data_server_reply>
    <status>1</status>
    <message>no command</message>
</data_server_reply>

suggesting that the server is alive and waiting for me.
ID: 47356 · Report as offensive     Reply Quote
alvin

Send message
Joined: 12 Mar 12
Posts: 29
Credit: 666,199
RAC: 0
Message 47357 - Posted: 20 Oct 2013, 12:17:43 UTC - in response to Message 47356.  

I've had uploads complete since you first reported this problem.

What happens if you paste

http://rapid-watch.badc.rl.ac.uk/cpdn_cgi/file_upload_handler

into a browser address bar?

I get

<data_server_reply>
    <status>1</status>
    <message>no command</message>
</data_server_reply>

suggesting that the server is alive and waiting for me.


thanks for clue
I get this from different computers - tested 3 of them
<data_server_reply>
<status>1</status>
<message>no command</message>
</data_server_reply>

ID: 47357 · Report as offensive     Reply Quote
MarkJ
Avatar

Send message
Joined: 28 Mar 09
Posts: 126
Credit: 9,825,980
RAC: 0
Message 47363 - Posted: 20 Oct 2013, 21:15:52 UTC - in response to Message 47356.  
Last modified: 20 Oct 2013, 21:23:59 UTC

I've had uploads complete since you first reported this problem.

What happens if you paste

http://rapid-watch.badc.rl.ac.uk/cpdn_cgi/file_upload_handler

into a browser address bar?

I get

<data_server_reply>
    <status>1</status>
    <message>no command</message>
</data_server_reply>

suggesting that the server is alive and waiting for me.

I haven't tried but then all the uploads start, they simply fail with the transient http error. I did post the http debug output in the other message thread about Rapit tasks. I have one upload in particular that regularly makes it to 91% before failing. I have also tried using uk-based proxy servers.

I would hate to waste the 350 days worth of CPU time because I can't upload them.
BOINC blog
ID: 47363 · Report as offensive     Reply Quote
Alex Plantema

Send message
Joined: 3 Sep 04
Posts: 126
Credit: 26,363,193
RAC: 0
Message 47486 - Posted: 7 Nov 2013, 7:43:20 UTC

The disk is full again:
07-11-2013 05:55:18 | climateprediction.net | Started upload of hadcm3n_7zdj_1980_40_008457130_1_2.zip
07-11-2013 05:55:21 | climateprediction.net | [error] Error reported by file upload server: Server is out of disk space
07-11-2013 05:55:21 | climateprediction.net | Temporarily failed upload of hadcm3n_7zdj_1980_40_008457130_1_2.zip: transient upload error
ID: 47486 · Report as offensive     Reply Quote
SandJ

Send message
Joined: 4 Jan 07
Posts: 9
Credit: 575,832
RAC: 0
Message 47487 - Posted: 7 Nov 2013, 7:44:48 UTC

Getting "Server is out of disk space" messages:

Thu 07 Nov 2013 02:41:52 GMT | climateprediction.net | Sending scheduler request: To send trickle-up message.
Thu 07 Nov 2013 02:41:52 GMT | climateprediction.net | Not requesting tasks: project is not highest priority
Thu 07 Nov 2013 02:41:57 GMT | climateprediction.net | Scheduler request completed
Thu 07 Nov 2013 02:42:16 GMT | climateprediction.net | Started upload of hadcm3n_o93c_1900_40_008466907_0_4.zip
Thu 07 Nov 2013 02:42:17 GMT | climateprediction.net | [error] Error reported by file upload server: Server is out of disk space
Thu 07 Nov 2013 02:42:17 GMT | climateprediction.net | Temporarily failed upload of hadcm3n_o93c_1900_40_008466907_0_4.zip: transient upload error
Thu 07 Nov 2013 02:42:17 GMT | climateprediction.net | Backing off 2 min 33 sec on upload of hadcm3n_o93c_1900_40_008466907_0_4.zip
Thu 07 Nov 2013 02:44:51 GMT | climateprediction.net | Started upload of hadcm3n_o93c_1900_40_008466907_0_4.zip
Thu 07 Nov 2013 02:44:52 GMT | climateprediction.net | [error] Error reported by file upload server: Server is out of disk space
Thu 07 Nov 2013 02:44:52 GMT | climateprediction.net | Temporarily failed upload of hadcm3n_o93c_1900_40_008466907_0_4.zip: transient upload error
Thu 07 Nov 2013 02:44:52 GMT | climateprediction.net | Backing off 4 min 47 sec on upload of hadcm3n_o93c_1900_40_008466907_0_4.zip
Thu 07 Nov 2013 02:49:40 GMT | climateprediction.net | Started upload of hadcm3n_o93c_1900_40_008466907_0_4.zip
Thu 07 Nov 2013 02:49:41 GMT | climateprediction.net | [error] Error reported by file upload server: Server is out of disk space
Thu 07 Nov 2013 02:49:41 GMT | climateprediction.net | Temporarily failed upload of hadcm3n_o93c_1900_40_008466907_0_4.zip: transient upload error
Thu 07 Nov 2013 02:49:41 GMT | climateprediction.net | Backing off 8 min 11 sec on upload of hadcm3n_o93c_1900_40_008466907_0_4.zip
Thu 07 Nov 2013 02:55:22 GMT | climateprediction.net | Computation for task hadcm3n_o93c_1900_40_008466907_0 finished
Thu 07 Nov 2013 02:59:36 GMT | climateprediction.net | Started upload of hadcm3n_o93c_1900_40_008466907_0_4.zip
Thu 07 Nov 2013 02:59:38 GMT | climateprediction.net | [error] Error reported by file upload server: Server is out of disk space
Thu 07 Nov 2013 02:59:38 GMT | climateprediction.net | Temporarily failed upload of hadcm3n_o93c_1900_40_008466907_0_4.zip: transient upload error
Thu 07 Nov 2013 02:59:38 GMT | climateprediction.net | Backing off 26 min 8 sec on upload of hadcm3n_o93c_1900_40_008466907_0_4.zip
Thu 07 Nov 2013 03:25:46 GMT | climateprediction.net | Started upload of hadcm3n_o93c_1900_40_008466907_0_4.zip
Thu 07 Nov 2013 03:25:48 GMT | climateprediction.net | [error] Error reported by file upload server: Server is out of disk space
Thu 07 Nov 2013 03:25:48 GMT | climateprediction.net | Temporarily failed upload of hadcm3n_o93c_1900_40_008466907_0_4.zip: transient upload error
Thu 07 Nov 2013 03:25:48 GMT | climateprediction.net | Backing off 59 min 48 sec on upload of hadcm3n_o93c_1900_40_008466907_0_4.zip
Thu 07 Nov 2013 04:25:38 GMT | climateprediction.net | Started upload of hadcm3n_o93c_1900_40_008466907_0_4.zip
Thu 07 Nov 2013 04:25:39 GMT | climateprediction.net | [error] Error reported by file upload server: Server is out of disk space
Thu 07 Nov 2013 04:25:39 GMT | climateprediction.net | Temporarily failed upload of hadcm3n_o93c_1900_40_008466907_0_4.zip: transient upload error
Thu 07 Nov 2013 04:25:39 GMT | climateprediction.net | Backing off 1 hr 51 min 26 sec on upload of hadcm3n_o93c_1900_40_008466907_0_4.zip
Thu 07 Nov 2013 06:17:06 GMT | climateprediction.net | Started upload of hadcm3n_o93c_1900_40_008466907_0_4.zip
Thu 07 Nov 2013 06:17:07 GMT | climateprediction.net | [error] Error reported by file upload server: Server is out of disk space
Thu 07 Nov 2013 06:17:07 GMT | climateprediction.net | Temporarily failed upload of hadcm3n_o93c_1900_40_008466907_0_4.zip: transient upload error
Thu 07 Nov 2013 06:17:07 GMT | climateprediction.net | Backing off 4 hr 3 min 34 sec on upload of hadcm3n_o93c_1900_40_008466907_0_4.zip


And from client_state.xml

<file>
    <name>hadcm3n_o93c_1900_40_008466907_0_4.zip</name>
    <nbytes>54706631.000000</nbytes>
    <max_nbytes>188743680.000000</max_nbytes>
    <md5_cksum>b30d6cab259e190cc09b6d0bed81eed4</md5_cksum>
    <status>1</status>
    <upload_url>http://rapid-watch.badc.rl.ac.uk/cpdn_cgi/file_upload_handler</upload_url>
    <persistent_file_xfer>
        <num_retries>7</num_retries>
        <first_request_time>1383792135.482826</first_request_time>
        <next_request_time>1383819641.528833</next_request_time>
        <time_so_far>9.032011</time_so_far>
        <last_bytes_xferred>115.000000</last_bytes_xferred>
        <is_upload>1</is_upload>
    </persistent_file_xfer>
</file>


http://rapid-watch.badc.rl.ac.uk/cpdn_cgi/file_upload_handler gives me:
<data_server_reply>
    <status>1</status>
    <message>no command</message>
</data_server_reply>
ID: 47487 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 47488 - Posted: 7 Nov 2013, 7:58:12 UTC
Last modified: 7 Nov 2013, 8:00:58 UTC

OK, I'll email them.
ID: 47488 · Report as offensive     Reply Quote
Profile The Ancient One

Send message
Joined: 5 Sep 04
Posts: 21
Credit: 1,795,892
RAC: 1,677
Message 47491 - Posted: 7 Nov 2013, 15:54:56 UTC

I too have this error see below:

07/11/2013 15:47:53 | climateprediction.net | Started upload of hadcm3n_823a_1980_40_008460649_1_3.zip
07/11/2013 15:47:54 | climateprediction.net | [error] Error reported by file upload server: Server is out of disk space
07/11/2013 15:47:54 | climateprediction.net | Temporarily failed upload of hadcm3n_823a_1980_40_008460649_1_3.zip: transient upload error
07/11/2013 15:47:54 | climateprediction.net | Backing off 3 hr 30 min 52 sec on upload of hadcm3n_823a_1980_40_008460649_1_3.zip

Regards
Jurgen
"All man born has a right to life and no man born has the right to take that life"
ID: 47491 · Report as offensive     Reply Quote
Profile Alan K

Send message
Joined: 22 Feb 06
Posts: 484
Credit: 29,599,157
RAC: 2,031
Message 47492 - Posted: 7 Nov 2013, 16:38:51 UTC - in response to Message 47491.  

I've got this as well. I guess the best thing to do is suspend network activity until it gets sorted.
ID: 47492 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 47494 - Posted: 7 Nov 2013, 18:08:15 UTC

The server in question is external to Oxford Uni.
The relevant people have been asked to urgently increase storage space.


ID: 47494 · Report as offensive     Reply Quote
Profile Byron Leigh Hatch @ team Carl ...
Avatar

Send message
Joined: 17 Aug 04
Posts: 289
Credit: 44,103,664
RAC: 0
Message 47496 - Posted: 7 Nov 2013, 19:07:41 UTC

Thank you Les.
ID: 47496 · Report as offensive     Reply Quote
Alex Plantema

Send message
Joined: 3 Sep 04
Posts: 126
Credit: 26,363,193
RAC: 0
Message 47503 - Posted: 9 Nov 2013, 0:48:56 UTC
Last modified: 9 Nov 2013, 0:49:18 UTC

Now something different happens. The file is uploaded until it reaches 100%, then the no space message follows:
09-11-2013 01:20:15 | climateprediction.net | Started upload of hadcm3n_7zdj_1980_40_008457130_1_2.zip
09-11-2013 01:42:14 | climateprediction.net | [error] Error reported by file upload server: can't write file /storage/incoming/uploader//hadcm3n_7zdj_1980_40_008457130_1_2.zip: No space left on server
09-11-2013 01:42:14 | climateprediction.net | Temporarily failed upload of hadcm3n_7zdj_1980_40_008457130_1_2.zip: transient upload error
ID: 47503 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 47504 - Posted: 9 Nov 2013, 1:19:05 UTC - in response to Message 47503.  

We're on it.
It's the same problem. They need a seriously bigger hard disk.

I don't know which of the several research centres is storing the data, or if someone there works weekends, so it may take a few days to sort out.

In the meantime, there's mowing the lawn, reading a few books, starting a jogging / running program, or just relaxing and going Ooooom a lot.

ID: 47504 · Report as offensive     Reply Quote
Art Masson
Avatar

Send message
Joined: 16 Oct 11
Posts: 254
Credit: 15,954,577
RAC: 2
Message 47506 - Posted: 9 Nov 2013, 14:50:26 UTC

I have two 52MB files that have been failing with this for about a week. Should we just continue to wait for a fix, or do I need to delete/cancel the jobs?
ID: 47506 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 47508 - Posted: 9 Nov 2013, 19:08:54 UTC - in response to Message 47506.  

This problem with disk space only occurred 3 days ago, so you may have a different problem.
And I've already said that patience is needed.

ID: 47508 · Report as offensive     Reply Quote
Jacob Klein

Send message
Joined: 28 Mar 13
Posts: 16
Credit: 5,383,625
RAC: 0
Message 47509 - Posted: 9 Nov 2013, 19:12:17 UTC - in response to Message 47508.  

Okay, patience is needed.

So, the problem started around 11/6/2013.

The main question I have is:
How long will BOINC allow a download to fail until BOINC does something crazy like abort it? My wife has her first completed task queued up, and it took 640 hours (~27 days) to complete. I don't want to see all that work wasted.
ID: 47509 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 47512 - Posted: 9 Nov 2013, 19:38:42 UTC - in response to Message 47509.  

It used to be 14 days, but for late versions of BOINC 6.* onwards, it's 90 days.


ID: 47512 · Report as offensive     Reply Quote
Jacob Klein

Send message
Joined: 28 Mar 13
Posts: 16
Credit: 5,383,625
RAC: 0
Message 47513 - Posted: 9 Nov 2013, 19:39:50 UTC - in response to Message 47512.  
Last modified: 9 Nov 2013, 19:40:50 UTC

Thank you, and thanks for making sure someone is trying to fix it.

I will continue to monitor the upload every few days, and hopefully it gets resolved, especially before 2/4/2014.
ID: 47513 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4342
Credit: 16,498,761
RAC: 5,627
Message 47514 - Posted: 9 Nov 2013, 19:40:53 UTC

How long will BOINC allow a download to fail until BOINC does something crazy like abort it?


Quite a while, certainly months rather than days. It is worth turning off network activity in BOINC manager to avoid repeated attempts at uploading and keeping an eye on this thread. I think that the number of upload attempts before it gives up has been increased but I can't remember what to. I have never had one fail due to this even when I haven't disabled network activity but it saves bandwidth if you do this. If running another project and you want to download work for it, just turn it on again when downloading work.
ID: 47514 · Report as offensive     Reply Quote
Jacob Klein

Send message
Joined: 28 Mar 13
Posts: 16
Credit: 5,383,625
RAC: 0
Message 47515 - Posted: 9 Nov 2013, 19:42:21 UTC - in response to Message 47514.  
Last modified: 9 Nov 2013, 19:43:10 UTC

Recommending to turn off network activity is silly! Don't do that! BOINC intelligently backs off the upload-retry-intervals automatically.
I'm attached to 26 other projects, doing work for them just fine, and I (obviously) require network activity to both download new tasks and upload completed results.
ID: 47515 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 47518 - Posted: 10 Nov 2013, 6:30:14 UTC - in response to Message 47515.  

It's NOT silly. For a lot of people it makes sense; e.g. those of us that are only running cpdn. Including me, where "network off" is the normal condition, only allowing uploads when I want, so that I can check for errors, as well as other reasons.

ID: 47518 · Report as offensive     Reply Quote
Previous · 1 · 2 · 3 · 4 · Next

Message boards : Number crunching : A different upload problem -- out of disk space on rapid-watch.badc.rl.ac.uk

©2024 climateprediction.net