climateprediction.net home page
Upload server is out of disk space

Upload server is out of disk space

Message boards : Number crunching : Upload server is out of disk space
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 · Next

AuthorMessage
bernard_ivo

Send message
Joined: 18 Jul 13
Posts: 399
Credit: 20,553,959
RAC: 652
Message 59055 - Posted: 20 Nov 2018, 21:03:36 UTC - in response to Message 59040.  

We nowadays micromanage these CPDN WUs so much so it would be nice our names to appear along researchers' ones when papers are published ;)
ID: 59055 · Report as offensive     Reply Quote
Profile Alan K

Send message
Joined: 22 Feb 06
Posts: 419
Credit: 18,919,008
RAC: 1,263
Message 59066 - Posted: 21 Nov 2018, 23:04:28 UTC - in response to Message 59038.  

Looks like the problem is back again tonight -

21/11/2018 22:38:26 | climateprediction.net | [file_xfer] http op done; retval 0 (Success)
21/11/2018 22:38:26 | climateprediction.net | [error] Error reported by file upload server: Server is out of disk space
21/11/2018 22:38:26 | climateprediction.net | [file_xfer] parsing upload response: <data_server_reply> <status>1</status> <message>Server is out of disk space</message></data_server_reply>
21/11/2018 22:38:26 | climateprediction.net | [file_xfer] parsing status: -127
21/11/2018 22:38:26 | climateprediction.net | [file_xfer] file transfer status -127 (transient upload error)
21/11/2018 22:38:26 | climateprediction.net | Temporarily failed upload of hadcm3s_st1856_190012_120_771_011669883_0_r44521125_5.zip: transient upload error
ID: 59066 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7578
Credit: 24,034,754
RAC: 70
Message 59068 - Posted: 22 Nov 2018, 0:55:29 UTC

Yes, it was discussed on Wednesday, UK time.

However, that's a hadcm3s.
Which server is it going to?
ID: 59068 · Report as offensive     Reply Quote
Profile geophi
Volunteer moderator

Send message
Joined: 7 Aug 04
Posts: 2096
Credit: 57,971,859
RAC: 5,479
Message 59069 - Posted: 22 Nov 2018, 1:14:06 UTC - in response to Message 59068.  

My hadcm3s uploads are also stalled, and they are trying to go to upload3
ID: 59069 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7578
Credit: 24,034,754
RAC: 70
Message 59070 - Posted: 22 Nov 2018, 1:24:35 UTC

Ah, everything going to the same place.

Well, tomorrow is another day, so we'll see then.
ID: 59070 · Report as offensive     Reply Quote
Hans Sveen

Send message
Joined: 31 Aug 04
Posts: 3
Credit: 13,470,397
RAC: 0
Message 59073 - Posted: 22 Nov 2018, 13:02:15 UTC
Last modified: 22 Nov 2018, 13:02:47 UTC

Hello!
One sam25 wu from batch 742 that had stalled upload for more than a half day
has just uploaded !
Thank You for fixing this!


Ps.
As stated on another thred, batch 742 may be successful if not run on Win 10!
Hans Sveen
Oslo,Norway

ID: 59073 · Report as offensive     Reply Quote
Hans Sveen

Send message
Joined: 31 Aug 04
Posts: 3
Credit: 13,470,397
RAC: 0
Message 59079 - Posted: 24 Nov 2018, 11:02:48 UTC
Last modified: 24 Nov 2018, 11:03:36 UTC

Hi!
During the night, early morning a new problem has emerged.
Server status says that everything is working ok, downloading is ok, just got a resend from batch 764.

Upload is another matter:

24.11.2018 11.40.21 | climateprediction.net | Backing off 04:15:26 on upload of wah2_eu25_e0qs_199512_13_745_011591405_1_r1094320493_13.zip
24.11.2018 11.40.21 | climateprediction.net | Started upload of wah2_eu25_e0qs_199512_13_745_011591405_1_r1094320493_out.zip
24.11.2018 11.40.21 | climateprediction.net | Temporarily failed upload of wah2_eu25_ee65_200912_13_745_011617407_0_r132976074_10.zip: connect() failed
24.11.2018 11.40.21 | climateprediction.net | Backing off 00:26:07 on upload of wah2_eu25_ee65_200912_13_745_011617407_0_r132976074_10.zip
24.11.2018 11.40.22 | | Project communication failed: attempting access to reference site
24.11.2018 11.40.24 | | Internet access OK - project servers may be temporarily down.
24.11.2018 11.40.43 | | Project communication failed: attempting access to reference site
24.11.2018 11.40.43 | climateprediction.net | Temporarily failed upload of wah2_eu25_e0qs_199512_13_745_011591405_1_r1094320493_out.zip: connect() failed
24.11.2018 11.40.43 | climateprediction.net | Backing off 03:16:57 on upload of wah2_eu25_e0qs_199512_13_745_011591405_1_r1094320493_out.zip
24.11.2018 11.40.44 | | Internet access OK - project servers may be temporarily down.



I know it will be fixed over the weekend!
Hans Sveen
Oslo,Norway

ID: 59079 · Report as offensive     Reply Quote
bernard_ivo

Send message
Joined: 18 Jul 13
Posts: 399
Credit: 20,553,959
RAC: 652
Message 59081 - Posted: 24 Nov 2018, 11:52:15 UTC - in response to Message 59079.  

yep, upload3 again. Got few zips queued.
ID: 59081 · Report as offensive     Reply Quote
Profile Alan K

Send message
Joined: 22 Feb 06
Posts: 419
Credit: 18,919,008
RAC: 1,263
Message 59085 - Posted: 24 Nov 2018, 23:36:57 UTC - in response to Message 59081.  

Isn't anyone on the server side capable of some basic maths to determine the disc space required for all the uploads? This is about the fourth time in ten days.
ID: 59085 · Report as offensive     Reply Quote
Jim1348

Send message
Joined: 15 Jan 06
Posts: 606
Credit: 26,339,909
RAC: 853
Message 59086 - Posted: 25 Nov 2018, 3:20:27 UTC - in response to Message 59085.  

The real problem is that if I have a lot of uploads stuck, and want to leave the project, I won't be able to get out.
ID: 59086 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7578
Credit: 24,034,754
RAC: 70
Message 59087 - Posted: 25 Nov 2018, 3:46:41 UTC

The project was down for a while a few hours ago, and it was the same the previous UK night, so it looks like they have some scripts running to move files to another server, which can be started remotely.

The problem seems to be that everything is getting dumped onto the same server.
(And not just what the public can "see".)

Luckily, I got rid of everything from my 2 new, fast computers during the week, but I've built up a pile of zips on the old HP. That's now been Suspended, and I'm waiting to see what happens next week.
Not helped by using wi-fi to get to the router, which is slow. I'm going to get some long cat-6 cable on Monday, and run it along the floor.
At the slow speeds of my landline, it should be OK I think.

We need a bigger cloud.
ID: 59087 · Report as offensive     Reply Quote
Andreas38871

Send message
Joined: 10 Aug 05
Posts: 4
Credit: 2,790,251
RAC: 0
Message 59088 - Posted: 25 Nov 2018, 8:52:42 UTC

That's weird, on one computer (Windows 8) I have no problems with the upload, in the other computer (Windows 7) I get the following error message since yesterday:

25.11.2018 09:43:01 | climateprediction.net | Started upload of wah2_sam25_m00z_200012_85_764_011653247_0_r374692659_42.zip
25.11.2018 09:43:24 | climateprediction.net | Temporarily failed upload of wah2_sam25_m00z_200012_85_764_011653247_0_r374692659_42.zip: connect() failed
25.11.2018 09:43:24 | climateprediction.net | Backing off 00:18:16 on upload of wah2_sam25_m00z_200012_85_764_011653247_0_r374692659_42.zip
25.11.2018 09:43:24 | | Project communication failed: attempting access to reference site
25.11.2018 09:43:25 | | Internet access OK - project servers may be temporarily down.


ID: 59088 · Report as offensive     Reply Quote
Eirik Redd

Send message
Joined: 31 Aug 04
Posts: 383
Credit: 193,304,814
RAC: 16,932
Message 59089 - Posted: 25 Nov 2018, 12:24:37 UTC - in response to Message 59085.  

Isn't anyone on the server side capable of some basic maths to determine the disc space required for all the uploads? This is about the fourth time in ten days.


Server side capable? So joke a joke! Server side cloud obviously run by undergrads with zero funding and no worries and zero maths
ID: 59089 · Report as offensive     Reply Quote
Profile JIM

Send message
Joined: 31 Dec 07
Posts: 1147
Credit: 21,159,011
RAC: 0
Message 59090 - Posted: 26 Nov 2018, 4:25:00 UTC - in response to Message 59040.  

Any idea when there might be space again on the upload server. There is no sense running the models if we can’t upload the data.
ID: 59090 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7578
Credit: 24,034,754
RAC: 70
Message 59092 - Posted: 26 Nov 2018, 5:28:53 UTC

No.
But it's currently 5.25 AM in the UK, in Autumn, so it'll be a few hours at least before enough people show up for a meeting on what to do.
And my last computer running models has been shut down since last week, so as not to accumulate too many files in the Transfers tab and get caught by that BOINC bug.

I'm hoping that they just kill off the lot, so we can start again with a smaller version of those models.
ID: 59092 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 3340
Credit: 9,948,339
RAC: 791
Message 59095 - Posted: 26 Nov 2018, 11:30:58 UTC - in response to Message 59092.  

My hadcm3s zip that was stalled has now uploaded. Don't have any other zips to upload for a bit so not sure if all is running again yet or not.
ID: 59095 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7578
Credit: 24,034,754
RAC: 70
Message 59096 - Posted: 26 Nov 2018, 11:43:17 UTC

The backlog on my HP, now connected to the router via 60 feet of cable, is uploading fast. Which still means about 3 hours to go. :(
ID: 59096 · Report as offensive     Reply Quote
gchrist

Send message
Joined: 17 Jul 05
Posts: 7
Credit: 6,120,263
RAC: 0
Message 59097 - Posted: 26 Nov 2018, 12:28:34 UTC

My backlog is also uploading. However, sometimes they are backing off, as if the server cannot handle all the upload requests. The waiting files are from batches 768 and 769; the files from 773 always went through.
ID: 59097 · Report as offensive     Reply Quote
Jim1348

Send message
Joined: 15 Jan 06
Posts: 606
Credit: 26,339,909
RAC: 853
Message 59098 - Posted: 26 Nov 2018, 12:54:08 UTC - in response to Message 59097.  

However, sometimes they are backing off, as if the server cannot handle all the upload requests.

I allow four uploads at a time from each of two machines. However, even though sometimes four start up, they usually back off to two or three.
But at least they are going, up to about 200 kbps each. That is progress.
ID: 59098 · Report as offensive     Reply Quote
Profile Iain Inglis
Volunteer moderator

Send message
Joined: 16 Jan 10
Posts: 1058
Credit: 6,463,915
RAC: 0
Message 59099 - Posted: 26 Nov 2018, 13:12:52 UTC

Note that batch #773 (global, 145 months) has even bigger Zip files, 71 MB, which is over 10 GB upload in total. Fortunately the uploads go to upload9, which seems to accept upload requests immediately. My stalled uploads to upload3 have now all cleared.
ID: 59099 · Report as offensive     Reply Quote
Previous · 1 · 2 · 3 · 4 · 5 · Next

Message boards : Number crunching : Upload server is out of disk space

©2022 climateprediction.net