climateprediction.net home page
Upload server is out of disk space

Upload server is out of disk space

Message boards : Number crunching : Upload server is out of disk space
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 · Next

AuthorMessage
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4314
Credit: 16,378,503
RAC: 3,632
Message 59100 - Posted: 26 Nov 2018, 15:12:14 UTC - in response to Message 59099.  

Which still means about 3 hours to go. :(


Ah, clearly it is your uploads that are clogging up the system Les.
ID: 59100 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 59101 - Posted: 26 Nov 2018, 21:11:39 UTC

It took 5 hours, but it got up to 30kb/s on each channel, instead of the 10-14kb/s I was getting over the wi-fi link. Just as well there was only a dozen zips.

I watched for a while, and I could actually see the congestion.
BOINC would start 2, there'd be a long delay, and then they'd time out. Then another pair would try.
When one got going, it quickly got up to the full 60kb/s, so that if the 2nd one did manage to start, it would struggle to get anywhere.
But, this morning they were all gone, so, hopefully other people are getting rid of their backlog as well.

Now I'm having problems with a cam25 on upload 6.
(105 Megs each)
ID: 59101 · Report as offensive     Reply Quote
Profile JIM

Send message
Joined: 31 Dec 07
Posts: 1152
Credit: 22,053,321
RAC: 4,417
Message 59102 - Posted: 26 Nov 2018, 22:31:44 UTC - in response to Message 59101.  

Mine are all gone.
ID: 59102 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4314
Credit: 16,378,503
RAC: 3,632
Message 59103 - Posted: 27 Nov 2018, 7:33:43 UTC - in response to Message 59101.  

When one got going, it quickly got up to the full 60kb/s


Only marginally slower than my own setup here which seems to max out at about 90kb/s (with the wind in the right direction.) Pretty sure that if I had fast machines I would be getting the same problems you are with uploads.
ID: 59103 · Report as offensive     Reply Quote
Speedy

Send message
Joined: 20 Jul 05
Posts: 25
Credit: 414,873
RAC: 406
Message 59321 - Posted: 7 Jan 2019, 3:28:53 UTC

We are out of disk space again by the looks of things.
07-Jan-19 11:30:42 AM | climateprediction.net | Started upload of wah2_sas50_a0b9_200812_22_778_011703429_2_r1423304724_1.zip
07-Jan-19 11:30:44 AM | climateprediction.net | [error] Error reported by file upload server: Server is out of disk space
07-Jan-19 11:30:44 AM | climateprediction.net | Temporarily failed upload of wah2_sas50_a0b9_200812_22_778_011703429_2_r1423304724_1.zip: transient upload error
07-Jan-19 11:30:44 AM | climateprediction.net | Backing off 00:59:27 on upload of wah2_sas50_a0b9_200812_22_778_011703429_2_r1423304724_1.zip

The WAH 2 unit I am processing requires 124.79 MB per upload/Trickles message. One thing I find interesting to note is when I click on the task under my account it is listed under trickles even though the file hasn't been up loaded. We may see some improvements and about 5 hours it is currently 3:30 a.m. over in the UK as of this post. I think I will let my machine get 3 trickles waiting in the transfer list on the I will suspend until they have cleared so I don't get any errors.
ID: 59321 · Report as offensive     Reply Quote
Profile Vicki

Send message
Joined: 28 Nov 15
Posts: 50
Credit: 4,099,809
RAC: 0
Message 59324 - Posted: 7 Jan 2019, 9:32:07 UTC - in response to Message 59321.  

Dear fellow readers
Yip me to as far as the server is out of upload space error is concerned & I have gotten that over the last few days as the backlogged of zips to send back mounts up. most of my stuck trickles are 50ish meg per file and are global series 769 & 770 series and still have a long way until completion of calculations for said work unis. one wonders how large the receiving servers capacity is & how much space is still not enough for the amount of work units being sent out to be crunched. time to invent the 200 teragig sd card & use that for incoming trickles!
Aside from all that, Happy New Year & may it bring what you want as well as what you need. For me that means continuing to learn the skills no one has taught me yet, including hair plating, which is becoming a priority as my hair approaches 4.5 years long & with summer temperatures, hair in a bun is cooler than down, but still the yearn to see it plated before acquiring the skill reminds me that changing gender marker doesn't mean learning stops there. you can read the ups & downs of my 13th year of living female here. Copy & paste this link into your browser to read it. be warned that some might call it blunt, I call it speaking my mind. https://vickiraewest.atavist.com/the-13th-year-of-living-female-my-path
Kind Regards
Ms Vicki
ID: 59324 · Report as offensive     Reply Quote
ccpmc

Send message
Joined: 9 Nov 18
Posts: 3
Credit: 1,117,025
RAC: 0
Message 59325 - Posted: 7 Jan 2019, 12:39:09 UTC

2019/01/07 19:56:00 | climateprediction.net | [error] Error reported by file upload server: Server is out of disk space
ID: 59325 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4314
Credit: 16,378,503
RAC: 3,632
Message 59326 - Posted: 7 Jan 2019, 16:33:48 UTC - in response to Message 59325.  

My uploads have started to clear. I know Sarah had been working on freeing up some disk space. Don't know whether it will be enough yet or not though.
ID: 59326 · Report as offensive     Reply Quote
nairb

Send message
Joined: 3 Sep 04
Posts: 105
Credit: 5,646,090
RAC: 102,785
Message 59327 - Posted: 7 Jan 2019, 17:04:54 UTC

Uploads started but stop at 100% and gets no further. It then restarts at 0%.....

Not fixed yet....
ID: 59327 · Report as offensive     Reply Quote
Jim1348

Send message
Joined: 15 Jan 06
Posts: 637
Credit: 26,751,529
RAC: 653
Message 59328 - Posted: 7 Jan 2019, 19:11:37 UTC

I am to blame. I just put a new Ryzen 2600 on CPDN. The last time I added a machine, the whole Oxford network went down for two months the next day.

I am reducing the load by taking one machine off, and it should clear shortly.
ID: 59328 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 59329 - Posted: 7 Jan 2019, 19:36:02 UTC

As long as we know who to blame. :)

So many problems for the project people, now that they're back.
Hopefully, a few strong coffees will fortify them for the long, boring, grind ahead.
ID: 59329 · Report as offensive     Reply Quote
[SG]Felix

Send message
Joined: 4 Oct 15
Posts: 34
Credit: 9,069,332
RAC: 14,637
Message 59330 - Posted: 7 Jan 2019, 20:30:37 UTC

i paused my 4 wus until my 70 waiting zips are uploaded.
ID: 59330 · Report as offensive     Reply Quote
Speedy

Send message
Joined: 20 Jul 05
Posts: 25
Credit: 414,873
RAC: 406
Message 59331 - Posted: 7 Jan 2019, 20:49:32 UTC - in response to Message 59327.  

Uploads started but stop at 100% and gets no further. It then restarts at 0%.....

Not fixed yet....

I have to agree. I hope Oxford is on an unlimited data plan or at least a very cheap one as it is a waste of traffic for them. I'm okay I am on an unlimited plan. I currently have 3 waiting to upload if I get to 6 I will wait for them to clear before continuing. Unless someone has a safe recommendation on numbers. I am at a slight disadvantage because when I am able to upload it is night time in server land
ID: 59331 · Report as offensive     Reply Quote
Profile Iain Inglis
Volunteer moderator

Send message
Joined: 16 Jan 10
Posts: 1079
Credit: 6,904,049
RAC: 6,657
Message 59333 - Posted: 7 Jan 2019, 21:06:47 UTC - in response to Message 59331.  

[Speedy wrote:]
... Unless someone has a safe recommendation on numbers. ...


Many models have a 2 GB limit for backlogged files, so pausing models before reaching that limit is a good idea.
ID: 59333 · Report as offensive     Reply Quote
Speedy

Send message
Joined: 20 Jul 05
Posts: 25
Credit: 414,873
RAC: 406
Message 59334 - Posted: 7 Jan 2019, 21:42:50 UTC - in response to Message 59333.  

[Speedy wrote:]
... Unless someone has a safe recommendation on numbers. ...


Many models have a 2 GB limit for backlogged files, so pausing models before reaching that limit is a good idea.

Thanks, in that case I think I can afford to visit one for quite a while before I need to pause
ID: 59334 · Report as offensive     Reply Quote
Jean-David Beyer

Send message
Joined: 5 Aug 04
Posts: 1045
Credit: 16,506,818
RAC: 5,001
Message 59335 - Posted: 7 Jan 2019, 21:46:26 UTC - in response to Message 59331.  
Last modified: 7 Jan 2019, 21:48:08 UTC

I have the same problem,
Uploads started but stop at 100% and gets no further. It then restarts at 0%.....
It seems to have tried to upload the first trickle at least three times while I was watching, using up an hour of cpu time, so it was probably more. I can upload 75 Megabits/Second if the server can accept them.

But they do not get logged yet. My latest logs of C.P.:
31-Dec-2018 13:42:10 [climateprediction.net] Sending scheduler request: To fetch work.
31-Dec-2018 13:42:10 [climateprediction.net] Requesting new tasks for CPU
31-Dec-2018 13:42:12 [climateprediction.net] Scheduler request completed: got 0 new tasks
31-Dec-2018 13:42:12 [climateprediction.net] No tasks sent
31-Dec-2018 13:42:12 [climateprediction.net] No tasks are available for UK Met Office HadAM3P (global only) with MOSES II landsurface scheme
31-Dec-2018 13:42:12 [climateprediction.net] No tasks are available for UK Met Office Coupled Model Full Resolution Ocean
31-Dec-2018 13:42:12 [climateprediction.net] No tasks are available for UK Met Office HadCM3 short
31-Dec-2018 13:42:12 [climateprediction.net] No tasks are available for Weather At Home 2 (wah2) (region independent)
ID: 59335 · Report as offensive     Reply Quote
Alex Plantema

Send message
Joined: 3 Sep 04
Posts: 126
Credit: 26,363,193
RAC: 0
Message 59336 - Posted: 7 Jan 2019, 22:41:54 UTC - in response to Message 59324.  

you can read the ups & downs of my 13th year of living female here.
Hi Vicki, congrats on your new gender!
ID: 59336 · Report as offensive     Reply Quote
Speedy

Send message
Joined: 20 Jul 05
Posts: 25
Credit: 414,873
RAC: 406
Message 59345 - Posted: 8 Jan 2019, 4:57:47 UTC - in response to Message 59087.  

We need a bigger cloud.

Do we have any idea how big the current storage capacity is? My uploads are uploading but are not being accepted. I gather there is a rather large backlog to get through with limited storage capacity
ID: 59345 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 59347 - Posted: 8 Jan 2019, 6:25:30 UTC

The project people are working at moving data off the relevant Upload server and onto various NASs.
It all backed up because every one there has been on holidays.
ID: 59347 · Report as offensive     Reply Quote
Speedy

Send message
Joined: 20 Jul 05
Posts: 25
Credit: 414,873
RAC: 406
Message 59348 - Posted: 8 Jan 2019, 7:46:44 UTC - in response to Message 59347.  

The project people are working at moving data off the relevant Upload server and onto various NASs.
It all backed up because every one there has been on holidays.

Thanks for the information Les. I would find it interesting watching the data being transferred.
ID: 59348 · Report as offensive     Reply Quote
Previous · 1 · 2 · 3 · 4 · 5 · Next

Message boards : Number crunching : Upload server is out of disk space

©2024 climateprediction.net