climateprediction.net home page
Upload failures

Upload failures

Message boards : Number crunching : Upload failures
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 5 · 6 · 7 · 8 · 9 · 10 · 11 . . . 19 · Next

AuthorMessage
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4314
Credit: 16,376,846
RAC: 3,590
Message 60422 - Posted: 24 Jun 2019, 14:28:17 UTC - in response to Message 60418.  

From David Wallom

We have migrated the scheduler to the backup system this morning though this means that Andy is doing some fiddling with the https configuration at the moment.


My sam25 uploads that were stuck while the site was down have all uploaded now but I don't know what effect the fiddling will have with anything else just that the transitioner is showing red on the Project Status page currently.
ID: 60422 · Report as offensive     Reply Quote
bernard_ivo

Send message
Joined: 18 Jul 13
Posts: 438
Credit: 24,487,746
RAC: 3,014
Message 60427 - Posted: 24 Jun 2019, 15:45:35 UTC - in response to Message 60422.  

Thanks Dave,

zips going to Oxford seem to be uploading, however JASMIN is still out of disk space.
ID: 60427 · Report as offensive     Reply Quote
Jim1348

Send message
Joined: 15 Jan 06
Posts: 637
Credit: 26,751,529
RAC: 653
Message 60430 - Posted: 24 Jun 2019, 19:31:19 UTC - in response to Message 60422.  

All of my Linux zips on three machines have gone, so that is progress. But I have over 50 WAH2 zips on my windows machine still stuck. It seems to be discrimination against North America.
ID: 60430 · Report as offensive     Reply Quote
Lockleys

Send message
Joined: 13 Jan 07
Posts: 195
Credit: 10,581,566
RAC: 0
Message 60432 - Posted: 24 Jun 2019, 21:58:49 UTC - in response to Message 60422.  

My SAM25 zips have all gone but my SAM50 zips are all still stuck.
ID: 60432 · Report as offensive     Reply Quote
mmonnin

Send message
Joined: 28 May 17
Posts: 49
Credit: 14,706,403
RAC: 9,709
Message 60435 - Posted: 24 Jun 2019, 22:29:30 UTC - in response to Message 60430.  

All of my Linux zips on three machines have gone, so that is progress. But I have over 50 WAH2 zips on my windows machine still stuck. It seems to be discrimination against North America.


Same here. SAM50 and SAFR50 are pending. Those files are several times as big. ~17MB compared to 76/92MB each.
ID: 60435 · Report as offensive     Reply Quote
[SG]_Jupp

Send message
Joined: 7 Jun 19
Posts: 1
Credit: 9,899
RAC: 0
Message 60436 - Posted: 25 Jun 2019, 6:18:18 UTC - in response to Message 60435.  

Server is out of disk space

25.06.2019 08:16:44 | climateprediction.net | Started upload of wah2_safr50_n1vt_201112_13_818_011862688_0_r392739119_8.zip
25.06.2019 08:16:46 | climateprediction.net | [error] Error reported by file upload server: Server is out of disk space

I have 2 files 89MB and 90MB
ID: 60436 · Report as offensive     Reply Quote
mngn

Send message
Joined: 13 Jul 18
Posts: 38
Credit: 56,768,885
RAC: 5,312
Message 60438 - Posted: 25 Jun 2019, 12:41:39 UTC
Last modified: 25 Jun 2019, 13:01:25 UTC

sam50 uploads have started and are slowing down all other communication. 15 GBytes to go, should take at least two days.

Edit: They stopped again. Out of disk space.
ID: 60438 · Report as offensive     Reply Quote
rbpeake

Send message
Joined: 27 Feb 08
Posts: 41
Credit: 1,402,356
RAC: 0
Message 60439 - Posted: 25 Jun 2019, 13:12:46 UTC

One wonders if these issues could have been anticipated. We have done our part!
ID: 60439 · Report as offensive     Reply Quote
Jim1348

Send message
Joined: 15 Jan 06
Posts: 637
Credit: 26,751,529
RAC: 653
Message 60442 - Posted: 25 Jun 2019, 14:06:39 UTC - in response to Message 60439.  

One wonders if these issues could have been anticipated.

They are hoping to do better with the climate.
ID: 60442 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4314
Credit: 16,376,846
RAC: 3,590
Message 60443 - Posted: 25 Jun 2019, 14:28:32 UTC

One wonders if these issues could have been anticipated.


I suspect they are anticipated but there isn't enough redundancy in the systems of the universities who receive the data to cope with problems. In the past, this has been true of Oxford too. My guess is that the universities receiving the data have a standard protocol of how much space they give researchers irrespective of how much space they actually need.

Of course it may be nothing to do with bureaucracy and I am just letting my 30+ years working for the NHS show. ;)
ID: 60443 · Report as offensive     Reply Quote
rob

Send message
Joined: 5 Jun 09
Posts: 74
Credit: 2,905,029
RAC: 2,254
Message 60454 - Posted: 26 Jun 2019, 16:47:55 UTC

I've just looked - two tasks waiting to upload, a total of 23 zip files mostly around 90Mb each.
Surely, when sending work out one would have an idea of the returned file sizes, and either think about not sending a particular job out if there might be a space issue, or make sure that there is more than sufficient space available for the return files.
ID: 60454 · Report as offensive     Reply Quote
idahobear

Send message
Joined: 4 Nov 18
Posts: 4
Credit: 1,329,613
RAC: 0
Message 60456 - Posted: 26 Jun 2019, 18:08:39 UTC

I am having the same problem, but I have 35 zip files waiting. My log says your server is out of space.

6/26/2019 10:21:56 AM | climateprediction.net | Started upload of wah2_sam50_n59k_200612_24_822_011881979_0_r1523546388_3.zip
6/26/2019 10:21:56 AM | climateprediction.net | Started upload of wah2_sam50_n59k_200612_24_822_011881979_0_r1523546388_5.zip
6/26/2019 10:21:59 AM | climateprediction.net | [error] Error reported by file upload server: Server is out of disk space
6/26/2019 10:21:59 AM | climateprediction.net | [error] Error reported by file upload server: Server is out of disk space
6/26/2019 10:21:59 AM | climateprediction.net | Temporarily failed upload of wah2_sam50_n59k_200612_24_822_011881979_0_r1523546388_3.zip: transient upload error
6/26/2019 10:21:59 AM | climateprediction.net | Backing off 04:27:47 on upload of wah2_sam50_n59k_200612_24_822_011881979_0_r1523546388_3.zip
6/26/2019 10:21:59 AM | climateprediction.net | Temporarily failed upload of wah2_sam50_n59k_200612_24_822_011881979_0_r1523546388_5.zip: transient upload error
6/26/2019 10:21:59 AM | climateprediction.net | Backing off 05:49:53 on upload of wah2_sam50_n59k_200612_24_822_011881979_0_r1523546388_5.zip
ID: 60456 · Report as offensive     Reply Quote
Iceberg

Send message
Joined: 28 Dec 17
Posts: 18
Credit: 1,097,261
RAC: 147
Message 60457 - Posted: 26 Jun 2019, 18:48:54 UTC

Some have said that if you have many zip files to upload but they're not getting through to put your tasks on suspended status so they won't fail and wait until the zip files have been successfully uploaded. I only have about 12 files pending, but I've suspended mine until they make enough server space.

Moderators, is this what you would suggest?
ID: 60457 · Report as offensive     Reply Quote
Profile Alan K

Send message
Joined: 22 Feb 06
Posts: 484
Credit: 29,579,234
RAC: 4,572
Message 60461 - Posted: 26 Jun 2019, 21:56:15 UTC - in response to Message 60443.  

From the length of time this has been going on with Jasmin I would guess (perhaps incorrectly) that this is more than just a re-adjustment of quotas but an actual lack of physical hard drive space which of course takes time and money to solve.
ID: 60461 · Report as offensive     Reply Quote
Mephist0

Send message
Joined: 21 Feb 08
Posts: 47
Credit: 7,929,915
RAC: 0
Message 60463 - Posted: 26 Jun 2019, 22:17:47 UTC

Do we have any indication how it is going with the storage space problem? When will it be fixed?
ID: 60463 · Report as offensive     Reply Quote
Jim1348

Send message
Joined: 15 Jan 06
Posts: 637
Credit: 26,751,529
RAC: 653
Message 60464 - Posted: 26 Jun 2019, 22:19:08 UTC - in response to Message 60461.  

From the length of time this has been going on with Jasmin I would guess (perhaps incorrectly) that this is more than just a re-adjustment of quotas but an actual lack of physical hard drive space which of course takes time and money to solve.

I expect it is worse than that. I fear a repeat of the Big One last year.
When it crashes, it crashes.
ID: 60464 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 60465 - Posted: 26 Jun 2019, 22:38:54 UTC

Apparently the people who look after "jasmin" have been having problems for some time.
I don't know the fine details and I don't intend to ask.
ID: 60465 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 60466 - Posted: 26 Jun 2019, 22:42:59 UTC - in response to Message 60457.  

Yes, Suspending all of the tasks so that you don't get even more zips in the queue is definitely recommended.
ID: 60466 · Report as offensive     Reply Quote
crashtech

Send message
Joined: 1 Jun 17
Posts: 13
Credit: 23,919,122
RAC: 1,141,751
Message 60469 - Posted: 27 Jun 2019, 1:24:48 UTC

Wow, I picked a bad time to return to CPDN. My upload queues are choked with work that can't go anywhere. Hopefully suspending the project will help retain completed work on local machines. There are disk space warnings that make me think some work may end up being lost for good.
ID: 60469 · Report as offensive     Reply Quote
mngn

Send message
Joined: 13 Jul 18
Posts: 38
Credit: 56,768,885
RAC: 5,312
Message 60470 - Posted: 27 Jun 2019, 6:49:46 UTC

These are from the same computer. sam50 downloads are forbidden, anz50 are allowed?

2019-06-27 07:17:52 | climateprediction.net | Giving up on download of wah2_sam50_n4k0_200912_13_809_011823386.zip: permanent HTTP error

2019-06-27 08:18:30 | climateprediction.net | Finished download of wah2_anz50_n27g_201612_20_794_011765246.zip
ID: 60470 · Report as offensive     Reply Quote
Previous · 1 . . . 5 · 6 · 7 · 8 · 9 · 10 · 11 . . . 19 · Next

Message boards : Number crunching : Upload failures

©2024 climateprediction.net