climateprediction.net home page
Upload Failure

Upload Failure

Message boards : Number crunching : Upload Failure
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 3 · 4 · 5 · 6 · 7 · 8 · 9 . . . 10 · Next

AuthorMessage
Profile tullio

Send message
Joined: 6 Aug 04
Posts: 264
Credit: 965,476
RAC: 0
Message 44270 - Posted: 30 May 2012, 12:29:03 UTC

My uploads are OK but on my task page I see only this:
Task 14708840

The page is blank
Tullio

ID: 44270 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 44272 - Posted: 30 May 2012, 15:02:27 UTC - in response to Message 44270.  

The page isn't blank. It's just the server taking forever to download the page info.
This is caused by a problem with the graphics image which is supposed to be on the page, but isn't, and a program is waiting for this to be returned from 'somewhere'.
Known problem.

I guess the server is also still getting hammered by the after effects of the recent power problem.


Backups: Here
ID: 44272 · Report as offensive     Reply Quote
Profile Thyme Lawn
Volunteer moderator

Send message
Joined: 5 Aug 04
Posts: 1283
Credit: 15,824,334
RAC: 0
Message 44273 - Posted: 30 May 2012, 15:29:37 UTC - in response to Message 44270.  

That's a slightly different case of the HadAM3P regional model page load problem Tullio. Normally the load stalls after displaying the perturbed parameters but your task is doing it after the header. The page does eventually finish loading (it took nearly 20 minutes for me). I'm going to raise this with the project team again.

The task's trickle page opens without any problems.
"The ultimate test of a moral society is the kind of world that it leaves to its children." - Dietrich Bonhoeffer
ID: 44273 · Report as offensive     Reply Quote
Profile tullio

Send message
Joined: 6 Aug 04
Posts: 264
Credit: 965,476
RAC: 0
Message 44274 - Posted: 30 May 2012, 18:33:54 UTC

Thanks, Les and Thyme. Anyway, the task is running OK.
Tullio
ID: 44274 · Report as offensive     Reply Quote
Eirik Redd

Send message
Joined: 31 Aug 04
Posts: 391
Credit: 219,888,554
RAC: 1,481,373
Message 44283 - Posted: 2 Jun 2012, 8:11:17 UTC

On server-status page uploader1.atm appears to be down.
No upload problems reported by clients here yet.

E
ID: 44283 · Report as offensive     Reply Quote
BarryAZ

Send message
Joined: 13 Jul 05
Posts: 125
Credit: 11,778,421
RAC: 0
Message 44284 - Posted: 2 Jun 2012, 13:26:24 UTC - in response to Message 44283.  

Upload problems here -- so you can now note that upload problems by clients have been reported -- since late Friday night here (west coast US time)
ID: 44284 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4346
Credit: 16,535,294
RAC: 5,887
Message 44285 - Posted: 2 Jun 2012, 15:13:52 UTC - in response to Message 44284.  

Is that with the full resolution ocean models? Both pnw and eu regional models seem to be uploading zips fine here but I don't have any of the HADAMCM3N models running at the moment. Presumably they go to a different server.


Dave
ID: 44285 · Report as offensive     Reply Quote
BarryAZ

Send message
Joined: 13 Jul 05
Posts: 125
Credit: 11,778,421
RAC: 0
Message 44290 - Posted: 4 Jun 2012, 4:09:43 UTC - in response to Message 44285.  

Yup - it is the hadam3p == seems, over the years that the server used to support the hadam application uploads/downloads is the most fragile of the servers. Other uploads have not had problems this week for me.

Perhaps we should redirect by avoiding that set (and server) down the road to reduce the load on that server...
ID: 44290 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 44291 - Posted: 4 Jun 2012, 5:49:10 UTC - in response to Message 44290.  

The servers have changed over the years, and aliases were put in place a couple of months ago to make it easier to redirect data in the event of a server problem.
I suspect that the current problem is that one of the servers has had problems since the recent planed outage a week ago.

And it's currently very early on the 3rd of a 4 day weekend in the UK.

But all PNW's are uploading OK. I haven't yet started on the current batch of EU's, so I don't know about them. I'll get one and see what happens in ~12 hours.


Backups: Here
ID: 44291 · Report as offensive     Reply Quote
Profile astroWX
Volunteer moderator

Send message
Joined: 5 Aug 04
Posts: 1496
Credit: 95,522,203
RAC: 0
Message 44292 - Posted: 4 Jun 2012, 20:10:33 UTC

There was an upload issue with the U.Oregon (USA) server, which receives the first twelve PNW .zip files, a few hours ago but is okay now. (Penn State U. [also in USA] is in the server mix but I forget for which project[s]. We've always dealt with scattered servers --> scattered across U.Oxford and across the globe. Complicated.)
"We have met the enemy and he is us." -- Pogo
Greetings from coastal Washington state, the scenic US Pacific Northwest.
ID: 44292 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 44293 - Posted: 4 Jun 2012, 20:38:41 UTC

The first EU zip was at about 8 hours, and this has now uploaded OK.
All of the first 12 zips go to the same server, so there shouldn't be any problem there.
PNW zips are also uploading OK, as are the restart zips.

Any problems with the regional models would therefore mostly likely be due to network problems, and/or heavy loading on the servers, which should be transitory.


Backups: Here
ID: 44293 · Report as offensive     Reply Quote
Profile Greg van Paassen

Send message
Joined: 17 Nov 07
Posts: 142
Credit: 4,271,370
RAC: 0
Message 44294 - Posted: 4 Jun 2012, 21:41:33 UTC

Not exactly "transitory" in the usual meaning of the word... I have had the same two EU zip files failing to upload since early Saturday my time - that would be Friday afternoon in UTC (Friday evening in British Daylight Time). Currently there are another 104 zips in line behind these two. Oh, well.
ID: 44294 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 44295 - Posted: 4 Jun 2012, 22:46:57 UTC - in response to Message 44294.  

In that case you're suffering from the "too-many-zips" problem, in which case it's up to you to fix it.

Thyme Lawn has posted several cures for upload problems, of which there are a few varieties.
To start with, try this one.


Backups: Here
ID: 44295 · Report as offensive     Reply Quote
Profile Greg van Paassen

Send message
Joined: 17 Nov 07
Posts: 142
Credit: 4,271,370
RAC: 0
Message 44298 - Posted: 5 Jun 2012, 11:13:33 UTC - in response to Message 44295.  

Thanks, Les, I appreciate the help.

I added the time-out option to cc_config.xml, and set Boinc to try three transfers at a time as well. It's not helping, though: when I turned network activity back on, the first three files changed status to 'Uploading', and then within two seconds changed back to 'Retry in 09:30:00'. (With the time being different for each, of course.) Since the upload fails straight away it can't be a time-out problem.

It's not the "hnndler" problem, either. The problem is with uploader1. Its http server appears to be running OK, but attempting to access the file upload handler returns HTTP 500, internal server error.

I've set the http_transfer_debug option and I will see what new insight that brings when the first time-out expires.
ID: 44298 · Report as offensive     Reply Quote
Profile Gordon Hartman

Send message
Joined: 31 Aug 04
Posts: 5
Credit: 351,767
RAC: 0
Message 44300 - Posted: 5 Jun 2012, 15:12:38 UTC - in response to Message 44298.  

I can't upload either!

6/5/2012 11:11:50 AM | climateprediction.net | Started upload of hadam3p_eu_8ck3_2005_1_007870650_0_12.zip
6/5/2012 11:11:52 AM | climateprediction.net | Temporarily failed upload of hadam3p_eu_8ck3_2005_1_007870650_0_12.zip: transient HTTP error
6/5/2012 11:11:52 AM | climateprediction.net | Backing off 5 hr 36 min 54 sec on upload of hadam3p_eu_8ck3_2005_1_007870650_0_12.zip





ID: 44300 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4346
Credit: 16,535,294
RAC: 5,887
Message 44302 - Posted: 5 Jun 2012, 18:17:06 UTC

Both EU and PNW zips uploading fine from here and have been all of the 4 day weekend.

Dave
ID: 44302 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 44303 - Posted: 5 Jun 2012, 18:23:22 UTC - in response to Message 44298.  

Greg

When I had a lot stacked up during the recent pileup, I waited until the two that wanted to start first had timed out, clicked on one of the others way down the queue, and then on Retry now. That one started and then another one that hadn't had a turn yet. Both of then completed their upload, and then the others followed as fast as a bunch of sheep trying to get out of a barn.
I think that the original two still hung around for a while, but eventually they uploaded too.


Backups: Here
ID: 44303 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 44304 - Posted: 5 Jun 2012, 20:23:39 UTC

I've just finished uploading a number of files to Oxford. These included PNW, EU, and a restart zip. On one computer I simultaneously downloaded the numerous files needed to start a new EU model.
All of these went through with no problems.

So I'm fairly sure that any transfer problems people are having are occurring either on their computer, or at one of the nodes between them and Oxford.

More information will be needed to help than just "I can't upload files". :)
If and when I get time/remember, I'll try and locate all of the help posts that have been made over the past few years, and make a sticky linking to them.


Backups: Here
ID: 44304 · Report as offensive     Reply Quote
Profile Greg van Paassen

Send message
Joined: 17 Nov 07
Posts: 142
Credit: 4,271,370
RAC: 0
Message 44305 - Posted: 6 Jun 2012, 5:58:34 UTC
Last modified: 6 Jun 2012, 6:04:48 UTC

Thanks again, Les.

I suspended all work and re-started the boinc client (i.e., the background process, not the boinc manager). That seems to have done the trick --- seven failures, but the eighth file uploaded OK and now the ninth file is uploading.

EDIT: The successful uploads are going to cpdn-upload2.oerc.ox.ac.uk. The problems are with uploader1.atm.ox.ac.uk.
ID: 44305 · Report as offensive     Reply Quote
Harri Liljeroos
Avatar

Send message
Joined: 9 Dec 05
Posts: 111
Credit: 12,038,780
RAC: 1,393
Message 44306 - Posted: 6 Jun 2012, 5:58:38 UTC

I'm experiencing also problems with uploads. I just watched zips from three different wu's try to transfer. Zip files from two wus went just fine, from on wu they stop after 2 seconds (zip files 4-12). This failing wu was finished on 3rd of June, so Boinc has been trying to send those zips for several days, the zip 13 has already uploaded successfully.

Here is some message log (I removed some lines for other projects):
cimw7718.cimcorp5.com

812	climateprediction.net	06.06.12 08:40:08	Started upload of hadam3p_eu_82s2_2002_1_007750214_1_4.zip	
813	climateprediction.net	06.06.12 08:40:08	Started upload of hadam3p_eu_82s2_2002_1_007750214_1_5.zip	
814	climateprediction.net	06.06.12 08:40:10	Temporarily failed upload of hadam3p_eu_82s2_2002_1_007750214_1_4.zip: HTTP error	
815	climateprediction.net	06.06.12 08:40:10	Backing off 10 hr 17 min 25 sec on upload of hadam3p_eu_82s2_2002_1_007750214_1_4.zip	
816	climateprediction.net	06.06.12 08:40:10	Temporarily failed upload of hadam3p_eu_82s2_2002_1_007750214_1_5.zip: HTTP error	
817	climateprediction.net	06.06.12 08:40:10	Backing off 6 hr 58 min 2 sec on upload of hadam3p_eu_82s2_2002_1_007750214_1_5.zip	
818	climateprediction.net	06.06.12 08:40:10	Started upload of hadam3p_eu_cic0_2009_1_007981905_0_2.zip	
819	climateprediction.net	06.06.12 08:40:10	Started upload of hadam3p_eu_d2mw_2006_1_007969626_0_1.zip	
820	climateprediction.net	06.06.12 08:42:11	Finished upload of hadam3p_eu_cic0_2009_1_007981905_0_2.zip	
821	climateprediction.net	06.06.12 08:42:11	Started upload of hadam3p_eu_cic0_2009_1_007981905_0_3.zip	
822	climateprediction.net	06.06.12 08:42:14	Finished upload of hadam3p_eu_d2mw_2006_1_007969626_0_1.zip	
823	climateprediction.net	06.06.12 08:42:14	Started upload of hadam3p_eu_cic0_2009_1_007981905_0_4.zip	
824	climateprediction.net	06.06.12 08:44:22	Finished upload of hadam3p_eu_cic0_2009_1_007981905_0_3.zip	
825	climateprediction.net	06.06.12 08:44:22	Started upload of hadam3p_eu_d2mw_2006_1_007969626_0_2.zip	
826	climateprediction.net	06.06.12 08:44:28	Finished upload of hadam3p_eu_cic0_2009_1_007981905_0_4.zip	
827	climateprediction.net	06.06.12 08:44:28	Started upload of hadam3p_eu_cic0_2009_1_007981905_0_5.zip	
833	climateprediction.net	06.06.12 08:45:44	Finished upload of hadam3p_eu_d2mw_2006_1_007969626_0_2.zip	
834	climateprediction.net	06.06.12 08:45:44	Started upload of hadam3p_eu_d2mw_2006_1_007969626_0_3.zip	
835	climateprediction.net	06.06.12 08:45:51	Finished upload of hadam3p_eu_cic0_2009_1_007981905_0_5.zip	
836	climateprediction.net	06.06.12 08:45:53	Started upload of hadam3p_eu_82s2_2002_1_007750214_1_6.zip	
837	climateprediction.net	06.06.12 08:45:54	Temporarily failed upload of hadam3p_eu_82s2_2002_1_007750214_1_6.zip: HTTP error	
838	climateprediction.net	06.06.12 08:45:54	Backing off 6 hr 29 min 1 sec on upload of hadam3p_eu_82s2_2002_1_007750214_1_6.zip	
839	climateprediction.net	06.06.12 08:45:54	Started upload of hadam3p_eu_82s2_2002_1_007750214_1_7.zip	
840	climateprediction.net	06.06.12 08:45:55	Temporarily failed upload of hadam3p_eu_82s2_2002_1_007750214_1_7.zip: HTTP error	
841	climateprediction.net	06.06.12 08:45:55	Backing off 6 hr 15 min 20 sec on upload of hadam3p_eu_82s2_2002_1_007750214_1_7.zip	
842	climateprediction.net	06.06.12 08:45:55	Started upload of hadam3p_eu_82s2_2002_1_007750214_1_8.zip	
843	climateprediction.net	06.06.12 08:45:56	Temporarily failed upload of hadam3p_eu_82s2_2002_1_007750214_1_8.zip: HTTP error	
844	climateprediction.net	06.06.12 08:45:56	Backing off 10 hr 15 min 43 sec on upload of hadam3p_eu_82s2_2002_1_007750214_1_8.zip	
845	climateprediction.net	06.06.12 08:47:03	Finished upload of hadam3p_eu_d2mw_2006_1_007969626_0_3.zip	
846	climateprediction.net	06.06.12 08:47:05	Started upload of hadam3p_eu_82s2_2002_1_007750214_1_9.zip	
847	climateprediction.net	06.06.12 08:47:05	Started upload of hadam3p_eu_82s2_2002_1_007750214_1_10.zip	
848	climateprediction.net	06.06.12 08:47:07	Temporarily failed upload of hadam3p_eu_82s2_2002_1_007750214_1_9.zip: HTTP error	
849	climateprediction.net	06.06.12 08:47:07	Backing off 3 hr 10 min 20 sec on upload of hadam3p_eu_82s2_2002_1_007750214_1_9.zip	
850	climateprediction.net	06.06.12 08:47:07	Temporarily failed upload of hadam3p_eu_82s2_2002_1_007750214_1_10.zip: HTTP error	
851	climateprediction.net	06.06.12 08:47:07	Backing off 1 hr 0 min 0 sec on upload of hadam3p_eu_82s2_2002_1_007750214_1_10.zip	
852	climateprediction.net	06.06.12 08:47:08	Started upload of hadam3p_eu_82s2_2002_1_007750214_1_11.zip	
853	climateprediction.net	06.06.12 08:47:08	Started upload of hadam3p_eu_82s2_2002_1_007750214_1_12.zip	
854	climateprediction.net	06.06.12 08:47:09	Temporarily failed upload of hadam3p_eu_82s2_2002_1_007750214_1_11.zip: HTTP error	
855	climateprediction.net	06.06.12 08:47:09	Backing off 1 hr 5 min 3 sec on upload of hadam3p_eu_82s2_2002_1_007750214_1_11.zip	
856	climateprediction.net	06.06.12 08:47:09	Temporarily failed upload of hadam3p_eu_82s2_2002_1_007750214_1_12.zip: HTTP error	
857	climateprediction.net	06.06.12 08:47:09	Backing off 2 hr 15 min 53 sec on upload of hadam3p_eu_82s2_2002_1_007750214_1_12.zip	
858	climateprediction.net	06.06.12 08:54:40	Resuming task hadam3p_eu_cic0_2009_1_007981905_0 using hadam3p_eu version 609	


ID: 44306 · Report as offensive     Reply Quote
Previous · 1 . . . 3 · 4 · 5 · 6 · 7 · 8 · 9 . . . 10 · Next

Message boards : Number crunching : Upload Failure

©2024 climateprediction.net