climateprediction.net home page
ANOTHER UPLOAD PROBLEM

ANOTHER UPLOAD PROBLEM

Message boards : Number crunching : ANOTHER UPLOAD PROBLEM
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 7 · 8 · 9 · 10 · 11 · 12 · 13 . . . 33 · Next

AuthorMessage
Niall

Send message
Joined: 18 Dec 13
Posts: 62
Credit: 1,078,935
RAC: 0
Message 49731 - Posted: 15 Aug 2014, 3:04:19 UTC

Indeed. I had a similar number. I now have a couple of zips showing a transient error, but those invariably clear, so I'm not worried. The server shows the number of WUs in progress dropping very fast.

From something posted on the BOINC forum, Richard Haselgrove (and team?) deserve kudos for a lot of hard work getting the servers back online. Nice work. Hope you get time for a beer or several.
ID: 49731 · Report as offensive     Reply Quote
Lockleys

Send message
Joined: 13 Jan 07
Posts: 195
Credit: 10,581,566
RAC: 0
Message 49732 - Posted: 15 Aug 2014, 6:19:45 UTC

The backlog of uploads is now slowly clearing, but I am seeing a problem with _13 files failing.
For example:
15/08/2014 01:47:07 | climateprediction.net | Started upload of hadam3p_eu_h71v_2013_1_008862964_0_13.zip
15/08/2014 01:54:55 | climateprediction.net | Finished upload of hadam3p_eu_iawg_2013_1_008778564_2_6.zip
15/08/2014 01:54:55 | climateprediction.net | Started upload of hadam3p_eu_h71v_2013_1_008862964_0_5.zip
15/08/2014 02:00:42 | climateprediction.net | Sending scheduler request: To send trickle-up message.
15/08/2014 02:00:42 | climateprediction.net | Not requesting tasks: "no new tasks" requested via Manager
15/08/2014 02:00:46 | climateprediction.net | Scheduler request completed
15/08/2014 02:10:11 | climateprediction.net | [error] Error reported by file upload server: can't open file /storage/cpdn-restarts/incoming/uploader/hadam3p_eu_h71v_2013_1_008862964_0_13.zip: No such file or directory
15/08/2014 02:10:11 | climateprediction.net | Temporarily failed upload of hadam3p_eu_h71v_2013_1_008862964_0_13.zip: transient upload error
15/08/2014 02:10:11 | climateprediction.net | Backing off 00:05:12 on upload of hadam3p_eu_h71v_2013_1_008862964_0_13.zip
All my _13 files are failing in the same way. Other files are uploading just fine.
ID: 49732 · Report as offensive     Reply Quote
Profile Byron Leigh Hatch @ team Carl ...
Avatar

Send message
Joined: 17 Aug 04
Posts: 289
Credit: 44,103,664
RAC: 0
Message 49733 - Posted: 15 Aug 2014, 6:44:29 UTC

approximately 97 % of my Zip files have uploaded. Just 3 left in the BOINC transfer tab. here are some messages I'm getting in the BOINC event Log.


14/08/2014 10:50:27 PM | climateprediction.net | Started upload of hadam3p_eu_o4rv_2013_1_008832772_0_5.zip
14/08/2014 10:51:05 PM | climateprediction.net | Temporarily failed upload of hadam3p_eu_o4rv_2013_1_008832772_0_5.zip: transient HTTP error
14/08/2014 10:51:05 PM | climateprediction.net | Backing off 00:03:58 on upload of hadam3p_eu_o4rv_2013_1_008832772_0_5.zip
14/08/2014 10:51:05 PM | | Project communication failed: attempting access to reference site
14/08/2014 10:51:09 PM | | Internet access OK - project servers may be temporarily down.
14/08/2014 10:55:04 PM | climateprediction.net | Started upload of hadam3p_eu_o4rv_2013_1_008832772_0_5.zip
14/08/2014 10:55:27 PM | climateprediction.net | Temporarily failed upload of hadam3p_eu_o4rv_2013_1_008832772_0_5.zip: transient HTTP error
14/08/2014 10:55:27 PM | climateprediction.net | Backing off 00:04:37 on upload of hadam3p_eu_o4rv_2013_1_008832772_0_5.zip
14/08/2014 10:55:28 PM | | Project communication failed: attempting access to reference site
14/08/2014 10:55:30 PM | | Internet access OK - project servers may be temporarily down.
14/08/2014 11:00:04 PM | climateprediction.net | Started upload of hadam3p_eu_o4rv_2013_1_008832772_0_5.zip
14/08/2014 11:07:34 PM | climateprediction.net | Finished upload of hadam3p_eu_o4rv_2013_1_008832772_0_5.zip
14/08/2014 11:29:43 PM | climateprediction.net | Started upload of hadam3p_eu_o8qu_2013_1_008837919_0_13.zip
14/08/2014 11:29:45 PM | climateprediction.net | Started upload of hadam3p_eu_o8vv_2013_1_008838100_0_13.zip
14/08/2014 11:32:56 PM | climateprediction.net | [error] Error reported by file upload server: can't open file /storage/cpdn-restarts/incoming/uploader/hadam3p_eu_o8qu_2013_1_008837919_0_13.zip: No such file or directory
14/08/2014 11:32:56 PM | climateprediction.net | Temporarily failed upload of hadam3p_eu_o8qu_2013_1_008837919_0_13.zip: transient upload error
14/08/2014 11:32:56 PM | climateprediction.net | Backing off 03:52:14 on upload of hadam3p_eu_o8qu_2013_1_008837919_0_13.zip
14/08/2014 11:32:57 PM | climateprediction.net | Started upload of hadam3p_eu_obgq_2013_1_008841443_0_13.zip
14/08/2014 11:34:00 PM | climateprediction.net | [error] Error reported by file upload server: can't open file /storage/cpdn-restarts/incoming/uploader/hadam3p_eu_o8vv_2013_1_008838100_0_13.zip: No such file or directory
14/08/2014 11:34:00 PM | climateprediction.net | Temporarily failed upload of hadam3p_eu_o8vv_2013_1_008838100_0_13.zip: transient upload error
14/08/2014 11:34:00 PM | climateprediction.net | Backing off 03:59:51 on upload of hadam3p_eu_o8vv_2013_1_008838100_0_13.zip
14/08/2014 11:35:26 PM | climateprediction.net | [error] Error reported by file upload server: can't open file /storage/cpdn-restarts/incoming/uploader/hadam3p_eu_obgq_2013_1_008841443_0_13.zip: No such file or directory
14/08/2014 11:35:26 PM | climateprediction.net | Temporarily failed upload of hadam3p_eu_obgq_2013_1_008841443_0_13.zip: transient upload error
14/08/2014 11:35:26 PM | climateprediction.net | Backing off 03:39:50 on upload of hadam3p_eu_obgq_2013_1_008841443_0_13.zip
ID: 49733 · Report as offensive     Reply Quote
Helmer Bryd

Send message
Joined: 16 Aug 04
Posts: 156
Credit: 9,035,872
RAC: 2,928
Message 49734 - Posted: 15 Aug 2014, 7:53:18 UTC

Yeah, same here;
15-Aug-2014 09:46:53 [climateprediction.net] [error] Error reported by file upload server: can't open file /storage/cpdn-restarts/incoming/uploader/hadam3p_eu_l685_2013_1_008819294_1_13.zip: No such file or directory

ID: 49734 · Report as offensive     Reply Quote
Richard Haselgrove

Send message
Joined: 1 Jan 07
Posts: 1032
Credit: 36,238,077
RAC: 13,062
Message 49735 - Posted: 15 Aug 2014, 8:36:49 UTC - in response to Message 49731.  

From something posted on the BOINC forum, Richard Haselgrove (and team?) deserve kudos for a lot of hard work getting the servers back online. Nice work. Hope you get time for a beer or several.

Not me. I'm simply a messenger passing information back and forth. If that small cog in the wheel was helpful to anyone, then it was worth doing.

On that subject, I see that the problem of the _13.zip file uploads failing with the error "No such file or directory" has already been passed directly into the lab by one of the other messengers. I'm sure the team will be wanting to review the performance of the new database server first this morning, but after that there should be time (personal guess) to look at uploads too.
ID: 49735 · Report as offensive     Reply Quote
Eirik Redd

Send message
Joined: 31 Aug 04
Posts: 391
Credit: 219,896,461
RAC: 649
Message 49737 - Posted: 15 Aug 2014, 12:14:07 UTC

Fearless prediction -- since it is Friday.

Servers will fail at about 1700 - 1800 UTC .

Anybody on for some serious betting?
ID: 49737 · Report as offensive     Reply Quote
Profile Thyme Lawn
Volunteer moderator

Send message
Joined: 5 Aug 04
Posts: 1283
Credit: 15,824,334
RAC: 0
Message 49738 - Posted: 15 Aug 2014, 14:37:34 UTC - in response to Message 49735.  

On that subject, I see that the problem of the _13.zip file uploads failing with the error "No such file or directory" has already been passed directly into the lab by one of the other messengers.

My _13.zip uploads started working at around 0930 UTC.
"The ultimate test of a moral society is the kind of world that it leaves to its children." - Dietrich Bonhoeffer
ID: 49738 · Report as offensive     Reply Quote
Eirik Redd

Send message
Joined: 31 Aug 04
Posts: 391
Credit: 219,896,461
RAC: 649
Message 49739 - Posted: 15 Aug 2014, 14:52:07 UTC - in response to Message 49738.  

And what will that do?

maybe some "... 13 'files get uploaded -- good.

The totally broken upload situation -- let us pretend --. :)


On that subject, I see that the problem of the _13.zip file uploads failing with the error "No such file or directory" has already been passed directly into the lab by one of the other messengers.

My _13.zip uploads started working at around 0930 UTC.


ID: 49739 · Report as offensive     Reply Quote
Profile JIM

Send message
Joined: 31 Dec 07
Posts: 1152
Credit: 22,287,362
RAC: 5,696
Message 49740 - Posted: 16 Aug 2014, 4:56:21 UTC
Last modified: 16 Aug 2014, 5:22:19 UTC

We have another upload problem. The zip files in my transfer tab that built up during the server outage have stopped clearing and are building up again. Is this just server overload, or is there a new problem?

OPPS.

I just saw that cpdn.uploader2orec is listed at not running. Hopefully it won�t be Monday before the staff can get to it and get it running again>
ID: 49740 · Report as offensive     Reply Quote
Niall

Send message
Joined: 18 Dec 13
Posts: 62
Credit: 1,078,935
RAC: 0
Message 49741 - Posted: 16 Aug 2014, 19:36:39 UTC - in response to Message 49740.  

Agreed. This time it's the 13.zips that seem to be moving, while it appears everything else is stuck.
ID: 49741 · Report as offensive     Reply Quote
Lockleys

Send message
Joined: 13 Jan 07
Posts: 195
Credit: 10,581,566
RAC: 0
Message 49742 - Posted: 16 Aug 2014, 22:05:43 UTC - in response to Message 49741.  

Yep, same here.
Agreed. This time it's the 13.zips that seem to be moving, while it appears everything else is stuck.

ID: 49742 · Report as offensive     Reply Quote
Profile JIM

Send message
Joined: 31 Dec 07
Posts: 1152
Credit: 22,287,362
RAC: 5,696
Message 49743 - Posted: 17 Aug 2014, 2:19:16 UTC

The zip 13 files are definitely being accepted while all other hadam3p zips are hanging. I have had 2 WU�s finish since this problem started and there is a zip file 6, 7, 8, 9, two 10�s, two 11�s and two 12�s stuck in my transfer tab, but, no sign of the 13�s. They uploaded fine.

ID: 49743 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 49744 - Posted: 17 Aug 2014, 4:27:52 UTC

What is the server name that the stuck files want to upload to?

ID: 49744 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4472
Credit: 18,448,326
RAC: 22,385
Message 49745 - Posted: 17 Aug 2014, 5:30:34 UTC

I think it is cpdnupload2.oerc but I don't know the right file to look at in my BOINC folder to find it.
ID: 49745 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 49746 - Posted: 17 Aug 2014, 5:51:57 UTC - in response to Message 49745.  

Hi Dave

The file is client_state.xml
I always copy this and paste it "elsewhere", then look at the copy. Just to be safe. :)

Scan it for the 4 character file name, and keep going until you reach the upload section.

If it is that uploader, then it's a Monday job. :(
Possibly the storage section needs re-mounting.


ID: 49746 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4472
Credit: 18,448,326
RAC: 22,385
Message 49747 - Posted: 17 Aug 2014, 6:49:59 UTC - in response to Message 49746.  

Thanks Les,

It is http://cpdn-upload2.oerc.ox.ac.uk/cgi-bin/file_upload_handler
which is had gone red on the Server Status page which is what made me suspect it.

I think those of us who have been around a while had already worked out that it was a Monday, post 9.00am job. I have suspended internet access for BOINC and will wait an hour or so after the colour has changed before trying again.
ID: 49747 · Report as offensive     Reply Quote
Philipp Marc Neuhaus

Send message
Joined: 27 Aug 04
Posts: 3
Credit: 1,954,812
RAC: 397
Message 49749 - Posted: 17 Aug 2014, 12:20:27 UTC

Upload problem - after a lot of work units could be uploaded last week, again since a couple of days some are remaining with the message transient HTTP error - see excerpt of log below:

Sun Aug 17 13:38:23 2014 | climateprediction.net | Temporarily failed upload of hadam3p_eu_l9o4_2013_1_008823757_0_6.zip: transient HTTP error
Sun Aug 17 13:38:23 2014 | climateprediction.net | Backing off 04:40:08 on upload of hadam3p_eu_l9o4_2013_1_008823757_0_6.zip
Sun Aug 17 13:38:31 2014 | climateprediction.net | Temporarily failed upload of hadam3p_eu_lbw2_2013_1_008826635_1_10.zip: transient HTTP error
Sun Aug 17 13:38:31 2014 | climateprediction.net | Backing off 04:50:51 on upload of hadam3p_eu_lbw2_2013_1_008826635_1_10.zip
Sun Aug 17 13:38:32 2014 | | Internet access OK - project servers may be temporarily down.
Sun Aug 17 13:38:39 2014 | climateprediction.net | Started upload of hadam3p_eu_lbw2_2013_1_008826635_1_10.zip
Sun Aug 17 13:38:39 2014 | climateprediction.net | Started upload of hadam3p_eu_l9o4_2013_1_008823757_0_6.zip
Sun Aug 17 13:39:56 2014 | | Project communication failed: attempting access to reference site
Sun Aug 17 13:39:56 2014 | climateprediction.net | Temporarily failed upload of hadam3p_eu_lbw2_2013_1_008826635_1_10.zip: transient HTTP error
Sun Aug 17 13:39:56 2014 | climateprediction.net | Backing off 04:39:06 on upload of hadam3p_eu_lbw2_2013_1_008826635_1_10.zip
Sun Aug 17 13:39:56 2014 | climateprediction.net | Temporarily failed upload of hadam3p_eu_l9o4_2013_1_008823757_0_6.zip: transient HTTP error
Sun Aug 17 13:39:56 2014 | climateprediction.net | Backing off 05:07:26 on upload of hadam3p_eu_l9o4_2013_1_008823757_0_6.zip
Sun Aug 17 13:39:57 2014 | | Internet access OK - project servers may be temporarily down.
Sun Aug 17 13:40:47 2014 | | Re-reading cc_config.xml
Sun Aug 17 13:40:47 2014 | | cc_config.xml not found - using defaults
Sun Aug 17 13:40:47 2014 | | log flags: file_xfer, sched_ops, task
Sun Aug 17 13:40:54 2014 | climateprediction.net | Started upload of hadam3p_eu_lbw2_2013_1_008826635_1_10.zip
Sun Aug 17 13:40:54 2014 | climateprediction.net | Started upload of hadam3p_eu_l9o4_2013_1_008823757_0_6.zip
Sun Aug 17 13:42:05 2014 | | Project communication failed: attempting access to reference site
Sun Aug 17 13:42:05 2014 | climateprediction.net | Temporarily failed upload of hadam3p_eu_l9o4_2013_1_008823757_0_6.zip: transient HTTP error
Sun Aug 17 13:42:05 2014 | climateprediction.net | Backing off 03:44:57 on upload of hadam3p_eu_l9o4_2013_1_008823757_0_6.zip
Sun Aug 17 13:42:06 2014 | | Internet access OK - project servers may be temporarily down.
Sun Aug 17 13:42:06 2014 | climateprediction.net | Temporarily failed upload of hadam3p_eu_lbw2_2013_1_008826635_1_10.zip: transient HTTP error
Sun Aug 17 13:42:06 2014 | climateprediction.net | Backing off 05:48:43 on upload of hadam3p_eu_lbw2_2013_1_008826635_1_10.zip
Sun Aug 17 13:42:36 2014 | climateprediction.net | Started upload of hadam3p_eu_lbw2_2013_1_008826635_1_10.zip
Sun Aug 17 13:42:36 2014 | climateprediction.net | Started upload of hadam3p_eu_l9o4_2013_1_008823757_0_6.zip
Sun Aug 17 13:43:44 2014 | | Project communication failed: attempting access to reference site
Sun Aug 17 13:43:44 2014 | climateprediction.net | Temporarily failed upload of hadam3p_eu_l9o4_2013_1_008823757_0_6.zip: transient HTTP error
Sun Aug 17 13:43:44 2014 | climateprediction.net | Backing off 05:51:52 on upload of hadam3p_eu_l9o4_2013_1_008823757_0_6.zip
Sun Aug 17 13:43:46 2014 | | Internet access OK - project servers may be temporarily down.
Sun Aug 17 13:43:52 2014 | | Project communication failed: attempting access to reference site
Sun Aug 17 13:43:52 2014 | climateprediction.net | Temporarily failed upload of hadam3p_eu_lbw2_2013_1_008826635_1_10.zip: transient HTTP error
Sun Aug 17 13:43:52 2014 | climateprediction.net | Backing off 03:46:31 on upload of hadam3p_eu_lbw2_2013_1_008826635_1_10.zip
Sun Aug 17 13:43:54 2014 | | Internet access OK - project servers may be temporarily down.
Sun Aug 17 13:43:55 2014 | climateprediction.net | Started upload of hadam3p_eu_lbw2_2013_1_008826635_1_10.zip
Sun Aug 17 13:43:55 2014 | climateprediction.net | Started upload of hadam3p_eu_l9o4_2013_1_008823757_0_6.zip
Sun Aug 17 13:45:06 2014 | | Project communication failed: attempting access to reference site
Sun Aug 17 13:45:06 2014 | climateprediction.net | Temporarily failed upload of hadam3p_eu_lbw2_2013_1_008826635_1_10.zip: transient HTTP error
Sun Aug 17 13:45:06 2014 | climateprediction.net | Backing off 05:17:53 on upload of hadam3p_eu_lbw2_2013_1_008826635_1_10.zip
Sun Aug 17 13:45:07 2014 | | Internet access OK - project servers may be temporarily down.
Sun Aug 17 13:45:11 2014 | | Project communication failed: attempting access to reference site
Sun Aug 17 13:45:11 2014 | climateprediction.net | Temporarily failed upload of hadam3p_eu_l9o4_2013_1_008823757_0_6.zip: transient HTTP error
Sun Aug 17 13:45:11 2014 | climateprediction.net | Backing off 04:01:20 on upload of hadam3p_eu_l9o4_2013_1_008823757_0_6.zip
Sun Aug 17 13:45:12 2014 | | Internet access OK - project servers may be temporarily down.

ID: 49749 · Report as offensive     Reply Quote
Profile JIM

Send message
Joined: 31 Dec 07
Posts: 1152
Credit: 22,287,362
RAC: 5,696
Message 49754 - Posted: 18 Aug 2014, 11:23:10 UTC

I see that server cpdnupload2.orec is listed as being back �up�. This is good news as I presently have more than 50 zip files stuck in my transfer tabs on 3 machines ready to go. Backlog of zip files starting to clear. Lets hope they all go this time.

ID: 49754 · Report as offensive     Reply Quote
Profile JIM

Send message
Joined: 31 Dec 07
Posts: 1152
Credit: 22,287,362
RAC: 5,696
Message 49757 - Posted: 18 Aug 2014, 12:38:14 UTC

The transfer tab on my fastest machine now completely empty. Second machine is uploading. Some of those zip files are had been there for more than a week.

ID: 49757 · Report as offensive     Reply Quote
Profile Byron Leigh Hatch @ team Carl ...
Avatar

Send message
Joined: 17 Aug 04
Posts: 289
Credit: 44,103,664
RAC: 0
Message 49758 - Posted: 18 Aug 2014, 13:15:00 UTC

same here also I got some new work :)
ID: 49758 · Report as offensive     Reply Quote
Previous · 1 . . . 7 · 8 · 9 · 10 · 11 · 12 · 13 . . . 33 · Next

Message boards : Number crunching : ANOTHER UPLOAD PROBLEM

©2024 cpdn.org