climateprediction.net home page
Stuck upload issue

Stuck upload issue

Message boards : Number crunching : Stuck upload issue
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 · Next

AuthorMessage
Profile Alan K

Send message
Joined: 22 Feb 06
Posts: 291
Credit: 14,738,424
RAC: 184
Message 57656 - Posted: 16 Jan 2018, 22:57:56 UTC - in response to Message 57655.  

Log file extract:

16/01/2018 20:36:28 | climateprediction.net | [http] [ID#5] Info: Connected to upload6.cpdn.org (158.97.9.11) port 80 (#5)
16/01/2018 20:36:28 | climateprediction.net | [http] [ID#5] Sent header to server: POST /cgi-bin/file_upload_handler HTTP/1.1
16/01/2018 20:36:28 | climateprediction.net | [http] [ID#5] Sent header to server: Host: upload6.cpdn.org
16/01/2018 20:36:28 | climateprediction.net | [http] [ID#5] Sent header to server: User-Agent: BOINC client (windows_x86_64 7.8.3)
16/01/2018 20:36:28 | climateprediction.net | [http] [ID#5] Sent header to server: Accept: */*
16/01/2018 20:36:28 | climateprediction.net | [http] [ID#5] Sent header to server: Accept-Encoding: deflate, gzip
16/01/2018 20:36:28 | climateprediction.net | [http] [ID#5] Sent header to server: Content-Type: application/x-www-form-urlencoded
16/01/2018 20:36:28 | climateprediction.net | [http] [ID#5] Sent header to server: Accept-Language: en_GB
16/01/2018 20:36:28 | climateprediction.net | [http] [ID#5] Sent header to server: Content-Length: 68301092
16/01/2018 20:36:28 | climateprediction.net | [http] [ID#5] Sent header to server: Expect: 100-continue
16/01/2018 20:36:28 | climateprediction.net | [http] [ID#5] Sent header to server:
16/01/2018 20:36:28 | climateprediction.net | [http] [ID#5] Received header from server: HTTP/1.1 100 Continue
16/01/2018 20:36:51 | climateprediction.net | [http] [ID#5] Info: Recv failure: Connection was reset
16/01/2018 20:36:51 | climateprediction.net | [http] [ID#5] Info: Closing connection 5
16/01/2018 20:36:51 | climateprediction.net | [http] HTTP error: Failure when receiving data from the peer
16/01/2018 20:36:52 | climateprediction.net | [file_xfer] http op done; retval -184 (transient HTTP error)
16/01/2018 20:36:52 | climateprediction.net | [file_xfer] file transfer status -184 (transient HTTP error)
16/01/2018 20:36:52 | climateprediction.net | Temporarily failed upload of wah2_cam25_a0ds_200405_18_691_011370045_1_r1231016019_17.zip: transient HTTP error
so I think Andy is correct. Maybe there is a large backlog.
ID: 57656 · Report as offensive     Reply Quote
Falcon4

Send message
Joined: 4 Nov 06
Posts: 5
Credit: 3,758,105
RAC: 0
Message 57670 - Posted: 18 Jan 2018, 20:43:52 UTC - in response to Message 57624.  

6 days later, I've still got the exact same files in exactly the same spot trying to upload...

wah2_cam25_a0ib_200405_18_691_011370208_0_r706661527_12.zip - 30.03% (31.56/105.10 MB)
wah2_cam25_a0ib_200405_18_691_011370208_0_r706661527_13.zip - 9.93% (10.45/105.17 MB)
wah2_cam25_a0jx_200405_18_691_011370266_0_r76671198_10.zip - 52.51% (55.16/105.13 MB)
wah2_cam25_a0ea_200405_18_691_011370063_0_r1526698435_14.zip - 31.20% (32.83/105.20 MB)
wah2_cam25_a0ht_200405_18_691_011370190_0_r177040747_1.zip - 21.42% (22.56/105.30 MB)

Not sure if I should try messing with it... I really just kinda want to abort/clear these tasks at this point, as it's taking longer to upload than it did to compute in the first place :(
ID: 57670 · Report as offensive     Reply Quote
Profile Alan K

Send message
Joined: 22 Feb 06
Posts: 291
Credit: 14,738,424
RAC: 184
Message 57671 - Posted: 18 Jan 2018, 23:01:09 UTC - in response to Message 57655.  

This is the message from that link just now:

<data_server_reply>
<status>1</status>
<message>no command</message>
</data_server_reply>

so no change at the moment.
ID: 57671 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 2463
Credit: 3,124,201
RAC: 383
Message 57673 - Posted: 19 Jan 2018, 8:50:59 UTC

Will add another message to the batch's card.
ID: 57673 · Report as offensive     Reply Quote
BetelgeuseFive

Send message
Joined: 31 Aug 04
Posts: 6
Credit: 1,982,972
RAC: 0
Message 57682 - Posted: 21 Jan 2018, 9:44:43 UTC

I have two stuck uploads, one of them for over a week, the other for a couple of days:

21/01/2018 10:41:08 | climateprediction.net | Started upload of wah2_cam25_a047_200405_18_689_011368595_0_r2068114942_7.zip
21/01/2018 10:41:32 | | Project communication failed: attempting access to reference site
21/01/2018 10:41:32 | climateprediction.net | Temporarily failed upload of wah2_cam25_a047_200405_18_689_011368595_0_r2068114942_7.zip: transient HTTP error
21/01/2018 10:41:32 | climateprediction.net | Backing off 03:59:04 on upload of wah2_cam25_a047_200405_18_689_011368595_0_r2068114942_7.zip
21/01/2018 10:41:35 | | Internet access OK - project servers may be temporarily down.
21/01/2018 10:42:07 | climateprediction.net | Started upload of wah2_cam25_a03s_200405_18_689_011368580_0_r616315434_17.zip
21/01/2018 10:42:31 | | Project communication failed: attempting access to reference site
21/01/2018 10:42:31 | climateprediction.net | Temporarily failed upload of wah2_cam25_a03s_200405_18_689_011368580_0_r616315434_17.zip: transient HTTP error
21/01/2018 10:42:31 | climateprediction.net | Backing off 04:12:33 on upload of wah2_cam25_a03s_200405_18_689_011368580_0_r616315434_17.zip
21/01/2018 10:42:32 | | Internet access OK - project servers may be temporarily down.

Anything I can do about this on my side or does this need to be resolved on the server side ?

Tom
ID: 57682 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 2463
Credit: 3,124,201
RAC: 383
Message 57683 - Posted: 21 Jan 2018, 15:38:24 UTC

Sometimes exiting BOINC manager and restarting it can shift stuck uploads but I suspect this is a server side problem.I messaged the project about it on Friday.
ID: 57683 · Report as offensive     Reply Quote
Profile Iain Inglis
Volunteer moderator

Send message
Joined: 16 Jan 10
Posts: 979
Credit: 3,103,609
RAC: 95
Message 57686 - Posted: 22 Jan 2018, 0:30:20 UTC - in response to Message 57683.  
Last modified: 22 Jan 2018, 0:36:00 UTC

Sometimes exiting BOINC manager and restarting it can shift stuck uploads but I suspect this is a server side problem.I messaged the project about it on Friday.

... my CAM upload has been stuck since 7 January: that CAM upload server needs a good kick with a steel-toecapped Dr Martens. I had the same error with a couple of the 145 x ~50 MB Zips for global batch #696, but they cleared after a day or so. It would have been an aggravating waste of time to calculate and upload most - but not all - of over 7 GB data. So whatever was done to the "global" server needs to be done to the CAM server, sharpish.

Edit: As it is, my CAM model has calculated and uploaded 17/18 x 100 MB - i.e. ~2 GB - which I assume is useless to the project without the missing Zip. I'm tempted to abort it, so at least someone else could re-run it.
ID: 57686 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 2463
Credit: 3,124,201
RAC: 383
Message 57690 - Posted: 22 Jan 2018, 8:56:04 UTC
Last modified: 22 Jan 2018, 9:36:36 UTC

So whatever was done to the "global" server needs to be done to the CAM server, sharpish.



I suggested a while ago that the uploads could be redirected to Oxford as was done with the batch destined for a Korean server. I have repeated that suggestion with a slightly stronger wording.

Edit:I have been told they don't have the capacity at Oxford to take those two batches at present but will investigate to see if there is a server somewhere else that can take the uploads.
ID: 57690 · Report as offensive     Reply Quote
Profile geophi
Volunteer moderator

Send message
Joined: 7 Aug 04
Posts: 1821
Credit: 36,393,036
RAC: 10
Message 57692 - Posted: 22 Jan 2018, 16:57:49 UTC

These stuck uploads have been happening occasionally for more than a few years now. But it seems to be the models with really big upload files that have the most problems. A server reboot seems to release the stuck uploads. The first time I remember a recurrent problem was with the ANZ runs back in March of 2014.
ID: 57692 · Report as offensive     Reply Quote
tat

Send message
Joined: 11 Jan 18
Posts: 5
Credit: 129,961
RAC: 0
Message 57748 - Posted: 31 Jan 2018, 18:49:02 UTC
Last modified: 31 Jan 2018, 18:50:01 UTC

Hi. Am new to the project and so am not sure whether the upload issue that I'm experiencing, is out of the ordinary or not.

The tasks reach 100% according to the transfer tab in boinc manager but then the project goes into backoff, before then starting the counter again at 0%

Have emboldened the terms I'm not used to seeing in the event log below. Thanks

    31/01/2018 16:50:47 | climateprediction.net | Started upload of wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip
    31/01/2018 16:50:47 | climateprediction.net | Started upload of wah2_sas50_q2jw_201612_13_707_011450425_1_r573430588_8.zip
    31/01/2018 16:50:47 | climateprediction.net | Sending scheduler request: To send trickle-up message.
    31/01/2018 16:50:47 | climateprediction.net | Not requesting tasks: "no new tasks" requested via Manager
    31/01/2018 16:50:49 | climateprediction.net | Scheduler request completed
    31/01/2018 16:52:08 | climateprediction.net | [error] Error reported by file upload server: can't write file wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip: Disk quota exceeded
    31/01/2018 16:52:08 | climateprediction.net | Temporarily failed upload of wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip: transient upload error
    31/01/2018 16:52:08 | climateprediction.net | Backing off 00:02:04 on upload of wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip
    31/01/2018 16:52:09 | climateprediction.net | [error] Error reported by file upload server: can't write file wah2_sas50_q2jw_201612_13_707_011450425_1_r573430588_8.zip: Disk quota exceeded
    31/01/2018 16:52:09 | climateprediction.net | Temporarily failed upload of wah2_sas50_q2jw_201612_13_707_011450425_1_r573430588_8.zip: transient upload error
    31/01/2018 16:52:09 | climateprediction.net | Backing off 00:02:28 on upload of wah2_sas50_q2jw_201612_13_707_011450425_1_r573430588_8.zip
    31/01/2018 16:54:13 | climateprediction.net | Started upload of wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip
    31/01/2018 16:54:38 | climateprediction.net | Started upload of wah2_sas50_q2jw_201612_13_707_011450425_1_r573430588_8.zip
    31/01/2018 16:55:07 | climateprediction.net | [error] Error reported by file upload server: can't write file wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip: Disk quota exceeded
    31/01/2018 16:55:07 | climateprediction.net | Temporarily failed upload of wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip: transient upload error
    31/01/2018 16:55:07 | climateprediction.net | Backing off 00:06:38 on upload of wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip
    31/01/2018 16:55:37 | climateprediction.net | [error] Error reported by file upload server: can't write file wah2_sas50_q2jw_201612_13_707_011450425_1_r573430588_8.zip: Disk quota exceeded
    31/01/2018 16:55:37 | climateprediction.net | Temporarily failed upload of wah2_sas50_q2jw_201612_13_707_011450425_1_r573430588_8.zip: transient upload error
    31/01/2018 16:55:37 | climateprediction.net | Backing off 00:04:53 on upload of wah2_sas50_q2jw_201612_13_707_011450425_1_r573430588_8.zip
    31/01/2018 17:24:21 | climateprediction.net | Started upload of wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip
    31/01/2018 17:24:21 | climateprediction.net | Started upload of wah2_sas50_q2jw_201612_13_707_011450425_1_r573430588_8.zip
    31/01/2018 17:26:22 | climateprediction.net | [error] Error reported by file upload server: can't write file wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip: Disk quota exceeded
    31/01/2018 17:26:22 | climateprediction.net | Temporarily failed upload of wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip: transient upload error
    31/01/2018 17:26:22 | climateprediction.net | Backing off 00:08:57 on upload of wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip
    31/01/2018 17:26:28 | climateprediction.net | [error] Error reported by file upload server: can't write file wah2_sas50_q2jw_201612_13_707_011450425_1_r573430588_8.zip: Disk quota exceeded
    31/01/2018 17:26:28 | climateprediction.net | Temporarily failed upload of wah2_sas50_q2jw_201612_13_707_011450425_1_r573430588_8.zip: transient upload error
    31/01/2018 17:26:28 | climateprediction.net | Backing off 00:11:09 on upload of wah2_sas50_q2jw_201612_13_707_011450425_1_r573430588_8.zip
    31/01/2018 17:35:20 | climateprediction.net | Started upload of wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip
    31/01/2018 17:37:26 | climateprediction.net | [error] Error reported by file upload server: can't write file wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip: Disk quota exceeded
    31/01/2018 17:37:26 | climateprediction.net | Temporarily failed upload of wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip: transient upload error
    31/01/2018 17:37:26 | climateprediction.net | Backing off 00:25:42 on upload of wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip
    31/01/2018 17:49:14 | climateprediction.net | Started upload of wah2_sas50_q2jw_201612_13_707_011450425_1_r573430588_8.zip
    31/01/2018 17:50:56 | climateprediction.net | [error] Error reported by file upload server: can't write file wah2_sas50_q2jw_201612_13_707_011450425_1_r573430588_8.zip: Disk quota exceeded
    31/01/2018 17:50:56 | climateprediction.net | Temporarily failed upload of wah2_sas50_q2jw_201612_13_707_011450425_1_r573430588_8.zip: transient upload error
    31/01/2018 17:50:56 | climateprediction.net | Backing off 00:24:50 on upload of wah2_sas50_q2jw_201612_13_707_011450425_1_r573430588_8.zip
    31/01/2018 18:12:22 | climateprediction.net | Started upload of wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip
    31/01/2018 18:14:40 | climateprediction.net | [error] Error reported by file upload server: can't write file wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip: Disk quota exceeded
    31/01/2018 18:14:40 | climateprediction.net | Temporarily failed upload of wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip: transient upload error
    31/01/2018 18:14:40 | climateprediction.net | Backing off 00:43:48 on upload of wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip

ID: 57748 · Report as offensive     Reply Quote
Profile astroWX
Volunteer moderator

Send message
Joined: 5 Aug 04
Posts: 1496
Credit: 92,841,900
RAC: 68
Message 57750 - Posted: 31 Jan 2018, 19:08:37 UTC - in response to Message 57748.  
Last modified: 31 Jan 2018, 19:10:17 UTC

I'm experiencing similar problems over the last half-day. I'm not yet aware of the problem(s).

Hang in with us. These things get sorted out sooner or later. (Servers are located around the globe and problems could be anywhere and of any type. [Curiously, I'm experiencing problems with EAS, SAS, EU, and NAM tasks -- currently, one task is uploading on each machine, albeit very slowly.])

Welcome to the project and the boards.
"We have met the enemy and he is us." -- Pogo
Greetings from coastal Washington state, the scenic US Pacific Northwest.
ID: 57750 · Report as offensive     Reply Quote
bernard_ivo

Send message
Joined: 18 Jul 13
Posts: 342
Credit: 9,943,922
RAC: 50
Message 57751 - Posted: 31 Jan 2018, 19:09:16 UTC - in response to Message 57748.  
Last modified: 31 Jan 2018, 19:10:29 UTC

I'm having the same issue. It's has been like this for 12hours (8:00 CET) Mine all are SAS50 batch 708, 707
ID: 57751 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 2463
Credit: 3,124,201
RAC: 383
Message 57754 - Posted: 31 Jan 2018, 20:53:22 UTC - in response to Message 57751.  
Last modified: 31 Jan 2018, 20:53:52 UTC

I noticed this morning, the site was down for a couple of hours. This may or may not be relevant. I have had one upload from 706 and one from 708 go through after the site came back on stream. They went through at the normal speed for me which is total upload speed of about 100Mbit/second but that is I suspect a limitation of my connection rather than anything to do with the project.
ID: 57754 · Report as offensive     Reply Quote
Speedy

Send message
Joined: 20 Jul 05
Posts: 21
Credit: 279,093
RAC: 75
Message 57755 - Posted: 31 Jan 2018, 22:40:53 UTC - in response to Message 57754.  

I noticed this morning, the site was down for a couple of hours. This may or may not be relevant. I have had one upload from 706 and one from 708 go through after the site came back on stream. They went through at the normal speed for me which is total upload speed of about 100Mbit/second but that is I suspect a limitation of my connection rather than anything to do with the project.

Dave I am curious to know how long take you to upload 45.97 meg looking at my manager takes just over 12 minutes but currently the project is in back off mode
ID: 57755 · Report as offensive     Reply Quote
Lockleys

Send message
Joined: 13 Jan 07
Posts: 187
Credit: 9,677,946
RAC: 71
Message 57756 - Posted: 1 Feb 2018, 0:03:43 UTC

I am having the same problem but they all eventually upload with the exception of a restart.zip for a cam25 model, batch 691 which is stuck at 27.63 percent.
ID: 57756 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 2463
Credit: 3,124,201
RAC: 383
Message 57757 - Posted: 1 Feb 2018, 7:05:59 UTC - in response to Message 57755.  

Dave I am curious to know how long take you to upload 45.97 meg looking at my manager takes just over 12 minutes but currently the project is in back off mode


Mine took 18minutes but there were two uploads going at once so I guess you can half that. Worth noting that I don't get any faster than that when I run a speed test on my connection.
ID: 57757 · Report as offensive     Reply Quote
keputnam

Send message
Joined: 31 Aug 04
Posts: 23
Credit: 3,253,607
RAC: 11
Message 57758 - Posted: 1 Feb 2018, 7:06:50 UTC

I'm now getting

1/31/2018 11:03:42 PM | climateprediction.net | [error] Error reported by file upload server: can't write file wah2_sas50_q3b2_201612_13_707_011451403_0_r448480803_11.zip: Disk quota exceeded

<sigh>

if it's not one project with capacity problems, it's another

ID: 57758 · Report as offensive     Reply Quote
Profile Thyme Lawn
Volunteer moderator

Send message
Joined: 5 Aug 04
Posts: 1276
Credit: 15,546,782
RAC: 6
Message 57759 - Posted: 1 Feb 2018, 9:49:26 UTC - in response to Message 57758.  

The project team are aware of the upload problem for WAH2 SAS50 batch 706, 707 and 708 files. Upload server upload2.cpdn.org has reached its disk quota and they are making more space available.
"The ultimate test of a moral society is the kind of world that it leaves to its children." - Dietrich Bonhoeffer
ID: 57759 · Report as offensive     Reply Quote
tat

Send message
Joined: 11 Jan 18
Posts: 5
Credit: 129,961
RAC: 0
Message 57762 - Posted: 1 Feb 2018, 13:28:13 UTC

Selfishly speaking, I'm glad it's not just me ;)

Thanks for the responses. And to those working hard on our behalf.
ID: 57762 · Report as offensive     Reply Quote
WB8ILI

Send message
Joined: 1 Sep 04
Posts: 130
Credit: 48,738,424
RAC: 234
Message 57764 - Posted: 1 Feb 2018, 18:52:41 UTC

I have had a stuck upload from batch 691 for so many weeks I have lost track.
ID: 57764 · Report as offensive     Reply Quote
Previous · 1 · 2 · 3 · 4 · 5 · Next

Message boards : Number crunching : Stuck upload issue

©2019 climateprediction.net