climateprediction.net home page
Upload failures

Upload failures

Message boards : Number crunching : Upload failures
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 12 · 13 · 14 · 15 · 16 · 17 · 18 · Next

AuthorMessage
Mephist0

Send message
Joined: 21 Feb 08
Posts: 45
Credit: 7,929,915
RAC: 101
Message 60744 - Posted: 30 Jul 2019, 10:03:28 UTC

I will move my boinc installation to another computer that does not need a proxy server in between to see if the behaviour changes since i have had issues a long time now... :)
ID: 60744 · Report as offensive     Reply Quote
Mephist0

Send message
Joined: 21 Feb 08
Posts: 45
Credit: 7,929,915
RAC: 101
Message 60746 - Posted: 30 Jul 2019, 11:19:28 UTC

Ok, so now on a new computer.. But still same issue.. Good now atleast i know its not the proxy server :)

So i guess the project still have issues with transient HTTP errors. :(
ID: 60746 · Report as offensive     Reply Quote
Mephist0

Send message
Joined: 21 Feb 08
Posts: 45
Credit: 7,929,915
RAC: 101
Message 60747 - Posted: 30 Jul 2019, 12:05:34 UTC - in response to Message 60597.  

OK, update:

The issue is one of bandwidth onto the system. We are constantly running 146 upload processes at the moment and have been since I announced on the board that we were restarting jasmin. This is one of the issues around backing up when not running.
We are investigating how we can setup another upload server within the infrastructure that can be run as part of a load balancing pair behind the jasmin-upload name.


David


Out of curiosity.. How much bandwidth does this server / or connection have available?

Is there some load graph available to view? :) I wanted to know if there is any hour during the day where it might be easier to upload results (since im manually uploading).
ID: 60747 · Report as offensive     Reply Quote
Mephist0

Send message
Joined: 21 Feb 08
Posts: 45
Credit: 7,929,915
RAC: 101
Message 60748 - Posted: 30 Jul 2019, 13:02:54 UTC

Sorry for all the posts..

My issues is resolved now. Turns out the network i ran all my computers on have a proxy and that one interferes with the transfers in some way. When i moved the files to my laptop and connected direct to the internet instead of my work network problem was resolved!

My 98 files are on the way to upload now. Getting around 10 Mbit in speed on a 100 Mbit line :) Hope they will register since they will upload on another machine...
ID: 60748 · Report as offensive     Reply Quote
Mephist0

Send message
Joined: 21 Feb 08
Posts: 45
Credit: 7,929,915
RAC: 101
Message 60749 - Posted: 30 Jul 2019, 14:12:11 UTC

For everyones info..

sam50 units are limited to 1 MiB/s in upload speed
anz50 units are limitet to 100 KiB/s in upload speed..

Observed sitting waiting for 98 files to upload :D
ID: 60749 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 2680
Credit: 3,259,434
RAC: 2,149
Message 60750 - Posted: 30 Jul 2019, 14:37:23 UTC - in response to Message 60749.  

For everyones info..

sam50 units are limited to 1 MiB/s in upload speed
anz50 units are limitet to 100 KiB/s in upload speed..

Observed sitting waiting for 98 files to upload :D


So for me, my bored band is the limiting factor.
ID: 60750 · Report as offensive     Reply Quote
Mephist0

Send message
Joined: 21 Feb 08
Posts: 45
Credit: 7,929,915
RAC: 101
Message 60751 - Posted: 30 Jul 2019, 19:41:37 UTC - in response to Message 60750.  

For everyones info..

sam50 units are limited to 1 MiB/s in upload speed
anz50 units are limitet to 100 KiB/s in upload speed..

Observed sitting waiting for 98 files to upload :D


So for me, my bored band is the limiting factor.


So whats your connection speed?
A while ago i had only mobile dongle to upload from a server park.. 3 Mbit (aprox 300KB/s).. THAT took a LOOOOONG time.... :D
ID: 60751 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7183
Credit: 22,883,095
RAC: 1,397
Message 60752 - Posted: 30 Jul 2019, 21:32:29 UTC - in response to Message 60751.  

Ha! You call that slow?
For years I was getting about 40 KB/s on a good day, and around 10-14 KB/s on a bad day.

Now that I've got super fast fiber, I get somewhere over a 100KB/s. (It's been so long since there was work that I've forgotten the exact number.)
ID: 60752 · Report as offensive     Reply Quote
Profile Alan K

Send message
Joined: 22 Feb 06
Posts: 328
Credit: 16,457,266
RAC: 4,170
Message 60754 - Posted: 30 Jul 2019, 22:20:26 UTC - in response to Message 60749.  

Rercent (today) upload to Jasmin:

30/07/2019 23:13:05 | climateprediction.net | Finished upload of wah2_sam50_n6no_201612_25_822_011884633_0_r2139349355_23.zip
30/07/2019 23:13:05 | climateprediction.net | [file_xfer] Throughput 744183 bytes/sec
ID: 60754 · Report as offensive     Reply Quote
Speedy

Send message
Joined: 20 Jul 05
Posts: 25
Credit: 354,898
RAC: 4
Message 60755 - Posted: 30 Jul 2019, 23:00:33 UTC - in response to Message 60749.  

For everyones info..

sam50 units are limited to 1 MiB/s in upload speed
anz50 units are limitet to 100 KiB/s in upload speed..

Observed sitting waiting for 98 files to upload :D

To see if you can increase your upload speed you could try changing Max files per project to 1. It may not have any effect but it's worth a try
<max_file_xfers_per_project>1</max_file_xfers_per_project>

ID: 60755 · Report as offensive     Reply Quote
Jean-David Beyer

Send message
Joined: 5 Aug 04
Posts: 376
Credit: 3,595,519
RAC: 201
Message 60756 - Posted: 31 Jul 2019, 0:20:35 UTC - in response to Message 60752.  
Last modified: 31 Jul 2019, 0:23:44 UTC

Right now, according to climateprediction.net I am getting:

Average upload rate 195.4 KB/sec
Average download rate 3390.97 KB/sec

Now my Verizon FiOS Internet connection gets:

Date Download Upload Test Server
7/30/2019 8:18:57 PM 81.62 Mbps 86.98 Mbps New York City, NY
6/25/2019 9:03:54 AM 82.02 Mbps 87.87 Mbps New York City, NY
6/14/2019 11:56:38 PM 82.44 Mbps 87.53 Mbps New York City, NY
6/1/2019 4:43:44 PM 82.44 Mbps 88.74 Mbps New York City, NY
ID: 60756 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 2680
Credit: 3,259,434
RAC: 2,149
Message 60761 - Posted: 31 Jul 2019, 4:53:37 UTC

Right now I am getting 88KB/s upload. I get about 5MB/s download on a clear day with a following wind.
ID: 60761 · Report as offensive     Reply Quote
Mephist0

Send message
Joined: 21 Feb 08
Posts: 45
Credit: 7,929,915
RAC: 101
Message 60762 - Posted: 31 Jul 2019, 9:07:16 UTC - in response to Message 60755.  

For everyones info..

sam50 units are limited to 1 MiB/s in upload speed
anz50 units are limitet to 100 KiB/s in upload speed..

Observed sitting waiting for 98 files to upload :D

To see if you can increase your upload speed you could try changing Max files per project to 1. It may not have any effect but it's worth a try
<max_file_xfers_per_project>1</max_file_xfers_per_project>


Yeah i had it at 1 when i was trying to fix my problems to not cause to many attempts on the climate servers. i got Max 1 MiB/s .. I had to increase the transfers slots to 10 since i was uploading both anz50 and sam50 units and it was taking forever. When i was transfering sam50 units only i got 10 MiB/s upload speed when i only were transfering anz50 i got a total of 1 MiB/s for all 10 transfers so there is defenatly a cap :) I normally have my transfers set to 2 (default)
ID: 60762 · Report as offensive     Reply Quote
Mephist0

Send message
Joined: 21 Feb 08
Posts: 45
Credit: 7,929,915
RAC: 101
Message 60763 - Posted: 31 Jul 2019, 11:37:48 UTC - in response to Message 60752.  

Ha! You call that slow?
For years I was getting about 40 KB/s on a good day, and around 10-14 KB/s on a bad day.

Now that I've got super fast fiber, I get somewhere over a 100KB/s. (It's been so long since there was work that I've forgotten the exact number.)


Haha.. Yeah it was slow for me since i had to be in the datacenter until the uploads completed and it was hundreds of files to upload at once ;)

You say "super fast fiber" ... Now i guess that the bottleneck is not your fibre connection but the CPDN bandwith cap instead :)

I'm on fibre too. I can get 1 Gbit/s but i dont need that kind of speed. I'm paying 25$/month for 100 Mbit down and 10 Mbit up. I don't need faster than that.. An awesome plus for this connection is that my real speed is 135/17 Mbit so a lot more than i pay for :)
ID: 60763 · Report as offensive     Reply Quote
Misty

Send message
Joined: 14 Feb 06
Posts: 50
Credit: 7,856,758
RAC: 77
Message 60764 - Posted: 31 Jul 2019, 13:23:07 UTC - in response to Message 60715.  

All zips for this cam25 model are failing to upload. Should I wait or abort?

Aborted stuck zips (##13 & 14) & now showing as Success/Completed. I hope no data were lost.
ID: 60764 · Report as offensive     Reply Quote
Profile Iain Inglis
Volunteer moderator

Send message
Joined: 16 Jan 10
Posts: 1013
Credit: 5,146,611
RAC: 1,975
Message 60765 - Posted: 31 Jul 2019, 20:06:54 UTC - in response to Message 60764.  

All zips for this cam25 model are failing to upload. Should I wait or abort?

Aborted stuck zips (##13 & 14) & now showing as Success/Completed. I hope no data were lost.

I sympathise with your position, having had a CAM25 stuck for weeks. However, on 27th July some of the Zips uploaded successfully so I’m leaving the remaining three transfers to continue attempting to upload. Aborting the transfer does destroy the data but I don’t blame anyone for doing that - it is the (sub-)project’s responsibility to provide the resources to accept data and if they don’t then they may lose data. Their fault I would say.
ID: 60765 · Report as offensive     Reply Quote
ian

Send message
Joined: 23 Feb 06
Posts: 19
Credit: 5,721,476
RAC: 602
Message 60869 - Posted: 26 Aug 2019, 19:52:51 UTC

my last running work unit (on all of my machines) -a cam25 - is now stuck on trickle up!! suppose I will just have to sit and wait AGAIN for it to clear and for more work.
ID: 60869 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7183
Credit: 22,883,095
RAC: 1,397
Message 60870 - Posted: 26 Aug 2019, 21:16:44 UTC

Are your SURE that it's a trickle, and not a zip file?
Vast difference, and instructions for dealing with this Cam zip problem have been posted a number of times.

Reports say that Aborting the stuck zip upload and then waiting, have resulted in a Successful completion.
ID: 60870 · Report as offensive     Reply Quote
Mario

Send message
Joined: 30 Apr 18
Posts: 3
Credit: 242,814
RAC: 3
Message 60898 - Posted: 5 Sep 2019, 1:47:00 UTC

I have 5 zip files that have become stuck in the transfer cue.

I'm trying to be patient and not abort anything prematurly.

9/2/2019 5:21:21 PM | climateprediction.net | Started upload of wah2_cam25_a08d_200405_18_691_011369850_0_r1625567930_15.zip
9/2/2019 5:21:43 PM | | Project communication failed: attempting access to reference site
9/2/2019 5:21:43 PM | climateprediction.net | Temporarily failed upload of wah2_cam25_a08d_200405_18_691_011369850_0_r1625567930_15.zip: transient HTTP error
9/2/2019 5:21:43 PM | climateprediction.net | Backing off 03:33:21 on upload of wah2_cam25_a08d_200405_18_691_011369850_0_r1625567930_15.zip
9/2/2019 5:21:45 PM | | Internet access OK - project servers may be temporarily down.
9/2/2019 6:56:35 PM | | Suspending computation - CPU is busy
9/2/2019 6:56:45 PM | | Resuming computation
9/2/2019 6:57:05 PM | | Suspending computation - CPU is busy
9/2/2019 6:57:35 PM | | Resuming computation
9/2/2019 8:06:31 PM | | Suspending GPU computation - computer is in use
9/2/2019 8:08:23 PM | climateprediction.net | Started upload of wah2_cam25_a08d_200405_18_691_011369850_0_r1625567930_13.zip
9/2/2019 8:08:33 PM | | Resuming GPU computation
9/2/2019 8:08:45 PM | climateprediction.net | Temporarily failed upload of wah2_cam25_a08d_200405_18_691_011369850_0_r1625567930_13.zip: transient HTTP error
9/2/2019 8:08:45 PM | climateprediction.net | Backing off 03:05:48 on upload of wah2_cam25_a08d_200405_18_691_011369850_0_r1625567930_13.zip
9/2/2019 8:08:46 PM | | Project communication failed: attempting access to reference site
9/2/2019 8:08:48 PM | | Internet access OK - project servers may be temporarily down.
9/2/2019 9:27:06 PM | | Suspending computation - CPU is busy
9/2/2019 9:27:16 PM | | Resuming computation
9/2/2019 9:43:27 PM | climateprediction.net | Started upload of wah2_cam25_a08d_200405_18_691_011369850_0_r1625567930_3.zip
9/2/2019 9:43:27 PM | climateprediction.net | Started upload of wah2_cam25_a08d_200405_18_691_011369850_0_r1625567930_15.zip
9/2/2019 9:43:50 PM | | Project communication failed: attempting access to reference site
9/2/2019 9:43:50 PM | climateprediction.net | Temporarily failed upload of wah2_cam25_a08d_200405_18_691_011369850_0_r1625567930_3.zip: transient HTTP error
9/2/2019 9:43:50 PM | climateprediction.net | Backing off 04:28:27 on upload of wah2_cam25_a08d_200405_18_691_011369850_0_r1625567930_3.zip
9/2/2019 9:43:50 PM | climateprediction.net | Temporarily failed upload of wah2_cam25_a08d_200405_18_691_011369850_0_r1625567930_15.zip: transient HTTP error
9/2/2019 9:43:50 PM | climateprediction.net | Backing off 03:32:43 on upload of wah2_cam25_a08d_200405_18_691_011369850_0_r1625567930_15.zip
9/2/2019 9:43:52 PM | | Internet access OK - project servers may be temporarily down.

Mario
ID: 60898 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7183
Credit: 22,883,095
RAC: 1,397
Message 60899 - Posted: 5 Sep 2019, 2:14:16 UTC - in response to Message 60898.  

As has been said, CAMS are a pain.
The only real "cure" is to wait until a zip has uploaded, and then tries another upload from scratch.
At that point, assume that it got through the previous time, and Abort it.
Do the same for each of your 5 files.

At this point, BOINC should wake up to the fact that everything has "uploaded", and finish off the transfer process. Perhaps not immediately.

It's been so long since I had to do this that I forget the timings of what happens.
ID: 60899 · Report as offensive     Reply Quote
Previous · 1 . . . 12 · 13 · 14 · 15 · 16 · 17 · 18 · Next

Message boards : Number crunching : Upload failures

©2020 climateprediction.net