climateprediction.net home page
Download errors on UK Met Office HadAM4 at N216 resolution v8.52 tasks

Download errors on UK Met Office HadAM4 at N216 resolution v8.52 tasks

Message boards : Number crunching : Download errors on UK Met Office HadAM4 at N216 resolution v8.52 tasks
Message board moderation

To post messages, you must log in.

Previous · 1 · 2

AuthorMessage
Richard Haselgrove

Send message
Joined: 1 Jan 07
Posts: 943
Credit: 34,181,338
RAC: 6,507
Message 64444 - Posted: 9 Sep 2021, 18:19:19 UTC

Just downloaded four new tasks - 2xN144 and 2xN216. All downloads completed successfully.
ID: 64444 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4342
Credit: 16,499,590
RAC: 5,672
Message 64445 - Posted: 9 Sep 2021, 18:54:44 UTC - in response to Message 64444.  

Just downloaded four new tasks - 2xN144 and 2xN216. All downloads completed successfully.

Good news, I had a completed task upload before the website said the servers were back up and my couple of glances at the tasks running and ready to send were not enough to confirm that new tasks were being downloaded.
ID: 64445 · Report as offensive     Reply Quote
Richard Haselgrove

Send message
Joined: 1 Jan 07
Posts: 943
Credit: 34,181,338
RAC: 6,507
Message 64671 - Posted: 21 Oct 2021, 8:10:46 UTC

I'm back again.

21/10/2021 09:04:24 | climateprediction.net | Started download of hadam4h_h1jm_201302_4_920_012117220.zip
21/10/2021 09:04:25 | climateprediction.net | [http] [ID#24419] Info: Hostname download.cpdn.org was found in DNS cache
21/10/2021 09:04:25 | climateprediction.net | [http] [ID#24419] Info: Trying 129.67.193.131:80...
21/10/2021 09:04:25 | climateprediction.net | [http] [ID#24419] Info: connect to 129.67.193.131 port 80 failed: Connection refused
21/10/2021 09:04:25 | climateprediction.net | [http] HTTP error: Couldn't connect to server
21/10/2021 09:04:25 | climateprediction.net | Temporarily failed download of hadam4h_h1jm_201302_4_920_012117220.zip: connect() failed
Four more tasks, issued 21 Oct 2021, 4:56:37 UTC to host 1498009, won't be coming home.
ID: 64671 · Report as offensive     Reply Quote
bozz4science

Send message
Joined: 10 May 20
Posts: 50
Credit: 3,356,491
RAC: 437
Message 64672 - Posted: 21 Oct 2021, 8:44:54 UTC

Same stuff repeating on my end on a Linux VM (Host 1519938). Tasks are stuck in the "Download: retry in xx:xx" loop. According to the event log some files started to download okay, but got suddenly stuck with the message "Temporarily failed download of …" & "Backing off xx:xx on download of …".
ID: 64672 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4342
Credit: 16,499,590
RAC: 5,672
Message 64673 - Posted: 21 Oct 2021, 10:10:40 UTC

And I am getting,

Thu 21 Oct 2021 11:08:04 BST | climateprediction.net | Project requested delay of 3636 seconds
Thu 21 Oct 2021 11:08:09 BST | cpdnboinc_dev | Sending scheduler request: To fetch work.
Thu 21 Oct 2021 11:08:09 BST | cpdnboinc_dev | Requesting new tasks for CPU
Thu 21 Oct 2021 11:08:10 BST |  | Project communication failed: attempting access to reference site
Thu 21 Oct 2021 11:08:10 BST | cpdnboinc_dev | Scheduler request failed: Couldn't connect to server
Thu 21 Oct 2021 11:08:11 BST |  | Internet access OK - project servers may be temporarily down.
E

Email sent.
ID: 64673 · Report as offensive     Reply Quote
Jean-David Beyer

Send message
Joined: 5 Aug 04
Posts: 1056
Credit: 16,520,115
RAC: 1,176
Message 64675 - Posted: 21 Oct 2021, 12:07:35 UTC - in response to Message 64673.  

It seems to be working for me. I got four tasks last evening, one at a time, an hour apart. This is the last of them:
Wed 20 Oct 2021 09:04:04 PM EDT | climateprediction.net | Sending scheduler request: To fetch work.
Wed 20 Oct 2021 09:04:04 PM EDT | climateprediction.net | Requesting new tasks for CPU
Wed 20 Oct 2021 09:04:06 PM EDT | climateprediction.net | Scheduler request completed: got 1 new tasks
Wed 20 Oct 2021 09:04:06 PM EDT | climateprediction.net | Project requested delay of 3636 seconds
Wed 20 Oct 2021 09:04:08 PM EDT | climateprediction.net | Started download of hadam4h_h0c7_200602_4_920_012115657.zip
Wed 20 Oct 2021 09:04:08 PM EDT | climateprediction.net | Started download of h0c7_920_atmos.gz
Wed 20 Oct 2021 09:04:11 PM EDT | climateprediction.net | Finished download of hadam4h_h0c7_200602_4_920_012115657.zip
Wed 20 Oct 2021 09:04:11 PM EDT | climateprediction.net | Started download of ic_N216_2003_03_000052_f.nc.gz
Wed 20 Oct 2021 09:04:18 PM EDT | climateprediction.net | Finished download of ic_N216_2003_03_000052_f.nc.gz
Wed 20 Oct 2021 09:04:39 PM EDT | climateprediction.net | Finished download of h0c7_920_atmos.gz
Wed 20 Oct 2021 09:04:40 PM EDT | climateprediction.net | Starting task hadam4h_h0c7_200602_4_920_012115657_0

So they must have fixed it between our two observations.
ID: 64675 · Report as offensive     Reply Quote
Richard Haselgrove

Send message
Joined: 1 Jan 07
Posts: 943
Credit: 34,181,338
RAC: 6,507
Message 64676 - Posted: 21 Oct 2021, 13:04:47 UTC - in response to Message 64675.  

So they must have fixed it between our two observations.
Other way round. Your times are before ours. So they must have broken it between our two observations, or some unconnected external event must have broken it.
ID: 64676 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4342
Credit: 16,499,590
RAC: 5,672
Message 64677 - Posted: 21 Oct 2021, 14:19:57 UTC

Other way round. Your times are before ours.

And still broken as of 50 minutes ago. Backoff will let me check again in 10.
ID: 64677 · Report as offensive     Reply Quote
Jean-David Beyer

Send message
Joined: 5 Aug 04
Posts: 1056
Credit: 16,520,115
RAC: 1,176
Message 64678 - Posted: 21 Oct 2021, 16:08:32 UTC - in response to Message 64676.  

Other way round. Your times are before ours.


My mental arithmetic is not what it used to be.
ID: 64678 · Report as offensive     Reply Quote
leloft

Send message
Joined: 7 Jun 17
Posts: 23
Credit: 44,434,789
RAC: 2,600,991
Message 64680 - Posted: 21 Oct 2021, 20:30:00 UTC - in response to Message 64672.  

Same stuff repeating on my end on a Linux VM (Host 1519938). Tasks are stuck in the "Download: retry in xx:xx" loop. According to the event log some files started to download okay, but got suddenly stuck with the message "Temporarily failed download of …" & "Backing off xx:xx on download of …".

Me too. Same issue on two hosts (ID: 1522999; ID: 1523002), although other hosts have received work units after the first one reported problems. Do I need to do anything, or does this get resolved server-side?
Many thanks
ID: 64680 · Report as offensive     Reply Quote
bozz4science

Send message
Joined: 10 May 20
Posts: 50
Credit: 3,356,491
RAC: 437
Message 64681 - Posted: 21 Oct 2021, 20:32:48 UTC

Just retried now. Downloads finally work smoothly again at least on my host.
ID: 64681 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4342
Credit: 16,499,590
RAC: 5,672
Message 64682 - Posted: 21 Oct 2021, 20:58:43 UTC - in response to Message 64681.  

Just retried now. Downloads finally work smoothly again at least on my host.


I got project has no tasks available but there are another two batches on the way shortly.
ID: 64682 · Report as offensive     Reply Quote
Richard Haselgrove

Send message
Joined: 1 Jan 07
Posts: 943
Credit: 34,181,338
RAC: 6,507
Message 64683 - Posted: 21 Oct 2021, 20:59:46 UTC - in response to Message 64681.  

So they do. Forced a work fetch, and got four new tasks - all downloaded properly.

They might have told us.
ID: 64683 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4342
Credit: 16,499,590
RAC: 5,672
Message 64684 - Posted: 22 Oct 2021, 10:01:20 UTC - in response to Message 64683.  

So they do. Forced a work fetch, and got four new tasks - all downloaded properly.

They might have told us.


Strange, before heading to work this morning, I tried again and got the no work available message again but while out another task has downloaded so all seems to be working again.
ID: 64684 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 64685 - Posted: 22 Oct 2021, 12:24:17 UTC

They may have re-issued the ones that failed downloading.
ID: 64685 · Report as offensive     Reply Quote
Richard Haselgrove

Send message
Joined: 1 Jan 07
Posts: 943
Credit: 34,181,338
RAC: 6,507
Message 64686 - Posted: 22 Oct 2021, 12:56:28 UTC - in response to Message 64685.  

Looking at the four I got last night, they were all created on 20 October, but my download late on 21 October was the first time they'd been issued. So they probably blithely went on deploying work, unaware that it couldn't be downloaded, Left hand, right hand.

host 1498009
ID: 64686 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 64687 - Posted: 22 Oct 2021, 13:12:37 UTC

They were emailed about it, but there's been no reply, so no idea what's going on there. :(
ID: 64687 · Report as offensive     Reply Quote
Previous · 1 · 2

Message boards : Number crunching : Download errors on UK Met Office HadAM4 at N216 resolution v8.52 tasks

©2024 climateprediction.net