climateprediction.net home page
Server Status

Server Status

Message boards : Number crunching : Server Status
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4314
Credit: 16,376,018
RAC: 3,616
Message 48032 - Posted: 23 Jan 2014, 9:35:21 UTC

Just looked at the server status and it seems to be stuck with 44,697 tasks in progress and 2 ready to send which is the same as it was over 12 hours ago.
ID: 48032 · Report as offensive     Reply Quote
Profile mo.v
Volunteer moderator
Avatar

Send message
Joined: 29 Sep 04
Posts: 2363
Credit: 14,611,758
RAC: 0
Message 48033 - Posted: 23 Jan 2014, 9:47:34 UTC

Hello Dave

I noticed last night that it's stuck at 13:59:00 and cannot be updated. After this had been confirmed by another member I emailed the programmers. It means there's an updating script that isn't running; normally it should update every minute.

The script that should export our credits to the external stats sites still isn't working either.
Cpdn news
ID: 48033 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4314
Credit: 16,376,018
RAC: 3,616
Message 48034 - Posted: 23 Jan 2014, 10:02:48 UTC - in response to Message 48033.  

I did notice it seemed stuck yesterday, I should have known someone would have been on to it without my posting anything!
ID: 48034 · Report as offensive     Reply Quote
Profile JIM

Send message
Joined: 31 Dec 07
Posts: 1152
Credit: 22,053,321
RAC: 4,417
Message 48036 - Posted: 24 Jan 2014, 3:07:24 UTC

I have a trickle stuck that I can�t upload. Could this be related to the problem with the server status page?

ID: 48036 · Report as offensive     Reply Quote
Profile JIM

Send message
Joined: 31 Dec 07
Posts: 1152
Credit: 22,053,321
RAC: 4,417
Message 48037 - Posted: 24 Jan 2014, 4:55:09 UTC

I see now why I can�t trickle. Messages says that the project is down for maintenance.

ID: 48037 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 48038 - Posted: 24 Jan 2014, 5:53:20 UTC - in response to Message 48037.  

Andy said while the forum was still down, that the background processes had been stopped to make a backup.

So, expect turbulence.


Backups: Here
ID: 48038 · Report as offensive     Reply Quote
Profile mo.v
Volunteer moderator
Avatar

Send message
Joined: 29 Sep 04
Posts: 2363
Credit: 14,611,758
RAC: 0
Message 48039 - Posted: 24 Jan 2014, 10:13:13 UTC

Dave said:

'I did notice it seemed stuck yesterday, I should have known someone would have been on to it without my posting anything!'

That's not a reason not to post! You may well be the first or the only person to notice something. If problems are posted on the forum it often reassures other members who worried or puzzled about what's going on.

Cpdn news
ID: 48039 · Report as offensive     Reply Quote
Profile mo.v
Volunteer moderator
Avatar

Send message
Joined: 29 Sep 04
Posts: 2363
Credit: 14,611,758
RAC: 0
Message 48041 - Posted: 24 Jan 2014, 11:56:16 UTC

The SS page is now updating.
Cpdn news
ID: 48041 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4314
Credit: 16,376,018
RAC: 3,616
Message 48042 - Posted: 24 Jan 2014, 20:11:07 UTC - in response to Message 48041.  

Thanks Mo, I would still post whatever for the reasons you state. If it hadn't been for a phone call I probably would have posted when I first noticed it was stuck.
ID: 48042 · Report as offensive     Reply Quote
Profile Bonsai911

Send message
Joined: 9 Sep 04
Posts: 228
Credit: 30,229,255
RAC: 3,258
Message 48069 - Posted: 30 Jan 2014, 14:42:25 UTC

RUNNING again!

Keep on the good work,

Bonsai911


ID: 48069 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4314
Credit: 16,376,018
RAC: 3,616
Message 48178 - Posted: 17 Feb 2014, 12:00:19 UTC

Server status seems to be stuck again, this time at 18.09 16th Feb.
ID: 48178 · Report as offensive     Reply Quote
metalius
Avatar

Send message
Joined: 28 Nov 06
Posts: 89
Credit: 11,328,674
RAC: 2,783
Message 48180 - Posted: 17 Feb 2014, 21:01:13 UTC

It looks like, today the Trickle server is not accepting trickles.
ID: 48180 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 48182 - Posted: 17 Feb 2014, 22:33:06 UTC

The outage was a VM failure.
Some programs may not have been restarted yet.
I'll email them.

ID: 48182 · Report as offensive     Reply Quote
Nuadormrac
Avatar

Send message
Joined: 14 Oct 05
Posts: 44
Credit: 2,814,260
RAC: 5,635
Message 48185 - Posted: 17 Feb 2014, 23:37:41 UTC
Last modified: 17 Feb 2014, 23:38:23 UTC

I had a WU finish while the project showed up as offline by BOINCstats, when BOINC started trying to upload the WUs. Ever since then, it hasn't been able to upload, giving:

2/17/2014 6:23:20 PM | climateprediction.net | Started upload of hadam3p_eu_8824_2001_1_007660469_2_12.zip
2/17/2014 6:23:28 PM | climateprediction.net | Temporarily failed upload of hadam3p_eu_8824_2001_1_007660469_2_12.zip: connect() failed
2/17/2014 6:23:28 PM | climateprediction.net | Backing off 3 hr 13 min 29 sec on upload of hadam3p_eu_8824_2001_1_007660469_2_12.zip


and

2/17/2014 6:31:34 PM | climateprediction.net | Started upload of hadam3p_eu_8824_2001_1_007660469_2_13.zip
2/17/2014 6:35:53 PM | climateprediction.net | [error] Error reported by file upload server: can't open file /storage/cpdn-restarts/incoming/uploader/hadam3p_eu_8824_2001_1_007660469_2_13.zip: No such file or directory
2/17/2014 6:35:53 PM | climateprediction.net | Temporarily failed upload of hadam3p_eu_8824_2001_1_007660469_2_13.zip: transient upload error
2/17/2014 6:35:53 PM | climateprediction.net | Backing off 4 hr 38 min 28 sec on upload of hadam3p_eu_8824_2001_1_007660469_2_13.zip


It seems that after whatever brought the project servers down yesterday, they don't want me to upload the completed task here...

Server status is showing all as running, but if it's stuck, that might mean nothing...
ID: 48185 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 48187 - Posted: 18 Feb 2014, 0:18:58 UTC

"Server Status" shows whether or not a server is running.
It doesn't indicate anything about the programs that are supposed to be running on the server(s).


Backups: Here
ID: 48187 · Report as offensive     Reply Quote
Profile geophi
Volunteer moderator

Send message
Joined: 7 Aug 04
Posts: 2167
Credit: 64,403,322
RAC: 5,085
Message 48189 - Posted: 18 Feb 2014, 2:59:50 UTC

I'm getting similar errors to Nuadormrac

Mon 17 Feb 2014 07:25:53 PM CST climateprediction.net [error] Error reported by file upload server: can't open file /storage/incoming/uploader_main/hadam3p_pnw_ubeu_2003_1_008507842_1_10.zip: Read-only file system
Mon 17 Feb 2014 07:25:53 PM CST climateprediction.net Temporarily failed upload of hadam3p_pnw_ubeu_2003_1_008507842_1_10.zip: transient upload error
Mon 17 Feb 2014 07:25:53 PM CST climateprediction.net Backing off 3 hr 49 min 39 sec on upload of hadam3p_pnw_ubeu_2003_1_008507842_1_10.zip
Mon 17 Feb 2014 08:48:01 PM CST climateprediction.net Started upload of hadam3p_pnw_ubeu_2003_1_008507842_1_11.zip
Mon 17 Feb 2014 08:48:18 PM CST climateprediction.net [error] Error reported by file upload server: can't open file /storage/incoming/uploader_main/hadam3p_pnw_ubeu_2003_1_008507842_1_11.zip: Read-only file system
Mon 17 Feb 2014 08:48:18 PM CST climateprediction.net Temporarily failed upload of hadam3p_pnw_ubeu_2003_1_008507842_1_11.zip: transient upload error
Mon 17 Feb 2014 08:48:18 PM CST climateprediction.net Backing off 1 min 0 sec on upload of hadam3p_pnw_ubeu_2003_1_008507842_1_11.zip

ID: 48189 · Report as offensive     Reply Quote
Profile Alan K

Send message
Joined: 22 Feb 06
Posts: 484
Credit: 29,579,234
RAC: 4,572
Message 48192 - Posted: 18 Feb 2014, 9:52:26 UTC - in response to Message 48189.  

I've got 4 models that haven't had any trickles accepted since 16th. Also zip upload problems:-
18/02/2014 02:39:04 | climateprediction.net | Started upload of hadcm3n_7x8g_1980_40_008454355_3_1.zip
18/02/2014 02:39:11 | climateprediction.net | Finished upload of hadcm3n_7x8g_1980_40_008454355_3_1.zip
18/02/2014 02:39:11 | climateprediction.net | Started upload of hadam3p_eu_n01q_2012_1_008504618_1_11.zip
18/02/2014 02:39:11 | climateprediction.net | Started upload of hadam3p_eu_n01q_2012_1_008504618_1_12.zip
18/02/2014 02:39:13 | climateprediction.net | Temporarily failed upload of hadam3p_eu_n01q_2012_1_008504618_1_11.zip: connect() failed
18/02/2014 02:39:13 | climateprediction.net | Backing off 03:23:57 on upload of hadam3p_eu_n01q_2012_1_008504618_1_11.zip
18/02/2014 02:39:13 | climateprediction.net | Temporarily failed upload of hadam3p_eu_n01q_2012_1_008504618_1_12.zip: connect() failed
18/02/2014 02:39:13 | climateprediction.net | Backing off 00:04:43 on upload of hadam3p_eu_n01q_2012_1_008504618_1_12.zip
18/02/2014 02:39:13 | climateprediction.net | Started upload of hadam3p_eu_n029_2012_1_008504637_1_1.zip
18/02/2014 02:39:14 | climateprediction.net | Started upload of hadam3p_eu_n01q_2012_1_008504618_1_13.zip
18/02/2014 02:39:15 | climateprediction.net | Temporarily failed upload of hadam3p_eu_n029_2012_1_008504637_1_1.zip: connect() failed
18/02/2014 02:39:15 | climateprediction.net | Backing off 00:06:04 on upload of hadam3p_eu_n029_2012_1_008504637_1_1.zip
18/02/2014 02:39:25 | climateprediction.net | [error] Error reported by file upload server: can't open file /storage/cpdn-restarts/incoming/uploader/hadam3p_eu_n01q_2012_1_008504618_1_13.zip: No such file or directory
18/02/2014 02:39:25 | climateprediction.net | Temporarily failed upload of hadam3p_eu_n01q_2012_1_008504618_1_13.zip: transient upload error

Do these give any clues?
ID: 48192 · Report as offensive     Reply Quote
Profile astroWX
Volunteer moderator

Send message
Joined: 5 Aug 04
Posts: 1496
Credit: 95,522,203
RAC: 0
Message 48199 - Posted: 18 Feb 2014, 18:31:52 UTC - in response to Message 48192.  

I had the same issues. However, the '_13' file (a summary restart dump) goes to Oxford and mine finally cleared overnight (Pacific time zone, GMT-8). The first twelve go to Oregon State U. and are still broken. It's 1024 Pacific time, so the techs have only been on the job a couple hours and might be slow starting this morning, clearing a three-day weekend accumulation of issues. (yesterday was a holiday in the US).
"We have met the enemy and he is us." -- Pogo
Greetings from coastal Washington state, the scenic US Pacific Northwest.
ID: 48199 · Report as offensive     Reply Quote
Profile astroWX
Volunteer moderator

Send message
Joined: 5 Aug 04
Posts: 1496
Credit: 95,522,203
RAC: 0
Message 48201 - Posted: 18 Feb 2014, 19:42:25 UTC

My backlog is clearing to Oregon now.

Given the large number of tasks hammering the server in the post-restart period, we should not be surprised if uploads are slower than usual.
"We have met the enemy and he is us." -- Pogo
Greetings from coastal Washington state, the scenic US Pacific Northwest.
ID: 48201 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4314
Credit: 16,376,018
RAC: 3,616
Message 48202 - Posted: 19 Feb 2014, 8:27:23 UTC

Had a unit failed to download yesterday - don't know if that was part of the now fixed problem. I see that nothing else in the workunit has finished.

Tue 18 Feb 2014 12:10:49 GMT | climateprediction.net | Scheduler request completed: got 1 new tasks
Tue 18 Feb 2014 12:10:51 GMT | climateprediction.net | Started download of hadcm3n_ofq9_1900_40_008475508.zip
Tue 18 Feb 2014 12:10:51 GMT | climateprediction.net | Started download of ozone_hadcm3_1859.be.32.gz
Tue 18 Feb 2014 12:10:53 GMT | climateprediction.net | Giving up on download of hadcm3n_ofq9_1900_40_008475508.zip: permanent HTTP error
Tue 18 Feb 2014 12:10:53 GMT | climateprediction.net | Giving up on download of ozone_hadcm3_1859.be.32.gz: permanent HTTP error
Tue 18 Feb 2014 12:10:53 GMT | climateprediction.net | Started download of xabnu.astart.gz
Tue 18 Feb 2014 12:10:53 GMT | climateprediction.net | Started download of DMSallNH3SO21859.be.32.gz
Tue 18 Feb 2014 12:10:54 GMT | climateprediction.net | Giving up on download of xabnu.astart.gz: permanent HTTP error
Tue 18 Feb 2014 12:10:54 GMT | climateprediction.net | Giving up on download of DMSallNH3SO21859.be.32.gz: permanent HTTP error
Tue 18 Feb 2014 12:10:54 GMT | climateprediction.net | Started download of sulpc_oxidants_19_A2_1990f.gz
Tue 18 Feb 2014 12:10:54 GMT | climateprediction.net | Started download of xabnu.ostart.gz
Tue 18 Feb 2014 12:10:55 GMT | climateprediction.net | Giving up on download of sulpc_oxidants_19_A2_1990f.gz: permanent HTTP error
Tue 18 Feb 2014 12:10:55 GMT | climateprediction.net | Giving up on download of xabnu.ostart.gz: permanent HTTP error
ID: 48202 · Report as offensive     Reply Quote
1 · 2 · Next

Message boards : Number crunching : Server Status

©2024 climateprediction.net