climateprediction.net home page
Can't upload for 12 days

Can't upload for 12 days

Message boards : Number crunching : Can't upload for 12 days
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · Next

AuthorMessage
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7183
Credit: 22,883,095
RAC: 1,397
Message 41717 - Posted: 7 Mar 2011, 5:11:09 UTC - in response to Message 41716.  

Yes, all of this will happen, in due course.
And the rapid, huge accumulation of data will slow down "soon", when the Millennium project ends. Could be 6 months yet, though, as there's still at least one more batch of files to run. Which is also "soon".


Backups: Here
ID: 41717 · Report as offensive     Reply Quote
Profile Milo Thurston
Volunteer moderator
Volunteer developer

Send message
Joined: 2 Mar 06
Posts: 253
Credit: 363,646
RAC: 0
Message 41718 - Posted: 7 Mar 2011, 8:59:42 UTC - in response to Message 41716.  


Also, I for one am hoping that a large chunk of the massive data store you folks have was moved to near line storage so that it doesn't have to be backed up with your back up routines.


No model data are ever backed up - the uploaded data sit on the upload servers (until moved) or archive servers, trusting to RAID 5 or 6, and that's it. There's over 80TB now and there's nowhere to to back that up to.
However, the main project database does need to be backed up, and this is what has been causing problems; the MySQL database was taking up just under 500GB and was very slow to dump. Some of those data (handled messages from host) have been cleaned up and some old results archived so once everything is turned back on it should run much more smoothly, particularly when the new staff install a database slave.

The delay at the moment is that the credit calculation procedures are refusing to add new credit and I presume that that will be the top priority for the CPDN staff today (I expect to be seeing them shortly). Credit is re-calculated daily in its entirety from results/trickles.
I'm not an employee of CPDN.
ID: 41718 · Report as offensive     Reply Quote
old_user30961
Avatar

Send message
Joined: 21 Nov 04
Posts: 6
Credit: 902,269
RAC: 0
Message 41719 - Posted: 7 Mar 2011, 14:07:27 UTC - in response to Message 41715.  

The avatar shows my personal classical CD library of 8,000 CDs.
ID: 41719 · Report as offensive     Reply Quote
BarryAZ

Send message
Joined: 13 Jul 05
Posts: 125
Credit: 11,616,200
RAC: 0
Message 41720 - Posted: 7 Mar 2011, 16:12:56 UTC - in response to Message 41718.  

Milo, thanks for the explanations. These past few months have been a bit troublesome for the Climate project (at least from this users perspective). The frequent fully offline status, along with what seemed to be extensive scheduled daily maintenance/backup runs have been at least somewhat frustrating to work with and observe.

It sounds like (from your message and Les' message) that the light should be at the end of the tunnel -- but that (from what Les posted) it might be a very long tunnel yet.

I sort of wonder if there is some sort of 'size wall' that BOINC projects run up against when the combination of user count and length of the project conspire with the available database and storage technologies to present so large a volume of data that all sorts of difficulties arise. It does seem, that dealing with this difficulties presents something of a (to use an American phrase) 'whackamole' scenario where in addressing one problem another (related or not) pops up.



Also, I for one am hoping that a large chunk of the massive data store you


No model data are ever backed up - the uploaded data sit on the upload servers (until moved) or archive servers, trusting to RAID 5 or 6, and that's it. There's over 80TB now and there's nowhere to to back that up to.
However, the main project database does need to be backed up, and this is what has been causing problems; the MySQL database was taking up just under 500GB and was very slow to dump. Some of those data (handled messages from host) have been cleaned up and some old results archived so once everything is turned back on it should run much more smoothly, particularly when the new staff install a database slave.

The delay at the moment is that the credit calculation procedures are refusing to add new credit and I presume that that will be the top priority for the CPDN staff today (I expect to be seeing them shortly). Credit is re-calculated daily in its entirety from results/trickles.


ID: 41720 · Report as offensive     Reply Quote
Profile Milo Thurston
Volunteer moderator
Volunteer developer

Send message
Joined: 2 Mar 06
Posts: 253
Credit: 363,646
RAC: 0
Message 41721 - Posted: 7 Mar 2011, 16:54:47 UTC - in response to Message 41720.  

Milo, thanks for the explanations. These past few months have been a bit troublesome for the Climate project (at least from this users perspective). The frequent fully offline status, along with what seemed to be extensive scheduled daily maintenance/backup runs have been at least somewhat frustrating to work with and observe.


Sorry about that - I haven't officially worked for them since 1/12/10, and when I have I've had other things to deal with, e.g. the complete change of IP addresses for the department.
By the way, the new temporary sysadmin has finished the database upgrade and is waiting for users to report that their credits are OK before he turns all the daemons back on.

I'm not an employee of CPDN.
ID: 41721 · Report as offensive     Reply Quote
dajashby

Send message
Joined: 1 Sep 04
Posts: 55
Credit: 17,210,126
RAC: 0
Message 41722 - Posted: 7 Mar 2011, 20:20:47 UTC - in response to Message 41721.  

By the way, the new temporary sysadmin has finished the database upgrade and is waiting for users to report that their credits are OK before he turns all the daemons back on.


In that case I should mention that according to BOINCStats I have credit of 7,508,804, wheras Climate Prediction credits me with only 7,477,627. In the last few months BOINStats has usually been a day or two behind CPDN in this regard.
Derrick Ashby
ID: 41722 · Report as offensive     Reply Quote
Profile Milo Thurston
Volunteer moderator
Volunteer developer

Send message
Joined: 2 Mar 06
Posts: 253
Credit: 363,646
RAC: 0
Message 41723 - Posted: 7 Mar 2011, 20:28:11 UTC - in response to Message 41722.  


In that case I should mention that according to BOINCStats I have credit of 7,508,804, wheras Climate Prediction credits me with only 7,477,627. In the last few months BOINStats has usually been a day or two behind CPDN in this regard.


Thanks.
Of the few we've seen so far, most are the same or very close (a few points either way). Very few so far seem to show discrepancies this large.
This probably means everything will have to be switched on and discrepancies pursued later.
I'm not an employee of CPDN.
ID: 41723 · Report as offensive     Reply Quote
BarryAZ

Send message
Joined: 13 Jul 05
Posts: 125
Credit: 11,616,200
RAC: 0
Message 41724 - Posted: 7 Mar 2011, 20:50:12 UTC - in response to Message 41723.  

For what it's worth, when I check back toward the end of February - I had 9,281,477. The current status reports me at 9,273,965. So I've 'lost' about 7,500 credits (perhaps more as there may have been a few trickles before shutdown.



Thanks.
Of the few we've seen so far, most are the same or very close (a few points either way). Very few so far seem to show discrepancies this large.
This probably means everything will have to be switched on and discrepancies pursued later.


ID: 41724 · Report as offensive     Reply Quote
BarryAZ

Send message
Joined: 13 Jul 05
Posts: 125
Credit: 11,616,200
RAC: 0
Message 41725 - Posted: 7 Mar 2011, 20:52:25 UTC - in response to Message 41721.  

Right, I saw that information posted back in December, but as you did post here, you ran into the 'no good deed goes unpunished' loop. <smile>.




Sorry about that - I haven't officially worked for them since 1/12/10, and when I have I've had other things to deal with, e.g. the complete change of IP addresses for the department.


ID: 41725 · Report as offensive     Reply Quote
w1hue

Send message
Joined: 31 Aug 05
Posts: 18
Credit: 1,923,299
RAC: 81
Message 41726 - Posted: 7 Mar 2011, 23:39:02 UTC - in response to Message 41721.  


By the way, the new temporary sysadmin has finished the database upgrade and is waiting for users to report that their credits are OK before he turns all the daemons back on.


For what it's worth, according to my local BOINC manager my credit is 116237, and according to BOINCstats it is 115685.


ID: 41726 · Report as offensive     Reply Quote
3rkko

Send message
Joined: 12 Feb 08
Posts: 66
Credit: 4,877,652
RAC: 0
Message 41727 - Posted: 8 Mar 2011, 3:29:00 UTC

If it's of any help, BoincStats shows 1,503,116 credits and CPDN 1,510,255 credits.

ID: 41727 · Report as offensive     Reply Quote
Profile Byron Leigh Hatch @ team Carl Sagan
Avatar

Send message
Joined: 17 Aug 04
Posts: 289
Credit: 44,078,566
RAC: 2,012
Message 41728 - Posted: 8 Mar 2011, 4:11:09 UTC

according to BOINCStats I have credit of 350,574 ... Climate Prediction credits me with 351,970

ID: 41728 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7183
Credit: 22,883,095
RAC: 1,397
Message 41729 - Posted: 8 Mar 2011, 5:05:05 UTC

The reason that none of the external stats sites will be correct, is that the various daemons haven't been turned on yet.
So the daily credits export files haven't been run.

This will happen soon.

Watch the News and Announcements thread.


Backups: Here
ID: 41729 · Report as offensive     Reply Quote
Profile Byron Leigh Hatch @ team Carl Sagan
Avatar

Send message
Joined: 17 Aug 04
Posts: 289
Credit: 44,078,566
RAC: 2,012
Message 41730 - Posted: 8 Mar 2011, 5:13:00 UTC - in response to Message 41729.  

Thank you Les for that information.
ID: 41730 · Report as offensive     Reply Quote
BarryAZ

Send message
Joined: 13 Jul 05
Posts: 125
Credit: 11,616,200
RAC: 0
Message 41731 - Posted: 8 Mar 2011, 6:19:02 UTC - in response to Message 41729.  

Les, the numbers I was referring to were the not the stats site numbers. I was comparing numbers on the Climate user account. The user account numbers for me from late February are about 7,500 HIGHER than the user account numbers today.
ID: 41731 · Report as offensive     Reply Quote
old_user9248

Send message
Joined: 2 Sep 04
Posts: 3
Credit: 2,345,830
RAC: 0
Message 41732 - Posted: 8 Mar 2011, 12:03:19 UTC

I also have several results pending for upload and I just noticed on my Statistics tab that one my last update I actually lost work credits! (Graph goes down) Is that correct?
<img src="http://www.boincstats.com/stats/banner.php?cpid=f2d7bbf09c52089e143aa67c68549211">
ID: 41732 · Report as offensive     Reply Quote
Profile Thyme Lawn
Volunteer moderator

Send message
Joined: 5 Aug 04
Posts: 1278
Credit: 15,779,438
RAC: 7
Message 41735 - Posted: 8 Mar 2011, 13:00:36 UTC - in response to Message 41732.  

I also have several results pending for upload and I just noticed on my Statistics tab that one my last update I actually lost work credits! (Graph goes down) Is that correct?

Yes, but it shouldn't be the case. The lost credits will be restored after the project team finish their investigations.

See the news post here.
"The ultimate test of a moral society is the kind of world that it leaves to its children." - Dietrich Bonhoeffer
ID: 41735 · Report as offensive     Reply Quote
dajashby

Send message
Joined: 1 Sep 04
Posts: 55
Credit: 17,210,126
RAC: 0
Message 41745 - Posted: 8 Mar 2011, 20:34:07 UTC - in response to Message 41723.  


In that case I should mention that according to BOINCStats I have credit of 7,508,804, wheras Climate Prediction credits me with only 7,477,627. In the last few months BOINStats has usually been a day or two behind CPDN in this regard.


Thanks.
Of the few we've seen so far, most are the same or very close (a few points either way). Very few so far seem to show discrepancies this large.
This probably means everything will have to be switched on and discrepancies pursued later.


This morning those 31 thousand and something points came off my BoincStats total...
Derrick Ashby
ID: 41745 · Report as offensive     Reply Quote
old_user9248

Send message
Joined: 2 Sep 04
Posts: 3
Credit: 2,345,830
RAC: 0
Message 41747 - Posted: 9 Mar 2011, 2:56:00 UTC - in response to Message 41732.  

The issue seems to have corrected itself over the course of the day. :)
<img src="http://www.boincstats.com/stats/banner.php?cpid=f2d7bbf09c52089e143aa67c68549211">
ID: 41747 · Report as offensive     Reply Quote
dajashby

Send message
Joined: 1 Sep 04
Posts: 55
Credit: 17,210,126
RAC: 0
Message 41752 - Posted: 9 Mar 2011, 20:16:56 UTC - in response to Message 41747.  
Last modified: 9 Mar 2011, 20:20:45 UTC

Sorry, misread your message - my points are still missing - though I read that the problem has been identified.
Derrick Ashby
ID: 41752 · Report as offensive     Reply Quote
Previous · 1 · 2 · 3 · 4 · Next

Message boards : Number crunching : Can't upload for 12 days

©2020 climateprediction.net