climateprediction.net home page
Database Sluggish -- Moved Trickle Info \'Down\'

Database Sluggish -- Moved Trickle Info \'Down\'

Message boards : Number crunching : Database Sluggish -- Moved Trickle Info \'Down\'
Message board moderation

To post messages, you must log in.

AuthorMessage
old_user1
Avatar

Send message
Joined: 5 Aug 04
Posts: 907
Credit: 299,864
RAC: 0
Message 18489 - Posted: 20 Dec 2005, 19:58:47 UTC

Hi, it looks like with our new user numbers, it\'s not efficient to keep reporting trickle details on the user & host pages. You can still see the trickles for a result \"drill-down.\"
ID: 18489 · Report as offensive     Reply Quote
Profile geophi
Volunteer moderator

Send message
Joined: 7 Aug 04
Posts: 1908
Credit: 40,178,185
RAC: 9,912
Message 18491 - Posted: 20 Dec 2005, 20:20:44 UTC

So, in the past some trickles were listed on the User, Computer Summary, Result, and Trickles Information for Result. This change makes them just listed on the Trickles Information for Result?

I\'m thinking that lots of people used the trickles under the User page to kind of do a quick check on how their computers were doing, any problems, whether they were home or not. Now one either needs to do

Click on Your Account
Click on View Results
Click on the Result ID (unsure of associated host at that time)
Click on Trickle #

or, to see how a particular PC is doing:

Click on Your Account
Click on View Computers
Click on Computer ID
Click on Results
Click on Result ID
Click on Trickle #

I think the trickles on the User page would be pretty handy to have. Perhaps having the trickles removed from just the Computer Summary and Result pages will improve performance enough without the removal from the User Page?
ID: 18491 · Report as offensive     Reply Quote
old_user1
Avatar

Send message
Joined: 5 Aug 04
Posts: 907
Credit: 299,864
RAC: 0
Message 18492 - Posted: 20 Dec 2005, 20:46:25 UTC - in response to Message 18491.  

well actually the user page was the worst one, although I just show the latest 10 or 20 to the database it still has to get maybe 10K. perhaps they\'ll come back someday when I get a chance to optimize the tables etc, but for now they were taking 1-2 minutes per user request!
ID: 18492 · Report as offensive     Reply Quote
Profile geophi
Volunteer moderator

Send message
Joined: 7 Aug 04
Posts: 1908
Credit: 40,178,185
RAC: 9,912
Message 18493 - Posted: 20 Dec 2005, 20:48:39 UTC

I\'ll make this a sticky then since people will start noticing soon.
ID: 18493 · Report as offensive     Reply Quote
Profile [B@H] Ray
Avatar

Send message
Joined: 19 Aug 05
Posts: 104
Credit: 1,866,495
RAC: 0
Message 18496 - Posted: 20 Dec 2005, 21:15:10 UTC

Already noticed, thought it was server problems at first. I miss them there but understand the server load. Can get at them easy the way that they are. When the database is optomized even having the newest 5 rather than 20 would be handy, as that would let us know which computers reported in without having to chack all units.
Feels good to have a Sulpher model almost finished. Will be done in plunty of time for use in the next round of couppled models.

Cheers
Ray
Keep on crunching Pizza@Home
ID: 18496 · Report as offensive     Reply Quote
old_user56785
Avatar

Send message
Joined: 23 Feb 05
Posts: 55
Credit: 240,119
RAC: 0
Message 18502 - Posted: 20 Dec 2005, 23:09:57 UTC

Thanks for explaining Carl. Indeed saving a user page has been taking much longer than saving a result or trickle page!

Nevertheless a loss of overview for the users, but it might be more direct to access the trickle page directly.

In Internet Explorer the user can add and name the different trickle pages for models that have to be checked on trickle update.
(Other browser would have possiblities alike)

It takes a bit of reorganising but it might be more direct and with less paging around, once you are used to it.
ID: 18502 · Report as offensive     Reply Quote
Profile old_user85254

Send message
Joined: 27 Jun 05
Posts: 74
Credit: 199,198
RAC: 0
Message 18543 - Posted: 21 Dec 2005, 11:20:04 UTC
Last modified: 21 Dec 2005, 11:42:28 UTC

Hi Carl,

Thanks for the explanation

Obviously that kind of load is unacceptable and you had to do something. It does not make sense to include expensive information on a page which may be loaded for other reasons.

However with multiple computers, not all in the same place, it was very useful to have all recent trickles on the same page.

Three useful alternatives would be

1) Have a separate \'recent trickles\' page that users could click on if they wanted it. The advantage to the project is that the heavy load is only produced when the user specifically wanted it, no for example when the user only went to the page to see their latest stats or whatever.

2) Introduce a \'last successful trickle\' date column to the page that lists computers (not quite the same as the existing \'last contact\' which always might be for some kind of err, manual update, even machine reboot sometimes)

3) Introduce a \'last successful trickle\' column on the page that lists a user\'s results.

The advantage of any of these, from the viewpoint of a user with many machines, is that one page will give a snap overview of whether any of the machines / results needs some attention.

Perhaps with option 1 you could extract a derived table once an hour with trickle info and userid in it, with an index on userid. The query would be an hour old but that is fine for a quick-look facility.

With option 1, you could also make the script run at a lower priority than the rest of the web pages. This would mean two things, firstly, users asking that question would not slug the rest of the website, and secondly if those pages got overloaded the slowness of their response would deter their use for frivolous queries. The page would need to contain an explanation that \"at busy times this page may be slow\", and \"please do not repeatedly reload - this will only make it worse\".

(One way I\'\'ve done this is for the page to call a separate script via a \'nice\' command -- it got round a similar problem we had of one particular page slugging a whole website)

Any one of these options, or any other way of doing a quick \"is everyone happy\" check one to three times a day would be a welcome replacement. With 8 active machines I won\'t have the time to check on every machine every day without some way of doing it from a single bookmarked page.

And to be clear, this is not a complaint, it\'s a request.

You clearly did the right thing in the short term to de-slug the db. It is much better to have most of the old functionality running at a workable speed that have it all running as slow as it has been lately. The performance boost since your change is both obvious and welcome.

ID: 18543 · Report as offensive     Reply Quote
Profile old_user85254

Send message
Joined: 27 Jun 05
Posts: 74
Credit: 199,198
RAC: 0
Message 18544 - Posted: 21 Dec 2005, 11:32:35 UTC - in response to Message 18502.  
Last modified: 21 Dec 2005, 11:43:36 UTC

In Internet Explorer the user can add and name the different trickle pages for models that have to be checked on trickle update.
(Other browser would have possiblities alike)


hi again Kilcock

yes, in IE you set the trickle pages as \'favorites\'; in Firefox you bookmark them - same thing really.

It still means one bookmark/favorite for each model - with 8 boxes and 11 active model that still leaves 11 bookmarks to check.

An even better way in the firefox browser is to open each of these pages in separate tabs, then use the firefox facility to save the entire set of tabs as a single bookmark. This means that one click gets all the pages, and you simply scan from one tab to the next to look at them all.

Firefox is free and open source and available for all the platforms running CPDN, more info & downloads from www.firefox.com

It is still 12 clicks to see all the info, so I\'d still prefer some way of getting it all onto one page. Also it means that when I click on the one bookmark it sends 11 queries to the database all together - a well designed single page might be less of a load on the database

River~~

edits: speeling & typos ;-)
ID: 18544 · Report as offensive     Reply Quote
Martin P.

Send message
Joined: 5 Feb 05
Posts: 28
Credit: 1,846,059
RAC: 0
Message 18562 - Posted: 21 Dec 2005, 17:18:36 UTC - in response to Message 18544.  
Last modified: 21 Dec 2005, 17:18:58 UTC

In Internet Explorer the user can add and name the different trickle pages for models that have to be checked on trickle update.
(Other browser would have possiblities alike)


hi again Kilcock

yes, in IE you set the trickle pages as \'favorites\'; in Firefox you bookmark them - same thing really.

It still means one bookmark/favorite for each model - with 8 boxes and 11 active model that still leaves 11 bookmarks to check.

An even better way in the firefox browser is to open each of these pages in separate tabs, then use the firefox facility to save the entire set of tabs as a single bookmark. This means that one click gets all the pages, and you simply scan from one tab to the next to look at them all.

Firefox is free and open source and available for all the platforms running CPDN, more info & downloads from www.firefox.com

It is still 12 clicks to see all the info, so I\'d still prefer some way of getting it all onto one page. Also it means that when I click on the one bookmark it sends 11 queries to the database all together - a well designed single page might be less of a load on the database

River~~

edits: speeling & typos ;-)


I do agree with this fullheartedly! It gets especially annoying when you use computers with multiple CPUs, each CPU running its own workunit. For each CPU you need at least 4 clicks to find out if it reported in or not:
1. Your Account
2. Results
3. Result ID
4. Trickle #
All of this times the number of CPUs you use. And this is the best-case scenario!

ID: 18562 · Report as offensive     Reply Quote
staffann

Send message
Joined: 23 Oct 05
Posts: 22
Credit: 526,746
RAC: 0
Message 18569 - Posted: 21 Dec 2005, 18:23:42 UTC - in response to Message 18543.  

A \"recent trickles\" page would indeed be very welcome, if possible to do without excess db load. The way it is now is a bit annoying in the long run.
ID: 18569 · Report as offensive     Reply Quote
Profile Pooh Bear 27
Avatar

Send message
Joined: 5 Feb 05
Posts: 465
Credit: 1,914,189
RAC: 0
Message 18597 - Posted: 22 Dec 2005, 2:00:31 UTC

Suggestion...

The computer list shows the last time the computer talked to the server, how about each WU also show the last time it was trickled on that page? At least we would quickly know if trickles are coming in.


ID: 18597 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7165
Credit: 22,734,516
RAC: 9,285
Message 18599 - Posted: 22 Dec 2005, 2:38:38 UTC

All these suggestions require a commodity that the two programers don\'t have: time.
And the support staff have all moved on to other things.

ID: 18599 · Report as offensive     Reply Quote
Profile Ananas
Volunteer moderator

Send message
Joined: 31 Oct 04
Posts: 336
Credit: 3,316,482
RAC: 0
Message 18604 - Posted: 22 Dec 2005, 8:07:25 UTC
Last modified: 22 Dec 2005, 8:17:34 UTC

Question is, if the database even needs all those trickles. If I understood the concept of the 6-hourly fixup run right, only the latest / highest trickle for each active model is relevant.

Models in over/success state probably wouldn\'t need even that.

The information of this latest trickle could be in the table of results, so it wouldn\'t even be an extra record in an extra table. This would be a temporary redundancy - but only until the next fixup collected the trickles, so it would save a lot of queries and db space on the long run.
ID: 18604 · Report as offensive     Reply Quote
Profile old_user849

Send message
Joined: 14 Aug 04
Posts: 37
Credit: 276,676
RAC: 0
Message 18792 - Posted: 28 Dec 2005, 8:08:50 UTC

Hi,

Perhaps a script of some kind can be added so as to update credit displays (from Trickles) every, say, 8 hours, so as to better handle the server load.

As it is , my trickles RAC is riseing but on the stats display pages it is falling.


K.

ID: 18792 · Report as offensive     Reply Quote
Profile Honza
Volunteer moderator

Send message
Joined: 5 Aug 04
Posts: 390
Credit: 2,475,242
RAC: 0
Message 18794 - Posted: 28 Dec 2005, 8:20:10 UTC - in response to Message 18792.  

Perhaps a script of some kind can be added so as to update credit displays (from Trickles) every, say, 8 hours, so as to better handle the server load.
As it is , my trickles RAC is riseing but on the stats display pages it is falling.
As I understand, millions of trickles are to \'blame\' databse sluggish so recounting them several times a day is not an option (for the moment).
RAC never did good job for CPDN...
<i>phpBB forum for CPDN, all are </i><a href="http://www.climateprediction.net/board">invited</a>
ID: 18794 · Report as offensive     Reply Quote

Message boards : Number crunching : Database Sluggish -- Moved Trickle Info \'Down\'

©2020 climateprediction.net