climateprediction.net home page
No trickles on webpage

No trickles on webpage

Message boards : Number crunching : No trickles on webpage
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 5 · 6 · 7 · 8 · 9 · Next

AuthorMessage
Profile Bonsai911

Send message
Joined: 9 Sep 04
Posts: 228
Credit: 30,230,084
RAC: 3,092
Message 61927 - Posted: 8 Jan 2020, 8:51:39 UTC

once again...

For example https://www.cpdn.org/cpdnboinc/workunit.php?wuid=11935086

Cheers,

Bonsai911
ID: 61927 · Report as offensive     Reply Quote
Profile geophi
Volunteer moderator

Send message
Joined: 7 Aug 04
Posts: 2167
Credit: 64,403,322
RAC: 5,085
Message 61929 - Posted: 8 Jan 2020, 16:04:14 UTC - in response to Message 61927.  

@Bonsai911

How far along (what is the progress) for the tasks that aren't trickling? You have a whole bunch of tasks on that PC that are trickling just fine and have been all along. Usually when someone reports missing trickles, it's because a server reboot occurred and the trickle storage task didn't get restarted. The missing trickles here don't look to be a server side problem.
ID: 61929 · Report as offensive     Reply Quote
Profile Bonsai911

Send message
Joined: 9 Sep 04
Posts: 228
Credit: 30,230,084
RAC: 3,092
Message 61936 - Posted: 8 Jan 2020, 22:58:06 UTC - in response to Message 61929.  

Thanks for your reply!
I posted the message to early. The trickle appeared this evening.

Thanks and keep up the good work.

Bonsai911
ID: 61936 · Report as offensive     Reply Quote
Profile Bonsai911

Send message
Joined: 9 Sep 04
Posts: 228
Credit: 30,230,084
RAC: 3,092
Message 61953 - Posted: 10 Jan 2020, 9:13:35 UTC - in response to Message 61936.  

After further examination:

There is a delay between sending trickles and showing up on the website. After some hours I recognized the delay is growing.

Is there a server slowing down?

Questions over questions,

bonsai911
ID: 61953 · Report as offensive     Reply Quote
Profile Bonsai911

Send message
Joined: 9 Sep 04
Posts: 228
Credit: 30,230,084
RAC: 3,092
Message 61954 - Posted: 10 Jan 2020, 9:20:52 UTC - in response to Message 61953.  

ID: 61954 · Report as offensive     Reply Quote
lazlo_vii

Send message
Joined: 11 Dec 19
Posts: 108
Credit: 3,012,142
RAC: 0
Message 61955 - Posted: 10 Jan 2020, 9:23:11 UTC - in response to Message 61953.  
Last modified: 10 Jan 2020, 9:23:40 UTC

This is just a guess. I think that trickles updating and the work unit "back off timer" might be synced. If this is true then updating for new work would delay trickles updating on your account and vise versa. If this is not true then perhaps there is some other CPU intensive load on the systems. It could be that the weekly script to calculate points is delaying the trickle updates for a day or two.
ID: 61955 · Report as offensive     Reply Quote
Profile Bonsai911

Send message
Joined: 9 Sep 04
Posts: 228
Credit: 30,230,084
RAC: 3,092
Message 61956 - Posted: 10 Jan 2020, 9:27:57 UTC - in response to Message 61955.  
Last modified: 10 Jan 2020, 9:40:48 UTC

Maybe you are right.
So let's see and wait. In approx. two hours we know more.

We started up from this point
Last time contacted server 10 Jan 2020, 6:41:01 UTC

ID: 61956 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4314
Credit: 16,378,503
RAC: 3,632
Message 61957 - Posted: 10 Jan 2020, 12:08:29 UTC - in response to Message 61955.  

This is just a guess. I think that trickles updating and the work unit "back off timer" might be synced. If this is true then updating for new work would delay trickles updating on your account and vise versa. If this is not true then perhaps there is some other CPU intensive load on the systems. It could be that the weekly script to calculate points is delaying the trickle updates for a day or two.


There was a power outage on two of the physical machines in Oxford last night. Andy says all services have now been restored.

For the record, the time out for new work does not affect trickles being reported.
ID: 61957 · Report as offensive     Reply Quote
lazlo_vii

Send message
Joined: 11 Dec 19
Posts: 108
Credit: 3,012,142
RAC: 0
Message 61959 - Posted: 10 Jan 2020, 21:22:07 UTC
Last modified: 10 Jan 2020, 21:23:32 UTC

It does seem that something is up with the trickles on my N216 jobs. None of the trickles sent in the past six hours have shown up yet. I am not that worried about it though. My event logs show that the uploads completed without error and that means the model data went where it is supposed to.
ID: 61959 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4314
Credit: 16,378,503
RAC: 3,632
Message 61960 - Posted: 10 Jan 2020, 21:24:53 UTC - in response to Message 61959.  

It does seem that something is up with the trickles on my N218 jobs. None of the trickles sent in the past six hours have shown up yet. I am not that worried about it though. My event logs show that the uploads completed without error and that means the model data went where it is supposed to.


Will let Andy know. I suspect one of the scripts didn't get restarted after the power outage that affected some of the servers last night.
ID: 61960 · Report as offensive     Reply Quote
lazlo_vii

Send message
Joined: 11 Dec 19
Posts: 108
Credit: 3,012,142
RAC: 0
Message 61961 - Posted: 10 Jan 2020, 21:26:20 UTC - in response to Message 61960.  

Like I said, it'd not a big deal to me. I know that data is there and will get processed sooner or later.
ID: 61961 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4314
Credit: 16,378,503
RAC: 3,632
Message 61962 - Posted: 10 Jan 2020, 21:30:36 UTC - in response to Message 61961.  

Not a big deal for me either but sooner or later if Andy isn't told, there will be a flood of people posting here because they aren't getting their credit.

(Assuming the trickles aren't going through and just not showing up on the model pages!)

Email has been sent.
ID: 61962 · Report as offensive     Reply Quote
Profile Alan K

Send message
Joined: 22 Feb 06
Posts: 484
Credit: 29,579,234
RAC: 4,572
Message 61963 - Posted: 10 Jan 2020, 23:33:16 UTC - in response to Message 61959.  

The N216 models are for 4 months so the webpage will only show these at 25% completion intervals, which in my cas is about 3.5 days (14day estimated completion). How fast is your macine?
ID: 61963 · Report as offensive     Reply Quote
lazlo_vii

Send message
Joined: 11 Dec 19
Posts: 108
Credit: 3,012,142
RAC: 0
Message 61964 - Posted: 10 Jan 2020, 23:54:35 UTC - in response to Message 61963.  

The N216 models are for 4 months so the webpage will only show these at 25% completion intervals, which in my cas is about 3.5 days (14day estimated completion). How fast is your macine?


Right now I am running 6 models, 3 per 4-core CCX on Ryzen 3700X. They are progressing at little over 0.56% per hour. If my math is right that is between 17 and 19 sec/TS. They all reached the 25% trickle point before I posted to report the missing info. I can't check my math yet due the missing info.

https://www.cpdn.org/result.php?resultid=21871315
https://www.cpdn.org/result.php?resultid=21872035
https://www.cpdn.org/result.php?resultid=21872143
https://www.cpdn.org/result.php?resultid=21871965
https://www.cpdn.org/result.php?resultid=21872208
https://www.cpdn.org/result.php?resultid=21871936
ID: 61964 · Report as offensive     Reply Quote
Profile geophi
Volunteer moderator

Send message
Joined: 7 Aug 04
Posts: 2167
Credit: 64,403,322
RAC: 5,085
Message 61969 - Posted: 11 Jan 2020, 4:17:25 UTC - in response to Message 61964.  

If interested, you can check the average sec/TS by going into the

"boinc cliient directory"/projects/climateprediction.net/"name of model task directory"

and tailing the stdout_mon.txt file

tail -f stdout_mon.txt
ID: 61969 · Report as offensive     Reply Quote
lazlo_vii

Send message
Joined: 11 Dec 19
Posts: 108
Credit: 3,012,142
RAC: 0
Message 61970 - Posted: 11 Jan 2020, 6:10:44 UTC - in response to Message 61969.  
Last modified: 11 Jan 2020, 7:05:17 UTC

If interested, you can check the average sec/TS by going into the

"boinc cliient directory"/projects/climateprediction.net/"name of model task directory"

and tailing the stdout_mon.txt file

tail -f stdout_mon.txt



Useful! Thank you.

Edit: These N216 models seem to have semi-regularly spaced time-steps that take more than 4x CPU time than most of the other other time-steps. I think it is a little bit like the "key frames" used to encode video. It does make sense if you consider that in the models every "cell" (temp, air pressure, humidity, and a lot more than I think off of the top of my head) in the 3D matrix will influence it's neighboring cells (and those cells must influence their neighbors and so on until you reach the edges of the zone being modeled) at each time-step. If this is true then sooner or later a model will reach a "large change" state where one or more cells change in a way that critically influences the entire model.
ID: 61970 · Report as offensive     Reply Quote
Profile geophi
Volunteer moderator

Send message
Joined: 7 Aug 04
Posts: 2167
Credit: 64,403,322
RAC: 5,085
Message 61971 - Posted: 12 Jan 2020, 1:32:51 UTC - in response to Message 61970.  

These N216 models seem to have semi-regularly spaced time-steps that take more than 4x CPU time than most of the other other time-steps. I think it is a little bit like the "key frames" used to encode video. It does make sense if you consider that in the models every "cell" (temp, air pressure, humidity, and a lot more than I think off of the top of my head) in the 3D matrix will influence it's neighboring cells (and those cells must influence their neighbors and so on until you reach the edges of the zone being modeled) at each time-step. If this is true then sooner or later a model will reach a "large change" state where one or more cells change in a way that critically influences the entire model.

Those are the timesteps that include the radiation calculations. Most of the models do that calculation every 3 hours.

The hadam4/hadam4h models are global models with no regional component. The Weather At Home (WAH) models that run on Windows have both global and regional components, where the global model (at a lower resolution) is run for one day, then the regional model grid is run for that day, with boundary conditions from the global model, then on to the next day, etc.
ID: 61971 · Report as offensive     Reply Quote
lazlo_vii

Send message
Joined: 11 Dec 19
Posts: 108
Credit: 3,012,142
RAC: 0
Message 61972 - Posted: 12 Jan 2020, 4:15:27 UTC - in response to Message 61971.  

Thank you for the clarifications. One of the reasons I love DC projects so much is that I can always learn something new.
ID: 61972 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4314
Credit: 16,378,503
RAC: 3,632
Message 61974 - Posted: 14 Jan 2020, 9:56:05 UTC

And, getting back to the trickles, Andy emailed yesterday to say that the problem should be sorted.
ID: 61974 · Report as offensive     Reply Quote
bernard_ivo

Send message
Joined: 18 Jul 13
Posts: 438
Credit: 24,488,575
RAC: 2,962
Message 62270 - Posted: 30 Mar 2020, 15:47:38 UTC

Hi
It seems there might be a problem with trickles, after 21-22 March
I have at least 3 N216 that do not have their 3&4 trickle on the web despite they finished successfully and upload queues are empty.

here is an example https://www.cpdn.org/cpdnboinc/result.php?resultid=21871312
ID: 62270 · Report as offensive     Reply Quote
Previous · 1 . . . 5 · 6 · 7 · 8 · 9 · Next

Message boards : Number crunching : No trickles on webpage

©2024 climateprediction.net