climateprediction.net home page
Posts by alanb1951

Posts by alanb1951

1) Message boards : Number crunching : OpenIFS Discussion (Message 67878)
Posted 13 days ago by alanb1951
Post:
What is shrss?
According to the man page for atop on XUbuntu 22.04 it is "the resident size of shared memory (`shrss`)" (same as SHR in top?)

Cheers - Al.
2) Message boards : Number crunching : One of my computers is hoarding tasks and I don't know why (Message 67737)
Posted 17 days ago by alanb1951
Post:
Steven,

I notice that your system with more memory is running a fairly recent BOINC client (7.20.2) whereas the others that I looked at seem to be running 7.18.1.

If I recall correctly, the fix for the "use of max_concurrent may lead to excess work being fetched" problem didn't make it into the Linux client until the 7.20 versions; it may well be that if you get hold of a 7.20 client the problem will go away!

With or without use of [project_]max_concurrent, I used to have to moderate CPDN downloads by using No New Tasks as the default for CPDN, cutting the number of available "CPUs" before allowing new tasks, updating, then setting No New Tasks again and resetting the CPU count (in that order!)[1] -- it would always send at least enough work to occupy every visible "CPU"... As CPDN and WCG were my only projects doing CPU work, and I could limit work downloads for WCG at the server end, I wasn't seeing the overload issue until WCG went on hiatus, at which point the alternative projects I took CPU work from started to over-load if I used [project_]max_concurrent. However, once I found a 7.20 client that stopped.

Cheers - Al.

[1] Not very convenient -- much care needed to make sure no GPU tasks running at the time...
3) Message boards : Number crunching : OpenIFS tasks : make sure boinc client option 'Leave non-GPU tasks in memory' is selected! (Message 67305)
Posted 28 days ago by alanb1951
Post:
Richard,

Thanks for having a look to see what's going on...

The message is written by https://github.com/BOINC/boinc/blob/master/client/app_control.cpp#L1551, and seems to be controlled by

old_time = atp->checkpoint_cpu_time;  // the saved time of the last checkpoint
if (old_time != atp->checkpoint_cpu_time) {  // if they are different ...
so you shouldn't see two messages with the same time.

Edit - OK, so 1 second apart is indeed 'different'. But I can never unravel David Anderson's spaghetti code much beyond that.

But it isn't trying to checkpoint so unless something is writing a non-zero value to the task's checkpoint_cpu_time it should just do nothing (always zero) - or have I misread/misunderstood that section of code (quite likely; my opinion of David's code is much the same as yours!)?

And from a later message...

I think the wisest thing is to turn off that log flag (it'll spam the system journal in no time), and stop worrying about it.

That's what I did, but as soon as WCG's GPU application returns I either forego some performance analysis I'm doing that needs to know where it checkpoints (to make some sort of sense of a GPU activity trace) or I forego CPDN (as I currently have no machines with the capacity to consider setting up a second BOINC client with different log behaviour...)

Now, my tiny contribution wouldn't be missed (no sarcasm intended), but if someone could find out how to stop that spam...

Cheers - Al.
4) Message boards : Number crunching : OpenIFS tasks : make sure boinc client option 'Leave non-GPU tasks in memory' is selected! (Message 67292)
Posted 28 days ago by alanb1951
Post:
I asked about the checkpoint "log spam" in the OpenIFS discussion thread on 23rd December but. given when I posted it, that message is now two pages back :-)

For what it's worth, BOINC Manager on my machine doesn't seem to think the application checkpoints using the client checkpoint mechanism at all whilst it's running if you look at the task properties (there was never a last checkpoint time); that's consistent with my understanding of some of what Glenn has said about the matter, but it doesn't explain this -- is it something odd in the client libraries or something in the CPDN wrapper or main program?

It would be nice if it didn't do this, and it would be interesting to know why it does do it!

Cheers - Al.

P.S. Please tell me it's not using something in the BOINC checkpoint mechanism as a 1 second timer :-) ...
5) Message boards : Number crunching : OpenIFS Discussion (Message 67014)
Posted 23 Dec 2022 by alanb1951
Post:
I happened to try a couple of these tasks to see what effect they would have on the rest of my BOINC work-load. No problems there, but...

I usually have checkpoint debug turned on if I'm running certain WCG tasks or if I'm doing perf stat analyses (trying to dodge genuine checkpoints!). Imagine my surprise when I found that my BOINC log was being "spammed" with a checkpoint message once a second (or, more accurately, 9 or 10 times in every 10 or 11 seconds), with gaps of a few seconds whenever it was consolidating files or arranging an upload. Given that the BOINC standard checkpoint mechanism is apparently not being used by the application, this seems a bit strange :-)

[If this has already been discussed I missed it; that said, I don't suppose that many people do checkpoint debug most of the time!...]

Here's the front of the first task starting up...

22-Dec-2022 03:24:54 [climateprediction.net] Starting task oifs_43r3_ps_0497_1993050100_123_962_12179141_0
22-Dec-2022 03:24:54 [climateprediction.net] [cpu_sched] Starting task oifs_43r3_ps_0497_1993050100_123_962_12179141_0 using oif
s_43r3_ps version 105 in slot 6
22-Dec-2022 03:25:00 [climateprediction.net] [checkpoint] result oifs_43r3_ps_0497_1993050100_123_962_12179141_0 checkpointed
22-Dec-2022 03:25:01 [climateprediction.net] [checkpoint] result oifs_43r3_ps_0497_1993050100_123_962_12179141_0 checkpointed
22-Dec-2022 03:25:03 [climateprediction.net] [checkpoint] result oifs_43r3_ps_0497_1993050100_123_962_12179141_0 checkpointed
22-Dec-2022 03:25:04 [climateprediction.net] [checkpoint] result oifs_43r3_ps_0497_1993050100_123_962_12179141_0 checkpointed

and here's one of the intervals where I believe it was doing file movement/uploading...

22-Dec-2022 03:36:40 [climateprediction.net] [checkpoint] result oifs_43r3_ps_0497_1993050100_123_962_12179141_0 checkpointed
22-Dec-2022 03:36:41 [climateprediction.net] [checkpoint] result oifs_43r3_ps_0497_1993050100_123_962_12179141_0 checkpointed
22-Dec-2022 03:37:00 [World Community Grid] [checkpoint] result MCM1_0193439_9713_3 checkpointed
22-Dec-2022 03:37:04 [climateprediction.net] [checkpoint] result oifs_43r3_ps_0497_1993050100_123_962_12179141_0 checkpointed
22-Dec-2022 03:37:05 [climateprediction.net] [checkpoint] result oifs_43r3_ps_0497_1993050100_123_962_12179141_0 checkpointed

Now, the writing of these lines isn't a major I/O nuisance, but it is a space-consuming one! So eventually I got fed up and turned off checkpoint debug logging :-) -- fortunately, I'm not running WCG work that I want to monitor at present, and I would quite like to see what happens to throughput with one of these in a machine at the same time as a WCG ARP1 task (though there aren't any at present...) so I'll carry on with my [infinitely small] contribution for now.

If this shouldn't be happening, I hope it can be stopped... If, however, it's a natural part of how the programs are designed, I'd be interested to know why it happens.

Cheers - Al.
6) Message boards : Number crunching : Site problems (Message 64568)
Posted 2 Oct 2021 by alanb1951
Post:
Looks like file names may be slightly different in Red Hat for the record the ca-bundle.crt on Ubuntu is by default in /var/lib/boinc-client/

For what it's worth, on Ubuntu that entry in boinc-client is a link to /etc/ssl/certs/ca-certificates.crt, which gets updated as necessary - I would expect most Linux distributions that have a properly curated BOINC client package to do something similar, but I could be wrong :-)

And I note that we still have the (internal to CPDN) PHP warnings at the top of every page, so it looks like Oxford have some sorting out to do anyway...

Cheers - Al.

[Edit...] P.S. The certificate bundle was most recently updated on 28th September, 2021, round about when this root certificate expired.
7) Message boards : Number crunching : Site problems (Message 64549)
Posted 1 Oct 2021 by alanb1951
Post:
I wonder if any of their certificates have DST Root CA X3 at the top of their certificate chain. It lapsed on 29th September, I believe...

Cheers - Al.
8) Questions and Answers : Unix/Linux : *** Running 32bit CPDN from 64bit Linux - Discussion *** (Message 63311)
Posted 11 Jan 2021 by alanb1951
Post:
Disclaimer, I have never tried running CPDN in a VBox client, so what follows may be irrelevant, but it does relate to VBox performance issues.

I used to run 32-bit WCG tasks in a VBox client for throughput experiments (and usually the 32-bit tasks in the VM finished slightly quicker than similar 64-bit tasks on the host) I still use VBox for other, non BOINC-related, tasks such as a Windows VM for my camera update software(!) and a Linux VM for home banking and such like.

A couple of observations about VirtualBox: it seems that whenever I did a significant version upgrade (rather than a maintenance update) something would need re-configuring. If lucky, it would just be the video or sound; however, it was often something that affected performance!

Firstly, if you don't install up-to-date Guest Additions, performance or behaviour may suffer.

Then there's the VBox version... One major update I did turned off the user of KVM paravirtualization, for instance. Performance hit! And somewhere around my move from XUbuntu 18.04 to 20.04 (host and client, in this particular case), it turned out that one of my VMs was not using host cache for disk I/O - that VM was running like a slug until I found that issue via the VBox online forums.

My apologies if none of the above is news, but as I don't see massive performance hits if I time some of my home-grown non-interactive programs in a VM and on the host of that VM (trying to ensure nothing else is busy at the time), I wonder if some of it is "tuning" related.

I have never tried using Windows as a host, by the way; the above is all Linux-hosted. Someone hoating on Windows might know whether something akin to the KVM or host-cache might be an issue there...

Cheers - Al.

[Edited to add the rider about Windows hosts...]
9) Message boards : Number crunching : Big models (Message 62639)
Posted 7 Aug 2020 by alanb1951
Post:
Question: How do people feel about a monthly upload of around 193Mb?

The answer to that rather depends on how long it takes to produce a month's worth of data to upload! If these are going to be models that do several years in a single job, that could be several "months" per real-time day, after all.

And there's another issue that may be critical - checkpointing. If the checkpoints are as frequent as they were on those HadCM3s ones we had last Autumn, folks with ext4 filestore are going to be a tad unhappy! And, of course, if there are too many jobs running at once the machine could become disk-bound if running spinning media rather than solid-state...

(ext4 is more or less the default nowadays, I believe, and as far as I am aware there is no way to avoid more or less immediate writes without turning journaling off, which rather defeats the point... One could work around it by putting [part of] /var/lib/boinc-client on a separate ext3 partition and playing with cache parameters, I suppose, but not everyone is a Linux guru...)

The above said, it's good to know there might be some new work in the pipeline, and perhaps it'll be 64-bit and more tuned to modern hardware???

Cheers - Al.
10) Message boards : Number crunching : "No tasks sent" (Message 62420)
Posted 10 May 2020 by alanb1951
Post:
There should be some way to better spread these small test batches (if that’s what this micro-batch was) around so that they don’t all get sucked into this kind of black hole machine. Otherwise this will become more and more of a problem as processors get ever larger numbers of cores.


I agree. Usually this is done via the testing site. I have I suspect the slowest machine on there and on that these tasks would take about 20 days. The machine I use for testing site runs 24/7 which the machine with the tasks clearly doesn't as it is a lot faster yet still takes 65 days to turn tasks around..

I don't know why these were sent out on the main site rather than testing. The other option if there is a good reason for using the main site for them would be to send out say 100 instead of 15 which would hopefully get enough data back quickly.

Dave,

There is another reason a system can seem to take an age to send stuff back, which you may or may not have considered. I run CPDN on an i7-6700K and a Ryzen 3700X system, both of which take about 7.5 days to run one HadAM4h task but which seem to take a month or more to turn jobs around. In my case this is because if I'm not careful CPDN will send me as many tasks as I have "CPUs" available to BOINC but I only run one (i7) or two (Ryzen) CPDN tasks at once! Perhaps that machine that snagged all the WIndows jobs hit the same problem?...

I wish every BOINC site had the facility they've implemented at WCG whereby you can say "I never want more than N of these on my machine at once" and it won't send work that would exceed that. The alternative, of course, would be if the BOINC client respected the max_concurrent option and sent that as the number of CPUs instead of the actual number, which would offer the same sort of control!

Cheers - Al.
11) Message boards : Number crunching : Work available and being requested but none downloaded (Message 62381)
Posted 1 May 2020 by alanb1951
Post:
Although it probably won't help Bryn Mawr solve the problem, I thought I'd post my observations on how fetching from CPDN has worked recently.

I have two Linux machines that request CPDN tasks on occasion. I use app_config.xml to divide the work-load as I require, and set CPDN to a lower resource share to reflect the relative workloads wanted, but whenever a necessary CPDN work fetch takes place it will try to fetch enough tasks to have one for each "CPU" BOINC has access to! From that, it would appear that the only part of the request that is of relevance to the CPDN server is the inst value!

To ensure I only get as many tasks as I want my routine is usually as follows: suspend WCG (my only other CPU task source), reduce %age of CPUs to use to match the number of CPDN tasks I want on the machine, allow CPDN to get work then update CPDN - it will fetch just enough tasks to make up the numbers - then set CPDN back to "No New Tasks" to avoid future unwanted fetches, restore the original %age of CPUs and resume WCG!

I had last observed this on 25th April when I forgot to follow my usual CPDN fetch routine on my i7-7700K (BOINC is allowed 5 "CPUs" for CPU work [there's a GPU too...] and I only run 1 CPDN task at a time, so I try for two tasks at any one time...) and when it ran out of tasks it fetched 5 new ones because I hadn't put No New Tasks on previously!

However, as there seems to be a suggestion that this might be a recent change, I have just allowed my other machine (a Ryzen 3700X, with 13 of the 16 "CPUs" allowed to BOINC (and CPDN restricted to two)) to get more work; it had 2 running and 4 waiting [already more than I really wanted, but never mind...], so I paused WCG, set CPUs to 50% and allowed CPDN to fetch - it got two tasks as I expected (and the request was as shown below...)

Fri 01 May 2020 06:43:36 BST | climateprediction.net | [work_fetch] set_request() for CPU: ninst 8 nused_total 6.00 nidle_now 2.00 fetch share 1.00 req_inst 2.00 req_secs 86400.00
Fri 01 May 2020 06:43:36 BST | climateprediction.net | [work_fetch] request: CPU (86400.00 sec, 2.00 inst) NVIDIA GPU (0.00 sec, 0.00 inst)

Fri 01 May 2020 06:43:36 BST | climateprediction.net | Sending scheduler request: To fetch work.
Fri 01 May 2020 06:43:36 BST | climateprediction.net | Requesting new tasks for CPU
Fri 01 May 2020 06:43:42 BST | climateprediction.net | Scheduler request completed: got 2 new tasks

Note that I have my cache control set to 0.45 + 0.05 days so it asked for 1 day of work, (The Ryzen usually takes just over 7 days to do one of these, so if it were paying any attention to that it wouldn't send anything!) Note also there's a GPU, but I doubt that's making any difference to the actual request.

Pursuant to Les's comment about older clients, and other things that might come into play, my Ryzen is on XUbuntu 18.04.04 with a 5.3 kernel and uses BOINC client 7.9.3 (I didn't want the version available from Gianfranco's repository when I set it up), and the Intel box is on an earlier 18.04 with client 7.14.2.

So, as I said, probably not helpful for Bryn Mawr but it is a "data point"... (and perhaps a marker to try a different client?)

Good luck - Al.

[Edited to add some (obvious?) steps I'd forgotten to list...]
12) Message boards : Number crunching : New work Discussion (Message 62050)
Posted 27 Jan 2020 by alanb1951
Post:
@Wolfman1360
I vaguely remember discussion of Rosetta eating up l3 cache as well, but can't find the discussion anywhere.
Is this still true today and should I be limiting it alongside the n216 and n144?

Jim1348 has referred to local threads where this has come up; if you look in the threads about UK Met Office HadAM4 at N216 resolution and UK Met Office HadAM4 at N144 resolution you'll find several mentions of L3 cache bashing (especially in the N216 thread, but in this message in the N144 thread I actually replied to one of your posts, talking about workload mixes (and again in this message)... Jim1348 (and others) had some good contributions in those threads too. I don't recall many explicit references to Rosetta, but WCG MIP1 (which uses Rosetta) got some dishonourable mentions...

You may also have seen (or even participated in) threads about MIP1 at WCG -- because of the model construction it uses, the rule of thumb is that one MIP1 per 4 or 5 MB of L3 cache! I haven't got time to track those down at the moment - sorry!

For what it's worth, if you run MIP1 alongside N216 you'll see the same sort of hit as if running extra N216 tasks; N144 is nowhere near as bad!

Cheers - Al.

[Edited to fix a broken link, then to fix a typo I'd missed!]
13) Message boards : Number crunching : UK Met Office HadAM4 at N144 resolution (Message 61757)
Posted 21 Dec 2019 by alanb1951
Post:
@Wolfman1360
Will I see much of a performance gain by allowing all cores to crunch away at N144 with hyper threading enabled, or will this just be twice the time for little if any gain?
Right now I have n144 limited to 8 concurrent on the Ryzen and 4 on the 4770 and 2600 thus theoretically illiminating HT, and 4 216 on the Ryzen with 2 allowed on the 2 intels. I'll change this to 1 per your advice, thank you. Rosetta is currently using the remaining 8 threads of the Ryzen and 4 of the other two. I've been a strong supporter of WCG since 2016 and need a bit of a break from there, though I will try and get some of the arp.

Regarding cutting down the numbers - my comment about cache use for CPDN was specific to N216 tasks (which run HadAM4h - I presume the h is for high(er) resolution!) You would probably still be o.k. with 2 N144 tasks on an 8MB-cache machine most of the time, cutting down to one if more N216 turns up!

As for multiple tasks and hyperthreading - that one is a lot more complex, because unless you can guarantee which CPU thread runs a particular application you can't be sure whether you might get two heavy floating-point using applications on the same core (at which point you may well see a fairly substantial throughput drop per individual process, even if the applications aren't particularly hard on the L3 cache...) However, if you get lucky and one of the applications is mostly shuffling stuff around and doing pattern-matching or integer arithmetic a core may be able to keep both threads fairly busy!

I reckon on about a 15% drop on individual processes, but even then if I'm running 6 tasks on my 7700K or 14 tasks on my 3700X (instead of 4 or 8 respectively) I get more work done. If the drop was 40+ % I probably wouldn't use the hyperthreads...

Whatever your workload mix, there will probably be a point at which using more cores/threads will actually result in doing less science per hour - there was a thread over in the SETI forums about the best CPU workload on a Threadripper, and if I remember rightly the conclusion was that if more than 75% of the threads were used the performance drop became unacceptable. And I have noticed that if I don't leave two threads clear for operating system stuff on my 3700X performance takes a noticeable hit because of all the I/O services and such like causing extremely high numbers of cpu-migrations (expensive) as well as the expected context switches (reasonably efficient)...

I have lots of numbers about throughput for various workloads, but there really is no simple (or brief) answer to "what will happen if..." - you'll have to try things out. And if you are interested enough, and you have a Linux system on which you can get root access, you can get hold of the performance tools to check out throughput, context switches, cache access et cetera for yourself!

Good luck arriving at a workload mix that meets your objectives!

Cheers - Al.

P.S. My best machine for running individual processes is an i5-7600 (4 cores, no hyperthreading), which runs tasks about 10% faster than either my 7700K or 3700X despite being run at about 12% lower CPU clock speed. It only does WCG work but I only allow 1 MIP1, don't run ARP1 and use only 3 cores for BOINC.
14) Message boards : Number crunching : UK Met Office HadAM4 at N144 resolution (Message 61744)
Posted 20 Dec 2019 by alanb1951
Post:
@Wolfman1360

Les is right about N144 (HadAM4) not being anywhere near as much of a "nuisance" as N216 (HadAM4h)! He's also right about the i7 and its cache limitations.

However, you mentioned running WCG work on the same system... You may or may not be aware of the issues regarding MIP1 and L3 cache (it likes about 4 or 5MB too!) Some of the other projects also work on quite large memory grids but don't seem to put so much stress on L3 cache (though the cumulative effect might mount up if you run several at once.)

Best WCG projects to run alongside CPDN are MCM1 and anything VINA-based (e.g. FAHV if/when it returns, SCC1 when it returns, and the recently completed OET1 and ZIKA projects); those tend to require less volume of L3 cache and their performance does not seem to degrade as dramatically (or put excessive stress on other applications). The worst is definitely MIP1!

Good luck deciding on a workload mix that suits you on that Ryzen, and don't try to run more than one HadAM4h or MIP1 on that 4770! (I have an i7-7700K (8MB L3 cache) and that shows severe performance degradation if I let it run more than one of either of those!!!)

Cheers - Al.

[Edited for sentence order...]
15) Message boards : Number crunching : UK Met Office HadAM4 at N216 resolution (Message 61444)
Posted 2 Nov 2019 by alanb1951
Post:
Jim1348 - re your Ryzen tasks...

I followed the link in your post above to see how you might be getting on with tasks that didn't abort(!) and was intrigued to see tasks taking well over 20 seconds per time step. So far I've finished two (each took about 7 days, 7hours) and in both cases the average per time step is under 18 seconds...

So I wondered how many you are running at a time and, perhaps, what your overall workload is on that system. On mine I only allow 2 CPDN at a time (and I also only allow 2 WCG MIP1 (cache-killers!)) - I also only let BOINC have 14 out of 16 "CPUs"

Fun machines, aren't they! I could write an essay about the machine turning the CPU fan off with 14 tasks running, but I won't...

Cheers - Al.
16) Message boards : Number crunching : UK Met Office HadAM4 at N216 resolution (Message 61392)
Posted 26 Oct 2019 by alanb1951
Post:
In reply to Jim1348 in re Ryzen 3700X

I'm about to take delivery of a Ryzen 3700X (32MB L3 cache, though I gather access is constrained to 8MB per 2 cores (4 threads)); I'll be interested to see how that behaves as and when it gets some CPDN work to do (and will probably do some bulk tests with WCG MIP1 to get an idea if there's no CPDN work available!)

Cheers - Al.

[1] Someone over at WCG seemed to think 5MB cache was what a MIP1 job would like. The user offered no justification for that number but 4MB probably isn't enough for near-optimum performance.

Thanks a lot for the cache info. I was beginning to think that the issues were deeper than I had found.
I just happen to have a Ryzen 3700x, and was wondering what its large L3 cache would do here. But I would need to add more memory. So let us know, and I could do it.


I have finally got the beast up and running on Ubuntu 18.04-3 (kernel 5.0.0-32-generic). It has 32GB of 3200MHz RAM, boots from an NVMe SSD, and I've put /var on HDD RAID 1 so that logs and checkpoint files aren't hammering the SSD. (/home is on RAID as well - all my non-laptop builds are done like that...) I haven't done any tuning apart from making sure that the memory clock and fabric clock are fixed at 3200 and 1600 respectively.

It has taken until now to get a decent work-load built up; I'm currently running 12 WCG tasks (with a check to stop MIP1 from running more than two at a time) and 2 CPDN HadAM4h tasks at a time, along with one GPU task from SETI@Home, Einstein@Home or MilkyWay@Home, so the system is getting a fair work-out. It seems to have all clocks at about 3.95GHz, and the machine is drawing about 140W not counting the GPU.

As regards checkpointing and completion times, after the first checkpoint (which seems to cover a few more time steps) it seems to checkpoint about every 60 minutes. I haven't had one generate a trickle yet but at current rate of progress I expect a trickle at about 33 hours 20 minutes, and the tasks to finish in about 5 days 13 hours.

I'm going to let the machine run with that sort of work load for a while to make sure it's behaving consistently, and I plan on doing some experiments with more HadAM4h tasks running at once when the current two have finished. It might be interesting to find out how many I can run at once without serious degradation it the only other work on the machine is WCG MCM1 (which is very cache-friendly!)

I'll try to do some task-level performance stats at some point, but on AMD CPUs there's no direct way of getting a count of L3 cache misses (I think it counts them at the cache level rather than the CPU level...) so one key stat isn't available. Ah, well...

Hope this was of interest - Al.
17) Message boards : Number crunching : UK Met Office HadAM4 at N216 resolution (Message 61303)
Posted 21 Oct 2019 by alanb1951
Post:
TL;DR - you probably don't want to run more than one of these per 4MB+ of L3 cache...

Jim1328's time estimates for an i7-8700 prompted me to do some tests (see below) as my experience with the Microbiome application (MIP1) at WCG, which is also a memory hog, suggests that one should only run one instance of that per 4MB (or more[1]) of L3 cache; running more results in significant increases in cache misses, with a corresponding drop in overall CPU effectiveness (for any BOINC tasks running, not just the hogs!) -- indeed, running 4 at a time on a machine with 8MB cache resulted in CPU temperatures dropping by 10C or more and run times nearly double that of a single task (which I restricted using the max_concurrent mechanism)

Testing on an i5-7600 (6MB L3 cache, 4 cores, no hyper-threading, 8GB RAM, 3.5GHz clock) has shown HadAM4@N216 to be a cache-wrecker as well (no surprise there). I did tests with 1 HadAM4 task, 2 HadAM4 tasks, 3 HadAM4 tasks, and my normal workload if I have a CPDN task - 1 CPDN, 2 WCG.

Running a single HadAM4 task with no company yields a checkpoint every 81 minutes; running two at once yields checkpoints every 91 minutes; running three, checkpoints are about 110 minutes apart. This is consistent with changes in the number of instructions run in a fixed time interval, which I monitored with the perf stat command. As checkpoints seem to be taken once per model day and there are about 120 days per 4-month model I'd reckon these would complete in about 6.8 days (running 1 at a time), 7.6 days (running 2 at a time) or 9.2 days (3 at a time).

By the way, under my usual workload [avoiding MIP1 tasks as they mess up the cache too!], checkpoints are about 83 minutes apart, so it can be seen that the WCG tasks aren't really getting in the way. (If MIP1 tasks get in there, the checkpoints are about 86 minutes apart.)

There's one thing in favour of running lots of these on a multi-core machine - your power draw will drop (as evidenced by CPU temperatures!) as the cores end up waiting for memory accesses more and more often! But I suspect there comes a point where each task takes so long to run that it's just not worth it - I, for one, will continue to treat CPDN as minority work on my Intel machines in order to maximize throughput.

I'm about to take delivery of a Ryzen 3700X (32MB L3 cache, though I gather access is constrained to 8MB per 2 cores (4 threads)); I'll be interested to see how that behaves as and when it gets some CPDN work to do (and will probably do some bulk tests with WCG MIP1 to get an idea if there's no CPDN work available!)

Cheers - Al.

[1] Someone over at WCG seemed to think 5MB cache was what a MIP1 job would like. The user offered no justification for that number but 4MB probably isn't enough for near-optimum performance.
18) Message boards : Number crunching : Excessive checkpointing on new Linux hadcm3s tasks? (Message 61147)
Posted 2 Oct 2019 by alanb1951
Post:
Some follow-up information...

I looked at changing some of the cache control values as per Jim1348's notes; however, I don't think it makes any difference because I'm using ext4 filesystems and (as I understand it) they effectively force regular synchronization (5 seconds by default!). That would explain why iostat and cat /proc/diskstats didn't report any difference in amounts written when I tried (and might explain why some others aren't seeing changes...)

(Apparently, iostat and friends are supposed to report actual device activity, not user write requests...)

I'm actually measuring output to a spinning disk, not an SSD, so I can't use smartctl to confirm how much data is actually being written. Perhaps someone who is using an SSD and has adjusted those sysctl parameters could have a look at that?

If it turns out not to be possible to alter the checkpoint interval, I certainly won't be letting BOINC use an SSD if I intend to continue doing CPDN work! Each HadCM3s task writes about 383GB of checkpoint data during a 20-year model run, so 3 jobs -> 1 Terabyte!

By the way, the current HadAM4 jobs seem to checkpoint about once every 20 minutes on my machine (as against the once a minute of HadCM3s) but the checkpoint file is nearly 4 times the size of the pair of files written by HadCM3s tasks. It comes out at about 71GB of checkpoint data per 12-month model task.

Cheers - Al.
19) Message boards : Number crunching : Excessive checkpointing on new Linux hadcm3s tasks? (Message 61029)
Posted 27 Sep 2019 by alanb1951
Post:
This has been raised with the project. Check points are each model day. When this model type was introduced, computers were a lot slower and solid state disks didn't even exist or if they did cost an arm and a leg for even a 40GB one. So the problem has sort of crept up on us. I don't know how quick a fix it is to change checkpoints to every ten days or even monthly giving 12 checkpoints per zip file?

Dave,

Thanks for this!

I hope they can (and do) change this because I will not be running CPDN on my next system (Ryzen 3700X, I hope) if it's going to be hammering the discs like that if I get hadcm3s work units. (Thanks for the numbers and cache tuning stuff, Jim1348!)

I presume we aren't ever going to get the facility to deselect certain applications back; if that's the case they ought to try to make details like checkpoint frequency as consistent as they can across all applications available on a given platform (at least as far as the most frequent checkpointing is concerned).

I can understand an application ignoring the user's checkpoint guidelines by not checkpointing as often as the user allows, but checkpointing more often ought to be a no-no, as this situation demonstrates!... (It was theoretically possible to determine that limit and if the limit was reasonable enable some code on the checkpoint logic saying "how long since the last one? If longer than limit, do another..." -- I don't think that has changed.)

By the way, do we know what the checkpoint behaviour of HadAM4 and OpenIFS is/will be???

Cheers - Al.
20) Message boards : Number crunching : Excessive checkpointing on new Linux hadcm3s tasks? (Message 61008)
Posted 26 Sep 2019 by alanb1951
Post:
I recently landed a few of the new Linux tasks (batch 835). I don't usually turn on checkpoint debug in BOINC-Manager, but I had cause to need to do so on one of my machines and was surprised to see that these tasks were checkpointing about once a minute! I turned the logging on on my other machine that had some CPDN work and it was the same! (Turned logging off again!)

Now, on one machine I've got the checkpointing limit set to 600 seconds and on the other 240 seconds; it's obviously not respecting that!

As I said, I don't normally monitor this, so for all I know this could have been standard behaviour for as long as hadcm3s tasks have been available. Alternatively, it might only be doing this on my machines (though I suspect that's unlikely).

This is not exactly disc-I/O friendly if it's deliberate so I wonder has it always been like this, is this a side-effect of them trying to make Linux tasks more crash-proof, or is it a bug.

Any insight appreciated - Al.


Next 20

©2023 climateprediction.net