climateprediction.net (CPDN) home page
Thread 'New work Discussion'

Thread 'New work Discussion'

Message boards : Number crunching : New work Discussion
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 · 6 · 7 . . . 91 · Next

AuthorMessage
Alex Plantema

Send message
Joined: 3 Sep 04
Posts: 126
Credit: 26,610,380
RAC: 3,377
Message 55463 - Posted: 10 Jan 2017, 21:26:44 UTC - in response to Message 55462.  

Because of the far deadline CPDN doesn't get enough processor time when running multiple projects. That's why I don't run it together with other projects.
ID: 55463 · Report as offensive
ProfileJIM

Send message
Joined: 31 Dec 07
Posts: 1152
Credit: 22,363,583
RAC: 5,022
Message 55464 - Posted: 10 Jan 2017, 23:01:55 UTC - in response to Message 55462.  

The problem is that it works the other way. When I try to run CPDN alongside World Community Grid on the same machine, Boinc seems to fixate on short deadlines of WCG projects and hardly lets CPDN run at all. Unless I set the work buffer at less than 5 days it won’t let CPDN run at all. This despite the fact that the resource share is set at 600 for CPDN and 100 for WCG. Boinc seems to think that just because the deadline is 11 months it is alright to take 11 month to finish the tasks. Either CPDN needs shorter deadlines, or other project need longer, multi-month deadlines.
ID: 55464 · Report as offensive
ProfileDave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4541
Credit: 19,039,635
RAC: 18,944
Message 55465 - Posted: 11 Jan 2017, 8:10:41 UTC

The only answer I have found to this is to micro manage machines. I only do WCG work when no CPDN work is available and most of the time set it to no new tasks.
ID: 55465 · Report as offensive
bernard_ivo

Send message
Joined: 18 Jul 13
Posts: 438
Credit: 25,645,379
RAC: 1,966
Message 55466 - Posted: 11 Jan 2017, 8:58:55 UTC - in response to Message 55465.  

The only answer I have found to this is to micro manage machines. I only do WCG work when no CPDN work is available and most of the time set it to no new tasks.


I can't do that either as new work becomes visible after few hours when it is already gone. And real time server status would be nice. But if the idea is not micro management, then maybe shorter deadlines is one way to go, server side cleaning the obsolete WUs with "in progress" status (I have 5 and crunch 0), and steadier WU supply.
ID: 55466 · Report as offensive
ProfileDave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4541
Credit: 19,039,635
RAC: 18,944
Message 55467 - Posted: 11 Jan 2017, 9:46:35 UTC - in response to Message 55466.  

I used to do server updates as soon as I saw work appear. - If I see one machine get some, I then update the other two. I also abort unstarted tasks from WCG when I know there is CPDN work available. But as you say, micro managing is not as easy as it was. In one sense this is fairer as it means all have an equal chance of getting work. On the other hand, it also means that those who are active and flag issues quickly so the team at Oxford can get on to them may not get any work to recognise the problems.
ID: 55467 · Report as offensive
Grahamt

Send message
Joined: 23 Jul 10
Posts: 9
Credit: 2,099,795
RAC: 0
Message 55469 - Posted: 11 Jan 2017, 10:02:39 UTC

I run CPDN because I wanted to aid a project concerned with useful 'real-world' research to complement the more speculative work done by SETI@home. However, from the server status page and this discussion, it looks like a) there are now far too many users/computers for the work available and b) the scientists aren't communicating well with CPDN users.

Also, the way CPDN interacts with other projects managed by BOINC is unsatisfactory, as detailed in this thread. So I think it's time to find a different project to support. Just one thing will keep CPDN in my list of projects - the nonsense spouted by Trump & co. If the current workunit famine turns to feast again, I'll suspend all but one workunit so that my other three CPU cores can be used for other projects.
ID: 55469 · Report as offensive
ProfileAlan K

Send message
Joined: 22 Feb 06
Posts: 491
Credit: 31,247,913
RAC: 15,591
Message 55470 - Posted: 11 Jan 2017, 10:09:32 UTC - in response to Message 55464.  

Same here. I set WCG to no new tasks when there is CPDN work around only picking up again when we have nothing to crunch.
ID: 55470 · Report as offensive
ProfileDave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4541
Credit: 19,039,635
RAC: 18,944
Message 55471 - Posted: 11 Jan 2017, 10:13:09 UTC - in response to Message 55469.  

It used to be that all the work came from scientists based at Oxford. Now the Oxford team send out tasks on behalf of researchers from universities all over the world and don't do the actual research themselves. This means that they don't know about what work is coming up in advance. (They may get hints from any testing work that is done but that is all.) In theory this means that several researchers could all submit work on the same day giving everyone some work and then none be submitted for several weeks. I don't know if any of the scientists currently submitting work look at these forums. Perhaps the team at Oxford could suggest that they do? (Assuming that they currently do not.)
ID: 55471 · Report as offensive
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 55472 - Posted: 11 Jan 2017, 10:45:45 UTC

I doubt that the scientists look at anything BOINC. (Or read The Farmer's Almanac either.)
The project exists to provide computing capacity to the scientists. Once the data is created and sent back to their servers, there's no further contact with it. They can analysis it in whatever way they see fit.
And by the same reasoning, they are under no obligation to provide a steady flow of work.

Also, I think that the Server Status page is updating every 4 hours.

I'm not sure, but perhaps the big change in how BOINC works that happened a few years back, may have altered the way it handles the long cpdn times compared to the much shorter times of most other projects.
When I suggested a shorter time frame a year or so back, it didn't get anywhere.
The project has their own reasons for doing things in certain ways, and while I too think a much shorter time would help, I'm not inclined to ask again at the moment.
ID: 55472 · Report as offensive
bernard_ivo

Send message
Joined: 18 Jul 13
Posts: 438
Credit: 25,645,379
RAC: 1,966
Message 55473 - Posted: 11 Jan 2017, 11:47:17 UTC - in response to Message 55472.  

Nice discussion, but I think it should lead somewhere, as some of the things we all know and has been there for a while. CPDN is too important, but it may need to keep up with the changing environment.

Can we discuss what is good from our side, and what is possible/doable from the project side, and try to bring some message up? I guess with the recent upgrade there might be some reluctance, but for the time we as community can discuss and formulate few things (if we agree we should move the discussion to the Wish List or set up a new dedicated thread).
ID: 55473 · Report as offensive
ProfileDave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4541
Credit: 19,039,635
RAC: 18,944
Message 55474 - Posted: 11 Jan 2017, 12:21:42 UTC

Was about to update all machines having picked up a task on this one but on closer examination it is a pnw task on its last chance, so recent but not new work!
ID: 55474 · Report as offensive
Jim1348

Send message
Joined: 15 Jan 06
Posts: 637
Credit: 26,751,529
RAC: 653
Message 55477 - Posted: 11 Jan 2017, 15:10:53 UTC - in response to Message 55464.  

Boinc seems to think that just because the deadline is 11 months it is alright to take 11 month to finish the tasks. Either CPDN needs shorter deadlines, or other project need longer, multi-month deadlines.

A partial solution for me (other than I always use the default deadlines) is that I choose Rosetta as my other project. If you set that for the 24 hour run time, it works tolerably well with CPDN, but not perfectly.

However, CPDN needs shorter deadlines for other reasons as well. It is strange that we crunchers have to tell them the obvious, when it is in their own interest. There is something about academia (on various other projects too) that I find curious when you are trying to get something done.
ID: 55477 · Report as offensive
ProfileastroWX
Volunteer moderator

Send message
Joined: 5 Aug 04
Posts: 1496
Credit: 95,522,203
RAC: 0
Message 55478 - Posted: 11 Jan 2017, 21:05:26 UTC

For me, "Resource share" ratios set according to my desired project preference, coupled with "Run according to preferences" works reasonably well.

The reason for extended deadlines is linked to comparatively long run times for most CPDN work. (It was much worse when we always had plenty of work and tasks covered 160-year time spans.) People running other projects with CPDN complained that CPDN work hogged the resource. Hence the change to long deadlines. 40-year tasks are still relatively long in the "boinc project family."
"We have met the enemy and he is us." -- Pogo
Greetings from coastal Washington state, the scenic US Pacific Northwest.
ID: 55478 · Report as offensive
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 55482 - Posted: 13 Jan 2017, 13:05:29 UTC - in response to Message 55481.  

Everything is still the same, it's just that there's testing going on at the moment.
And far more computers, with lots of processors, than needed at present.
So those who need work need to keep a close eye on the queue, because when more tasks get released they won't last long.
ID: 55482 · Report as offensive
Profilegeophi
Volunteer moderator

Send message
Joined: 7 Aug 04
Posts: 2187
Credit: 64,822,615
RAC: 5,275
Message 55483 - Posted: 13 Jan 2017, 16:06:37 UTC - in response to Message 55481.  

As the dearth of new work gets longer I wonder whether this might be part of a trend.
Is there less climate research going on (unlikely, though research funding priorities are influenced by politics), or is it moving away from distributed computing?

There are a whole bunch of model types being tested, and others in the planning stage. How many work units for each type will be released to the main site is unknown by us moderators. But we often have a dearth of models to run around the holidays.

As for the politics, most of the model runs are for researchers in countries other than the U.S., so the imminent change in U.S. administrations shouldn't have much if anything to do with the amount of work here.
ID: 55483 · Report as offensive
DadX

Send message
Joined: 30 Aug 06
Posts: 27
Credit: 1,899,457
RAC: 1,339
Message 55484 - Posted: 13 Jan 2017, 18:23:14 UTC

To solve the prioritization problem with CPDN and other projects I use to run a Linux VM on Windows using VirtualBox. I have a 4 core value machine so I gave Linux one core, installed Boinc and used it to run CPDN. This in effect limited CPDN to one core and other projects running in Windows to 3. It did increase run times somewhat but I got the segregation I wanted and left reasonable performance for non-Boinc work. With models running under Linux being phased out of CPDN I no longer bother with this. You could run a Windows VM on a Windows host but that requires two Windows licenses which I don't wish to pay for.
ID: 55484 · Report as offensive
Profilegeophi
Volunteer moderator

Send message
Joined: 7 Aug 04
Posts: 2187
Credit: 64,822,615
RAC: 5,275
Message 55485 - Posted: 13 Jan 2017, 20:16:16 UTC - in response to Message 55484.  

With models running under Linux being phased out of CPDN I no longer bother with this. You could run a Windows VM on a Windows host but that requires two Windows licenses which I don't wish to pay for.

The models currently being beta tested are hacm3s and wah2_ri so both of those types should be able to run on linux at least.
ID: 55485 · Report as offensive
bernard_ivo

Send message
Joined: 18 Jul 13
Posts: 438
Credit: 25,645,379
RAC: 1,966
Message 55486 - Posted: 14 Jan 2017, 9:53:51 UTC - in response to Message 55485.  

The models currently being beta tested are hacm3s and wah2_ri so both of those types should be able to run on linux at least.


Is it possible some batches to be tested and released with shorter deadlines? Thus one can see how these behave in the crunchers environment nowadawys.
ID: 55486 · Report as offensive
ProfileDave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4541
Credit: 19,039,635
RAC: 18,944
Message 55487 - Posted: 14 Jan 2017, 17:50:52 UTC - in response to Message 55486.  

The models currently being beta tested are hacm3s and wah2_ri so both of those types should be able to run on linux at least.


Does that mean that the problem of the hadcm3s tasks not deleting their folder on completion in Linux has been resolved? I seem to remember at one point the linux version of these tasks was withdrawn because of that. I didn't find the deletion of the task folder too onerous but I can imagine the set and forget brigade could find themselves running out of space before too long if it is still happening. Haven't had any of the betas of them myself to find out in fact till current ones running under WINE are complete won't be getting any Linux work from beta or main site.
ID: 55487 · Report as offensive
Profilegeophi
Volunteer moderator

Send message
Joined: 7 Aug 04
Posts: 2187
Credit: 64,822,615
RAC: 5,275
Message 55488 - Posted: 14 Jan 2017, 18:33:19 UTC - in response to Message 55487.  

Yes. The folders for successful and failed tasks are cleaned up in Linux. I've run a bunch of 2 month tasks for testing without any problems or errors.

I then ran a 2 year hadcm3s and there was an error with that, but not the folder clearing problem that previously plagued the Linux versions.
ID: 55488 · Report as offensive
Previous · 1 · 2 · 3 · 4 · 5 · 6 · 7 . . . 91 · Next

Message boards : Number crunching : New work Discussion

©2024 cpdn.org