climateprediction.net home page
Linux/Mac/Windows segmentation

Linux/Mac/Windows segmentation

Message boards : Number crunching : Linux/Mac/Windows segmentation
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
Lockleys

Send message
Joined: 13 Jan 07
Posts: 195
Credit: 10,581,566
RAC: 0
Message 51761 - Posted: 4 Apr 2015, 14:34:57 UTC

When the incidence of models marked as Linux Only or Windows Only or any other combo first appeared, I assumed it was simply tactical to get some targeted crunching under way. But now that almost all models are unavailable to one group of crunchers or another, it begins to seem more entrenched.
Is it now becoming policy only to release models for one subgroup of users or another?
I'm sure there is good reasoning behind it but I haven't seen it explained.
ID: 51761 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4314
Credit: 16,378,503
RAC: 3,632
Message 51767 - Posted: 5 Apr 2015, 10:15:59 UTC

Not being a programmer other than having dabbled a long time ago I can't answer this fully. However, I can say that doing this will reduce the work for the programmers. There is work that needs to be done for each platform a batch is released for. If each batch only needs this to be done once rather than three times then clearly there are efficiency savings.

One possible downside is those who want to crunch primarily tasks concentrating on their own region. For myself I have preferences set for this but also have allow other model types if none of preferred type available ticked. Another is longer periods when one of more platforms are without work though that hasn't been a problem recently on Linux.

ID: 51767 · Report as offensive     Reply Quote
Eirik Redd

Send message
Joined: 31 Aug 04
Posts: 391
Credit: 219,888,554
RAC: 1,481,373
Message 51768 - Posted: 5 Apr 2015, 10:33:10 UTC - in response to Message 51761.  
Last modified: 5 Apr 2015, 10:34:42 UTC

I'm thinking the crew that's packaging the models for BOINC is totally overworked and overstressed and understaffed..

Nothing to do with user preferred client OS over another - aah - but - actually - each step away from the "zillion lines of Hadley FORTRAN code" costs - Money.
It seems obvious to me - that the path from "supercomputer" model to "distributed computing" model is -- [some kinda unix -> linux -> OSX -> MSWindows ]
Every conversion step takes time and money. Which in short supply.

So - packaging BOINC work-units for linux is cheap, but not many participants available. Packaging wu's for OSX and Windows costs more programmer time on the wrappers, and testing -- they payoff looks good but the up-front cost -- ??

So - right now - the huge backlog of "linux-only" - it's mostly because the CPDN hasn't got funding to package the wu's for the (marginally more expensive and much more available OS's)

My opinion only -- but think about it -- hope the project isn't starved for funds.









So the fast path for wrapping and packaging models is from whatever the

When the incidence of models marked as Linux Only or Windows Only or any other combo first appeared, I assumed it was simply tactical to get some targeted crunching under way. But now that almost all models are unavailable to one group of crunchers or another, it begins to seem more entrenched.
Is it now becoming policy only to release models for one subgroup of users or another?
I'm sure there is good reasoning behind it but I haven't seen it explained.

ID: 51768 · Report as offensive     Reply Quote
tullus

Send message
Joined: 16 May 13
Posts: 48
Credit: 475,901
RAC: 0
Message 51774 - Posted: 6 Apr 2015, 12:18:24 UTC - in response to Message 51768.  

As far as I know, part of the reason for the segmentation is that some apps failed, either consistently or with a high probability on the "wrong" operating system. Especially macs, which failed all version 7+ tasks.

There seems to be a huge drop in number of failed tasks after this change, so I would say it was/is a successful, although unfortunate, decision.
ID: 51774 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4314
Credit: 16,378,503
RAC: 3,632
Message 51775 - Posted: 6 Apr 2015, 18:58:14 UTC

Another reason is as Tullus suggests to make the models more robust and reduce the number of failed tasks.
ID: 51775 · Report as offensive     Reply Quote
Profile Hannah Rowlands

Send message
Joined: 30 Jan 14
Posts: 70
Credit: 60,900
RAC: 0
Message 51786 - Posted: 7 Apr 2015, 13:24:48 UTC

Hi,

I've just posted elsewhere on the forum about this (http://climateapps2.oerc.ox.ac.uk/cpdnboinc/forum_thread.php?id=8065#51785) - yes, we have decided as a policy to move to single operating systems for each application, which we hope will improve both the reliability and the science of the project.

Have a look at my forum post for more details.

Best wishes,
Hannah
Hannah Rowlands
--
No longer Communications Officer for climateprediction.net, as of October 2015
ID: 51786 · Report as offensive     Reply Quote
Lockleys

Send message
Joined: 13 Jan 07
Posts: 195
Credit: 10,581,566
RAC: 0
Message 51788 - Posted: 7 Apr 2015, 18:33:35 UTC - in response to Message 51786.  

Thanks Hannah for clarifying this. I understand the logic.
ID: 51788 · Report as offensive     Reply Quote
Eirik Redd

Send message
Joined: 31 Aug 04
Posts: 391
Credit: 219,888,554
RAC: 1,481,373
Message 51790 - Posted: 8 Apr 2015, 8:34:11 UTC

Thanks Hannah. Makes sense.
Hope more work for all of us coming soon.
ID: 51790 · Report as offensive     Reply Quote
tullus

Send message
Joined: 16 May 13
Posts: 48
Credit: 475,901
RAC: 0
Message 51794 - Posted: 9 Apr 2015, 14:47:41 UTC - in response to Message 51786.  

Thank you Hannah.

I was starting to wonder about the release of over 25 000 workunits for
Weather At Home Australia New Zealand region (hadam3p_anz) (Mac only)
but this app is now
Weather At Home Australia New Zealand region (hadam3p_anz) (Mac and Windows only)

"Mac only" would have meant an infinite supply for the next 100 years :)
ID: 51794 · Report as offensive     Reply Quote
Profile JIM

Send message
Joined: 31 Dec 07
Posts: 1152
Credit: 22,053,321
RAC: 4,417
Message 51795 - Posted: 9 Apr 2015, 15:16:48 UTC

I hope that the project developers will remember that while they may be more familiar with Linux that the real horsepower of this project is in Windows. The vast majority of computers that are attached run some form of that OS. If they want fast results Window is the way to go.

ID: 51795 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4314
Credit: 16,378,503
RAC: 3,632
Message 51796 - Posted: 9 Apr 2015, 15:23:11 UTC

"Mac only" would have meant an infinite supply for the next 100 years :)


I am noticing how slowly the current batch of Linux only models is going, though a lot more would be getting crunched if everyone had the 32bit libs installed!
ID: 51796 · Report as offensive     Reply Quote
Profile geophi
Volunteer moderator

Send message
Joined: 7 Aug 04
Posts: 2167
Credit: 64,403,322
RAC: 5,085
Message 51797 - Posted: 9 Apr 2015, 15:58:16 UTC - in response to Message 51796.  

I am noticing how slowly the current batch of Linux only models is going, though a lot more would be getting crunched if everyone had the 32bit libs installed!


Or, if there were 64bit versions of the apps. If they want lots of linux models crunched, that will likely be the only way. Too many crunchers just attach and don't even look to see if things go smoothly. Or they were successful with one version of Linux, upgrade to a later version for whatever good reason, and no longer can run the models with the default installation of that later distribution. It's the rare exception when they come here to check out the forums about problems.

ID: 51797 · Report as offensive     Reply Quote
MartinNZ

Send message
Joined: 22 Mar 06
Posts: 144
Credit: 24,695,428
RAC: 0
Message 51801 - Posted: 9 Apr 2015, 20:46:05 UTC - in response to Message 51796.  

Yeh, had noticed before on Tullus's graphs that the linux only tasks took ages to run down, but figured the researchers were happy enough.

It was an interesting exercise and brings another dimension to the age old OS debate.

Still, the break in work meant that I was able to give the insides of the PC a good and thorough sudsy jet wash, although I did have to get the sand blaster out to finally get the water cooler radiators clean. ;-)
ID: 51801 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4314
Credit: 16,378,503
RAC: 3,632
Message 51804 - Posted: 10 Apr 2015, 10:23:53 UTC

Too many crunchers just attach and don't even look to see if things go smoothly.


I guess that is a sign that Linux is coming of age. When I started with it, I quickly learned not to take anything for granted and getting anything working meant searching for things like drivers etc. Forums/bulletin boards were essential then and while not quite so essential now, I still haven't stopped looking at them!
ID: 51804 · Report as offensive     Reply Quote
Eirik Redd

Send message
Joined: 31 Aug 04
Posts: 391
Credit: 219,888,554
RAC: 1,481,373
Message 51838 - Posted: 16 Apr 2015, 8:58:20 UTC

I'm a Linux creature, and totally respect all other OS users. Actually number three on "top contributors" right now
And I'm hoping that the crew can get the next batches out to Windows hosts ASAP.
However, whatever.
When I look at the backlog (12000 or so) linux-only models, and know that my 34-core rig only completes 3 models per day. :(

When the crew gets a Windows-only batch ready -- time to recruit volunteers !!

And I don't mean anybody should be such an otaku as me - every model helps.

Keep on crunching.

ID: 51838 · Report as offensive     Reply Quote
MarkJ
Avatar

Send message
Joined: 28 Mar 09
Posts: 126
Credit: 9,825,980
RAC: 0
Message 51844 - Posted: 17 Apr 2015, 12:06:16 UTC

I wonder if it would be better for the project to create the work units as Vbox images, then they wouldn't need to target a specific OS and could have a base image that runs their preferred OS with the necessary data files and programs included.
BOINC blog
ID: 51844 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4314
Credit: 16,378,503
RAC: 3,632
Message 51845 - Posted: 17 Apr 2015, 14:00:36 UTC - in response to Message 51844.  

How does that work Mark? I and I suspect a lot of other crunchers have never used virtual boxes. What is the performance hit if any to crunching and other work particularly if running on minimum spec for memory?
ID: 51845 · Report as offensive     Reply Quote
Profile JIM

Send message
Joined: 31 Dec 07
Posts: 1152
Credit: 22,053,321
RAC: 4,417
Message 51846 - Posted: 17 Apr 2015, 15:03:30 UTC - in response to Message 51844.  

I wonder if it would be better for the project to create the work units as Vbox images, then they wouldn't need to target a specific OS and could have a base image that runs their preferred OS with the necessary data files and programs included.


This sounds like a good idea if it can be made to work. At the rate that those 12,000+ Linux only tasks are going down the Sun will burn out before they crunch them all.

ID: 51846 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4314
Credit: 16,378,503
RAC: 3,632
Message 51847 - Posted: 17 Apr 2015, 15:47:30 UTC - in response to Message 51846.  

If they had a 64 bit version or all the linux boxes had the 32 bit libs installed they would have gone by now.
ID: 51847 · Report as offensive     Reply Quote
Lockleys

Send message
Joined: 13 Jan 07
Posts: 195
Credit: 10,581,566
RAC: 0
Message 51848 - Posted: 17 Apr 2015, 18:25:26 UTC - in response to Message 51844.  

I did run models like this for the Large Hadron Collider for a while but I found that running their models like this was a very heavy load on the PC. Much heavier than CPDN. It made the PC noticeably less responsive for other work. And since running these models is supposed to be a background activity which just users the spare cycles left over when I have done my own real work and should be an invisible background activity, I stopped running them. The other SixTrack LHC models which don't run inside a virtual machine run just fine and so I still run those. Just not the Vbox ones.
ID: 51848 · Report as offensive     Reply Quote
1 · 2 · Next

Message boards : Number crunching : Linux/Mac/Windows segmentation

©2024 climateprediction.net