climateprediction.net home page
OpenIFS Discussion

OpenIFS Discussion

Message boards : Number crunching : OpenIFS Discussion
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 23 · 24 · 25 · 26 · 27 · 28 · 29 . . . 31 · Next

AuthorMessage
Glenn Carver

Send message
Joined: 29 Oct 17
Posts: 847
Credit: 13,960,536
RAC: 6,962
Message 68354 - Posted: 16 Feb 2023, 9:44:07 UTC

The idea with this batch of the OpenIFS BL tasks is to explore a wide range of parameters in the model that can be changed to look at storms in a future climate. When the results are back, they will be filtered to look for good candidates to run again but this time at a much higher model resolution. We're not sure yet whether boinc is a suitable framework for these higher resolution tasks (will need > 20Gb RAM, more output etc).
ID: 68354 · Report as offensive     Reply Quote
Yeti

Send message
Joined: 5 Aug 04
Posts: 172
Credit: 11,063,727
RAC: 28,803
Message 68355 - Posted: 16 Feb 2023, 9:57:50 UTC - in response to Message 68354.  

We're not sure yet whether boinc is a suitable framework for these higher resolution tasks (will need > 20Gb RAM, more output etc).
If you would enable Application-Selection for the User you could make it a different Application and only Users that have Opted-In should get them
Supporting BOINC, a great concept !
ID: 68355 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4365
Credit: 16,614,059
RAC: 1,518
Message 68356 - Posted: 16 Feb 2023, 10:33:36 UTC - in response to Message 68355.  

We're not sure yet whether boinc is a suitable framework for these higher resolution tasks (will need > 20Gb RAM, more output etc).
If you would enable Application-Selection for the User you could make it a different Application and only Users that have Opted-In should get them
I agree. Application selection for the user used to be available. I think it was stopped around the same time that model types were only available for single platforms but the way OIFS is developing to my mind at least makes its reintroduction a good idea.
ID: 68356 · Report as offensive     Reply Quote
Glenn Carver

Send message
Joined: 29 Oct 17
Posts: 847
Credit: 13,960,536
RAC: 6,962
Message 68357 - Posted: 16 Feb 2023, 12:36:19 UTC - in response to Message 68356.  
Last modified: 16 Feb 2023, 12:57:49 UTC

We're not sure yet whether boinc is a suitable framework for these higher resolution tasks (will need > 20Gb RAM, more output etc).
If you would enable Application-Selection for the User you could make it a different Application and only Users that have Opted-In should get them
I agree. Application selection for the user used to be available. I think it was stopped around the same time that model types were only available for single platforms but the way OIFS is developing to my mind at least makes its reintroduction a good idea.
Yes, that's how we're doing it. Each high res configuration will be done as a separate app so it can be controlled differently on the server & user's app_config.xml, but also selected on/off on the Project Preferences page. I suspect by default we'll have these apps deselected so it's an 'opt-in'. It's an identical OpenIFS binary, the model gets its configuration from the input files. This will be setup on the dev site first.
---
CPDN Visiting Scientist
ID: 68357 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4365
Credit: 16,614,059
RAC: 1,518
Message 68358 - Posted: 16 Feb 2023, 13:44:49 UTC
Last modified: 16 Feb 2023, 13:49:13 UTC

I see this one of mine ( the _2 so third attempt) crashed with the overflow message at the same point on all three attempts so good to see the consistent result across all three computers. (All from the Ryzen stable.
Work unit

This one That I got for second attempt after first computer had problems with the upload at the end, finished normally.
ID: 68358 · Report as offensive     Reply Quote
Vato

Send message
Joined: 4 Oct 19
Posts: 13
Credit: 7,353,923
RAC: 3,385
Message 68359 - Posted: 16 Feb 2023, 13:46:28 UTC - in response to Message 68357.  

this approach works well at other projects, and i heartily support this
ID: 68359 · Report as offensive     Reply Quote
Glenn Carver

Send message
Joined: 29 Oct 17
Posts: 847
Credit: 13,960,536
RAC: 6,962
Message 68361 - Posted: 16 Feb 2023, 14:11:18 UTC

Is anyone interested in running a standalone test of the higher resolution OpenIFS configurations we're planning on rolling out? It's the same executable that's gone out before but with a different set of input files. It would not run under the client, a short simple linux script is used to start the model to run 1 forecast day with a small amount of output data. There is NO CREDIT for this - it's just if anyone is interested to see what the model will do on their system.

The current batches use:
oifs_43r3: Global 125km grid resolution (max virt mem 6Gb / max resident : 5.2Gb) (memory use does vary between systems).

The new configurations would be (single core):
oifs_43r3_l255 : Global 75km grid resolution (max vmem 20Gb / max res 18Gb )
oifs_43r3_l319 : Global 60km grid resolution (max vmem 32Gb / max res 29Gb)

Please note the runtime memory requirements, which is why it would be useful to get some feedback before we decide exactly what to use. Do not attempt to run on machines with less memory than stated above, it will cripple your system.

Send Personal Message to me if interested rather than reply here. If there is sufficient interest, I'll share the files on dropbox. I'll post answers to PM'd questions here.
---
CPDN Visiting Scientist
ID: 68361 · Report as offensive     Reply Quote
AndreyOR

Send message
Joined: 12 Apr 21
Posts: 247
Credit: 12,074,507
RAC: 2,078
Message 68362 - Posted: 16 Feb 2023, 21:14:11 UTC - in response to Message 68357.  

We're not sure yet whether boinc is a suitable framework for these higher resolution tasks (will need > 20Gb RAM, more output etc).
If you would enable Application-Selection for the User you could make it a different Application and only Users that have Opted-In should get them
I agree. Application selection for the user used to be available. I think it was stopped around the same time that model types were only available for single platforms but the way OIFS is developing to my mind at least makes its reintroduction a good idea.
Yes, that's how we're doing it. Each high res configuration will be done as a separate app so it can be controlled differently on the server & user's app_config.xml, but also selected on/off on the Project Preferences page. I suspect by default we'll have these apps deselected so it's an 'opt-in'. It's an identical OpenIFS binary, the model gets its configuration from the input files. This will be setup on the dev site first.

I also agree that this should be reinstated as well as be an 'opt-in'. I think the only models that should be automatically opted-in are Windows and current, lowest resolution OIFS. All others require or will require either special configurations (32-bit libraries) or hardware (RAM, older Mac).
ID: 68362 · Report as offensive     Reply Quote
Yeti

Send message
Joined: 5 Aug 04
Posts: 172
Credit: 11,063,727
RAC: 28,803
Message 68363 - Posted: 16 Feb 2023, 21:18:55 UTC - in response to Message 68362.  

I also agree that this should be reinstated as well as be an 'opt-in'. I think the only models that should be automatically opted-in are Windows and current, lowest resolution OIFS. All others require or will require either special configurations (32-bit libraries) or hardware (RAM, older Mac).

Perhaps you can place the Infos about needed RAM / HD-Space / Libraries / ... there. Then everybody who wants to Opt-In will have seen these Details. A good example for this is Primegrid, the site of the project-Preferences is very informativ
Supporting BOINC, a great concept !
ID: 68363 · Report as offensive     Reply Quote
computezrmle

Send message
Joined: 9 Mar 22
Posts: 30
Credit: 1,057,332
RAC: 6,606
Message 68364 - Posted: 17 Feb 2023, 14:24:15 UTC

As far as a simple RAM monitoring shows oifs tasks seem to have a large but (most of the time) stable RAM requirement while a step is in progress.
At the beginning of each step there's a small additional peak and at the end of a step lots of RAM is released for a short time until the next step starts.

That short release might be responsible for BOINC client's intermediate miscalculation regarding the RAM estimation and finally lead to an overcommitment.

To make BOINC more stable the idea would be to check within oifs whether the maximum RAM requirement can be estimated before the task allocates it from the heap.
It might then be possible to reuse the same RAM until all steps are processed.
ID: 68364 · Report as offensive     Reply Quote
Jean-David Beyer

Send message
Joined: 5 Aug 04
Posts: 1083
Credit: 16,671,999
RAC: 5,277
Message 68365 - Posted: 17 Feb 2023, 16:15:03 UTC - in response to Message 68364.  
Last modified: 17 Feb 2023, 16:15:53 UTC

To make BOINC more stable the idea would be to check within oifs whether the maximum RAM requirement can be estimated before the task allocates it from the heap.
It might then be possible to reuse the same RAM until all steps are processed.


I do not know how the Oifs task allocates and frees its RAM, but if it uses the malloc, free, calloc, realloc suite of functions, this can be fairly easily arranged (at least, in principle). See the section 3 of the manual pages for how to do that. Basically, you allocate the amount of storage you think you need, with malloc(). If you need more, you can get it (if it is available) with ralloc(). This may have already been done in Oifs.

Generally, it is advised to Avoid using brk() and sbrk(): the malloc(3) memory allocation package is the portable and comfortable way of allocating memory.
ID: 68365 · Report as offensive     Reply Quote
Jean-David Beyer

Send message
Joined: 5 Aug 04
Posts: 1083
Credit: 16,671,999
RAC: 5,277
Message 68369 - Posted: 18 Feb 2023, 5:50:03 UTC - in response to Message 68361.  

Send Personal Message to me if interested rather than reply here.


User Glenn Carver (ID: 1560856) is not accepting private messages from you
ID: 68369 · Report as offensive     Reply Quote
gemini8

Send message
Joined: 4 Dec 15
Posts: 52
Credit: 2,182,959
RAC: 836
Message 68378 - Posted: 19 Feb 2023, 21:05:10 UTC

Usually there's a feature in the Boinc Server software to send beta-tasks to users via the client if those users have opted-in to receive those tasks.
I would have done so btw. if the CPDN preferences had supported this. ;-)
- - - - - - - - - -
Greetings, Jens
ID: 68378 · Report as offensive     Reply Quote
AndreyOR

Send message
Joined: 12 Apr 21
Posts: 247
Credit: 12,074,507
RAC: 2,078
Message 68382 - Posted: 20 Feb 2023, 1:32:28 UTC - in response to Message 68378.  

Usually there's a feature in the Boinc Server software to send beta-tasks to users via the client if those users have opted-in to receive those tasks.
I would have done so btw. if the CPDN preferences had supported this. ;-)

That's right and a number of projects have it, you also get some credit for it. Since CPDN already has a development site, it's unlikely they're going to change their system when it comes to beta testing.
ID: 68382 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4365
Credit: 16,614,059
RAC: 1,518
Message 68383 - Posted: 20 Feb 2023, 7:51:00 UTC - in response to Message 68382.  

Usually there's a feature in the Boinc Server software to send beta-tasks to users via the client if those users have opted-in to receive those tasks.
I would have done so btw. if the CPDN preferences had supported this. ;-)

That's right and a number of projects have it, you also get some credit for it. Since CPDN already has a development site, it's unlikely they're going to change their system when it comes to beta testing.
Credit for tasks on testing site for cpdn shows up as a separate project.
ID: 68383 · Report as offensive     Reply Quote
Profile Dave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4365
Credit: 16,614,059
RAC: 1,518
Message 68385 - Posted: 20 Feb 2023, 13:10:56 UTC - in response to Message 68383.  

Usually there's a feature in the Boinc Server software to send beta-tasks to users via the client if those users have opted-in to receive those tasks.
I would have done so btw. if the CPDN preferences had supported this. ;-)

That's right and a number of projects have it, you also get some credit for it. Since CPDN already has a development site, it's unlikely they're going to change their system when it comes to beta testing.
Credit for tasks on testing site for cpdn shows up as a separate project.


However as Glen has said, the test he wants to run is via a script to run in a terminal and not through BOINC at all.
ID: 68385 · Report as offensive     Reply Quote
Richard Haselgrove

Send message
Joined: 1 Jan 07
Posts: 947
Credit: 34,468,484
RAC: 4,162
Message 68387 - Posted: 20 Feb 2023, 16:37:57 UTC
Last modified: 20 Feb 2023, 16:41:10 UTC

First Africa task from batch 992 failed - Task 22314299

oifs_43r3_0016_2019110100_123_992_12213329_0

ABORT! 1 SUECOZC:ERROR OPENING FILE ECOZC
forrtl: error (76): Abort trap signal
..The child process has been killed with signal: 6
oifs_get_stat: Error. ifs.stat file is not open

I'll leave the rest to the experts.
ID: 68387 · Report as offensive     Reply Quote
Vato

Send message
Joined: 4 Oct 19
Posts: 13
Credit: 7,353,923
RAC: 3,385
Message 68390 - Posted: 20 Feb 2023, 16:56:54 UTC - in response to Message 68387.  

all mine on 2 machines give much the same
ID: 68390 · Report as offensive     Reply Quote
Richard Haselgrove

Send message
Joined: 1 Jan 07
Posts: 947
Credit: 34,468,484
RAC: 4,162
Message 68391 - Posted: 20 Feb 2023, 17:15:33 UTC

Distribution of this batch has been paused while the errors are investigated.
ID: 68391 · Report as offensive     Reply Quote
Jean-David Beyer

Send message
Joined: 5 Aug 04
Posts: 1083
Credit: 16,671,999
RAC: 5,277
Message 68392 - Posted: 20 Feb 2023, 17:39:36 UTC - in response to Message 68386.  

We use the large ensemble of the OpenIFS@Home to identify the connections between the tropical Indian Ocean and the North Atlantic region.

Technical information

CPDN app-name: oifs_43r3


Is that the name of the tasks I use in app_config.xml?

Like this?
app_config>
    <project_max_concurrent>6</project_max_concurrent>
    <app>
        <name>oifs_43r3</name>   <---<<<
        <max_concurrent>1</max_concurrent>
        </app>
   <app>
        <name>oifs_43r3_bl</name>
        <max_concurrent>5</max_concurrent>
        </app>
   <app>
        <name>oifs_43r3_ps</name>
        <max_concurrent>4</max_concurrent>
        </app>
    <app>
        <name>oifs_43r3_l255</name>
        <max_concurrent>1</max_concurrent>
        </app>
    <app>
        <name>oifs_43r3_l319</name>
        <max_concurrent>1</max_concurrent>
        </app>
    </app_config>



I am not accepting new tasks at the moment. My additional 64 GBytes of RAM are due in tomorrow sometime.
ID: 68392 · Report as offensive     Reply Quote
Previous · 1 . . . 23 · 24 · 25 · 26 · 27 · 28 · 29 . . . 31 · Next

Message boards : Number crunching : OpenIFS Discussion

©2024 climateprediction.net