1) After boot and logon, but before the BOINC client starts

It might be our old friend the staggered start

If it really troubles people to see so many uploads building up in number we can modify it for these longer model runs (3 month forecasts).

On the problem machine, I have stopped Atlas-Native for the moment, rebootet and started a fresh OpenIFS. I will enshure that the VM is not restarted and monitor closely whats happening

Since the only file we're interested in is the progress_file_*.xml, I wonder if you could write a small shell script ...

You're right, this is one of the latest batches not the 950 batch I mentioned before. The model did work, as can be seen by the 'Uploading final trickle' message near the bottom. I have emailed CPDN about this, I'm hoping they can re-classify this as successful, give you credit and stop it going out again.

Did this task restart at all? Even once to your knowledge? (power off, boinc client restarted?). That's the only way the task can behave like this.

I think it's related to the memory in use. The total memory in the machine is not important, what is though is the memory limit you set in the boinc client. There are two values: 'When computer is in use' & 'when computer is not in use'. For example I have 35% and 75% respectively. It's quite possible the Atlas tasks are in contention with OpenIFS for memory available to boinc (not the total memory on the machine). Say you have 60% for 'when not in use', that means the total all the boinc tasks can use is 19Gb. If the Atlas tasks use ~16Gb, then OpeniFS only has ~3Gb to play with depending on what the Atlas tasks are doing. OpenIFS makes heavy use of dynamic memory and will hit a peak of ~4Gb. If that happens, then boinc will have to kick it out of memory because the client is not allowed to use enough, and that will cause the model to have to restart. I think this happens even if you have 'leave non-GPU in memory' as it's hit the client limit (I might be wrong on this).

I suspect something like this is happening and when OpenIFS is kicked out of memory, any files still being written (or flushed to disk) are maybe not written properly, get corrupted and you see this message. I've not seen many tasks do this so I think it's rare but the less headroom the boinc client has for all the tasks you want to run simultaneously, I suspect the more likely this is.

My advice would be to make sure that the boinc client has plenty of spare memory headroom set according to your client's memory preference. If each multicore Atlas task takes ~4Gb, OpenIFS ~4.5Gb max, the total boinc can use should match the number of tasks you would like to run concurrently, plus a bit spare. Hope that makes sense.

To find the memory limit for the OpenIFS task (or any task), look in the client_state.xml file in the main boinc directory (e.g. /var/lib/boinc) and find an openifs workunit, similar to this:

<workunit>
    <name>oifs_43r3_ps_0582_1995050100_123_964_12181226</name>
    <app_name>oifs_43r3_ps</app_name>
    <version_num>105</version_num>
    <rsc_fpops_est>1574400000000000.000000</rsc_fpops_est>
    <rsc_fpops_bound>15744000000000000.000000</rsc_fpops_bound>
    <rsc_memory_bound>6010000000.000000</rsc_memory_bound>    <<<<< memory upper limit required for the task in bytes
    <rsc_disk_bound>7516192768.000000</rsc_disk_bound>                 <<<<< disk upper limit required for the task in bytes

Do not change this file! The rsc_memory_bound is in bytes and gives the highest amount of memory required by the task (that's every process the task runs, not just the model which only uses about 4.5Gb peak). If you use that number, and the same from the Atlas workunits, you can work out what the total memory of all the tasks is and set the boinc memory bounds appropriately. You want to avoid the OpenIFS tasks being stopped & restarted due to insufficient memory as much as possible for efficiency (otherwise the model has to repeat some previous steps).

Hope that all makes sense. If you think it's not the memory limits, let me know. The more clues I get to figure out what's going on the better.

What kind of disk drive are you using for the slots directory that the model is running in? Is it a fast NMVe, SATA SSD, or slower HDD? May not be important but perhaps useful to know.

Technical explanation: there is a small file in the slot called 'progress_info_*.xml' with some large number on the end. If you look at it, you'll see this line:

  <upload_file_number>49</upload_file_number>

the task uses this file to keep track of which upload file number is next. The only time the task needs to read this file is when the task needs to restart after having already run a while. It reads the file to get the upload_file_number. However, if that file is not present, or its corrupt and can't be read, the code falls back to the first upload file again. That means when the model finishes, the boinc client thinks some upload files are missing. They're not, they were never used. It's a peculiarity of the boinc client that you have to declare all the upload files for the task before it starts (security I guess).

I'll look into this some more and log it as an issue to address in the near future in the code. Right now I'm working on more pressing ones.

Apologies for the long post. I deliberately added detail in the hope this message will be a reference for anyone else with similar problems.

Cheers, Glenn

As Dave has already said there are plans for a multi-core OpenIFS app (not the current Hadley Centre models). I have a working version of the app in testing but there have been some issues getting it to work correctly across different machines, so for the time being it's still in development.

Server can't find key file.