 |
|
|
21)
Message boards :
Number crunching :
OpenIFS tasks : make sure boinc client option 'Leave non-GPU tasks in memory' is selected!
(Message 67352)
Posted 5 Jan 2023 by  PDW
Post: If it isn't checkpointing when I wake up I'll know that bit has worked and reduce the figure to induce checkpointing.It was checkpointing. So yes, BOINC needs fixing. |
|
22)
Message boards :
Number crunching :
OpenIFS tasks : make sure boinc client option 'Leave non-GPU tasks in memory' is selected!
(Message 67311)
Posted 4 Jan 2023 by PDW
Post: BOINC doesn't control checkpoints, except in the sense of setting a minimum interval between checkpoints - default 60 seconds. And that's a global setting - it doesn't make any difference whether it's a 3-minute GPU task for WCG, or a 14-hour CPDN task. But it does observe what it is supposed to do with them. boinc_time_to_checkpoint returns true only when sufficient time has passed since the last checkpoint. This minimum interval is the maximum of: A user preference (e.g. laptop users might want to checkpoint infrequently). An optional application-supplied, specified by calling boinc_set_min_checkpoint_period(int nsecs); So if the wrapper/application calls boinc_set_min_checkpoint_period() with a number > the longest amount of time it would expect to take then the BOINC code shouldn't try to PS. Apart from the ignorant, who leaves the default as 60 seconds ! |
|
23)
Message boards :
Number crunching :
OpenIFS tasks : make sure boinc client option 'Leave non-GPU tasks in memory' is selected!
(Message 67306)
Posted 4 Jan 2023 by PDW
Post: I was trying to work that out. In BOINC's case, 'checkpointed' should mean "BOINC has successefully written the files needed for a restart at ... [time]' I would hazard a guess that BOINC respects the initial "Request tasks to checkpoint at most every X seconds" set in the client but after that time has elapsed and because the checkpoint time is still showing zero it will then repeat the request every second as that code gets called every second (all being well). My next task isn't due to start until about 2am so have set the checkpoint time period to be 10 hours. If it isn't checkpointing when I wake up I'll know that bit has worked and reduce the figure to induce checkpointing. I thought the task/project could also specify a time period for checkpointing ? I know I have seen it being set inside LHC vboxwrapper code. If the project isn't using them internally can it not set a massively high period for their tasks to follow ? |
|
24)
Message boards :
Number crunching :
The uploads are stuck
(Message 67243)
Posted 3 Jan 2023 by PDW
Post: Messages flying about but no identification of exactly what the problem is yet. Andy thinks Jasmin have changed something in their infrastructure but not sure what. What is clear is that those involved are working on it and not sitting on their hands. (With the possible exception of those managing jasmin which I assume is the successor to Joint Academic Network. I never did work out what the rest of the new acronym is. The Joint Analysis System Meeting Infrastructure Needs |
|
25)
Message boards :
Number crunching :
OpenIFS Discussion
(Message 67026)
Posted 24 Dec 2022 by PDW
Post: Task will get full credit. Glen did post an explanation about tasks that finish successfully but appear to fail a few days ago. I will see if I can find it later. From what I recall, I didn't read it carefully enough to fully understand it. Glenn said this in post https://www.cpdn.org/forum_thread.php?id=9162&postid=66949#66949 : Agreed. I've asked CPDN if there is a way of getting the server to check the upload was received OK to reclassify this as a success. It may not be easy as the uploads go to a cloud server first. Not my expertise. |
|
26)
Message boards :
Number crunching :
OpenIFS Discussion
(Message 66993)
Posted 21 Dec 2022 by PDW
Post: Richard, perhaps there's a misunderstanding. CPDN use 'trickles' to upload model results. Unpack the trickle zip files and it will contain the model output files from the previous steps since the last trickle upload. The final upload file just contains the last remaining model outputs. I don't know how other projects use trickles but they are not administrative fragments, they contain actual model (scientific) data. The trickle files (named as trickle files) and the zip files are being treated differently. The zip files appear in the Transfer tab in BOINC Manager whereas the trickle files do not. The zip files are large and contain model data, the trickle files are tiny and look like this... <variety>orig</variety> <wu>oifs_43r3_ps_1351_2021050100_123_946_12164440</wu> <result>oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001</result> <ph></ph> <ts>10368000</ts> <cp>60877</cp> <vr></vr> |
|
27)
Message boards :
Number crunching :
OpenIFS Discussion
(Message 66984)
Posted 21 Dec 2022 by PDW
Post: I don't understand why you suspended it a few times? It would have worked without? The trickle file is quite small. I wanted to be around for when it finished to see what happened, part of that was to give the pending trickle a chance to go. I was under the impression we were thinking that the trickles failing to upload was a cause for concern and might be why some valid work was being marked invalid. I've now seen tasks fail and work with trickle files present after the tasks have completed. |
|
28)
Message boards :
Number crunching :
OpenIFS Discussion
(Message 66980)
Posted 20 Dec 2022 by PDW
Post: New day, new task ! This one had 90 minutes to run but would finish when I was out, it also had a trickle file that had been sitting there for 20 minutes or so. I suspended the task, the event log shows this and when I resumed it when I got back in (the trickle file had disappeared). Expecting the final trickle file to also take a while to go I clicked on Suspend task as soon as that trickle file appeared after the last zip file. The status of the task changed to suspended but I'm aware of what the critical_section code does on these occasions. After a few minutes, probably the 2 minutes sleep in the wrapper, the status of the task changed to Completed even though the final trickle file was still present in the project folder. A few more minutes passed before I clicked on Update. The task disappeared from the list of tasks, the trickle file got deleted, the task is showing as completed in my list of tasks on the server ! |
|
29)
Message boards :
Number crunching :
OpenIFS Discussion
(Message 66969)
Posted 19 Dec 2022 by PDW
Post: Just had another task fail with status 9 but also had "free(): invalid pointer" at the end of the stderr file... Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_122.zip Uploading the final file: upload_file_122.zip Uploading trickle at timestep: 10623600 15:13:12 (1519724): called boinc_finish(0) free(): invalid pointer At the end of the run there were 2 trickle files present in the project directory, the earliest one from 14:43. The latter one appeared very near the end of the run. -rw-r--r-- 1 boinc boinc 209 Dec 19 15:11 trickle_up_oifs_43r3_ps_1351_2021050100_123_946_12164440_2_1671462672.xml -rw-r--r-- 1 boinc boinc 25518410 Dec 19 15:10 oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_122.zip -rw-r--r-- 1 boinc boinc 209 Dec 19 14:43 trickle_up_oifs_43r3_ps_1351_2021050100_123_946_12164440_2_1671461035.xml Eventually the task completed in Boinc Manager with a computation error and the 2 trickles were still sitting in the directory, the zip file had gone. The task didn't report back to the server so after a few minutes I manually Updated and the task disappeared from the Manager. Looking in the project folder I saw that the 2 trickle files had also gone. The event log showed that the trickles and job files (jf_*) had been removed, it also said it removed all the zip files though they weren't present, none of the.gz or .gzt files were present (for zip or job files). Note: trickle names in event log end with ".sent" but that suffix was not seeen in the project directory, the .xml files were deleted ! Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/trickle_up_oifs_43r3_ps_1351_2021050100_123_946_12164440_2_1671461035.xml.sent Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/trickle_up_oifs_43r3_ps_1351_2021050100_123_946_12164440_2_1671462672.xml.sent Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/jf_1c36b977f0a0e944c1bdda1a7ee5396c Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/jf_1c36b977f0a0e944c1bdda1a7ee5396c.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/jf_1c36b977f0a0e944c1bdda1a7ee5396c.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/jf_475c23e3be91c1b31667a48915dcc36c Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/jf_475c23e3be91c1b31667a48915dcc36c.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/jf_475c23e3be91c1b31667a48915dcc36c.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/jf_ab0d2288d1c874c01a43253fbe058b4e Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/jf_ab0d2288d1c874c01a43253fbe058b4e.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/jf_ab0d2288d1c874c01a43253fbe058b4e.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_0.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_0.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_0.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_1.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_1.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_1.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_2.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_2.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_2.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_3.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_3.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_3.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_4.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_4.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_4.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_5.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_5.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_5.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_6.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_6.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_6.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_7.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_7.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_7.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_8.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_8.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_8.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_9.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_9.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_9.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_10.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_10.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_10.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_11.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_11.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_11.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_12.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_12.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_12.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_13.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_13.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_13.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_14.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_14.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_14.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_15.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_15.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_15.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_16.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_16.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_16.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_17.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_17.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_17.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_18.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_18.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_18.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_19.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_19.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_19.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_20.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_20.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_20.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_21.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_21.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_21.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_22.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_22.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_22.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_23.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_23.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_23.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_24.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_24.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_24.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_25.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_25.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_25.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_26.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_26.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_26.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_27.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_27.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_27.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_28.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_28.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_28.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_29.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_29.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_29.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_30.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_30.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_30.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_31.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_31.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_31.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_32.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_32.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_32.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_33.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_33.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_33.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_34.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_34.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_34.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_35.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_35.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_35.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_36.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_36.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_36.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_37.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_37.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_37.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_38.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_38.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_38.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_39.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_39.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_39.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_40.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_40.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_40.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_41.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_41.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_41.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_42.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_42.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_42.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_43.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_43.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_43.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_44.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_44.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_44.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_45.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_45.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_45.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_46.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_46.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_46.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_47.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_47.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_47.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_48.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_48.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_48.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_49.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_49.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_49.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_50.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_50.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_50.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_51.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_51.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_51.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_52.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_52.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_52.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_53.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_53.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_53.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_54.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_54.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_54.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_55.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_55.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_55.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_56.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_56.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_56.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_57.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_57.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_57.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_58.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_58.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_58.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_59.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_59.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_59.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_60.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_60.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_60.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_61.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_61.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_61.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_62.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_62.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_62.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_63.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_63.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_63.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_64.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_64.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_64.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_65.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_65.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_65.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_66.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_66.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_66.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_67.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_67.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_67.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_68.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_68.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_68.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_69.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_69.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_69.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_70.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_70.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_70.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_71.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_71.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_71.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_72.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_72.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_72.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_73.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_73.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_73.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_74.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_74.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_74.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_75.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_75.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_75.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_76.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_76.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_76.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_77.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_77.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_77.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_78.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_78.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_78.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_79.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_79.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_79.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_80.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_80.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_80.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_81.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_81.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_81.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_82.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_82.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_82.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_83.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_83.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_83.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_84.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_84.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_84.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_85.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_85.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_85.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_86.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_86.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_86.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_87.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_87.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_87.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_88.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_88.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_88.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_89.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_89.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_89.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_90.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_90.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_90.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_91.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_91.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_91.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_92.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_92.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_92.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_93.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_93.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_93.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_94.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_94.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_94.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_95.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_95.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_95.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_96.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_96.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_96.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_97.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_97.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_97.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_98.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_98.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_98.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_99.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_99.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_99.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_100.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_100.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_100.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_101.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_101.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_101.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_102.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_102.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_102.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_103.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_103.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_103.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_104.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_104.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_104.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_105.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_105.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_105.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_106.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_106.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_106.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_107.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_107.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_107.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_108.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_108.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_108.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_109.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_109.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_109.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_110.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_110.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_110.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_111.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_111.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_111.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_112.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_112.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_112.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_113.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_113.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_113.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_114.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_114.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_114.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_115.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_115.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_115.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_116.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_116.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_116.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_117.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_117.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_117.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_118.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_118.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_118.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_119.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_119.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_119.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_120.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_120.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_120.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_121.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_121.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_121.zip.gzt Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_122.zip Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_122.zip.gz Mon 19 Dec 2022 15:15:07 GMT | | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_122.zip.gzt |
|
30)
Message boards :
Number crunching :
OpenIFS Discussion
(Message 66962)
Posted 18 Dec 2022 by PDW
Post: Have now read the BasicApi wiki about boinc_end_critical_section :) One more thing that doesn't seem right (to me !), there is code to delete the files, from the project sub-directory, that get added to the zip. I'm assuming only the files listed in the stderr log are being added to the zip file ? There is also a progress file kept in there (something like progress_task-id.xml), if that isn't added to the zip and therefore not deleted in that process does that mean when the directory is removed it creates an error ? std::remove is expecting an empty directory, but I've no idea if it would still delete the directory and generate a 9 error, the directory does not remain for a 9 task. I can see the progress file being created but never deleted, unless it is included in the zip file. The file isn't in the sub-directory all the time. The only explanation I can think of is that it is zipped and hence deleted but not shown in the stderr log as being added. Or have I missed something ! |
|
31)
Message boards :
Number crunching :
OpenIFS Discussion
(Message 66959)
Posted 18 Dec 2022 by PDW
Post: This bit...?
fprintf(stderr,"Uploading the final file: %s\n",upload_file_name.c_str());
sleep_until(system_clock::now() + seconds(20));
boinc_upload_file(upload_file_name);
retval = boinc_upload_status(upload_file_name);
if (!retval) {
fprintf(stderr,"Finished the upload of the final file\n");
}
// Produce trickle
process_trickle(current_cpu_time,wu_name.c_str(),result_base_name,slot_path,current_iter);
}
boinc_end_critical_section();
[other skipped code]
// Now task has finished, remove the temp folder
std::remove(temp_path.c_str());
sleep_until(system_clock::now() + seconds(120));
// if finished normally
if (process_status == 1){
boinc_end_critical_section();
boinc_finish(0);
Never see trickles so no idea how long they can take. Also 2 boinc_end_critical_section's are performed ? [No idea what it does yet] Understand I'm looking at old code. Glenn, meant to say thanks for your earlier detailed explanation. |
|
32)
Message boards :
Number crunching :
OpenIFS Discussion
(Message 66956)
Posted 18 Dec 2022 by PDW
Post: 1) The last child process running the model completes 2) The final set of output files in the task directory below the project directory get zipped up, an intermediate 8 character file is created (begins with 'zi') in the project directory that grows and gets renamed to be the final ZIP file (very-long-name_14.zip) 3) The event log shows the upload starting and after a little while the upload being completed successfully 4) The event log shows a [slot] debug message that the very-long-name_14.zip has been removed 5) 2 more lines in the event log show [slot] debug messages that very-long-name_14.zip.gz and very-long-name_14.zip.gzt are also removed - neither of these two files are present in the project directory. There is no error message to say that these files couldn't be found to be deleted in the event log. 6) The slot files themselves are then removed, each with a debug message, about 1,950 of them. 7) The slot itself is deleted. Is the absence of the .zip.gz and .zip.gzt files the reason an error 9 is raised ? I can probably go and look to see if a working (not got a 9) task had those as well... Edit: Don't think it is that. After every upload the [slot] debugs shows it trying to delete all 3 types of .zip file. Have only witnessed one final .zip file being created and not the other 2 files at the same time, but probably happens for all zips for all tasks (0 or 9). |
|
33)
Message boards :
Number crunching :
OpenIFS Discussion
(Message 66938)
Posted 17 Dec 2022 by PDW
Post: In case it hasn't been mentioned previously: if an oifs task has bugged out part way through then a directory for that task may be left in the project folder that you may want to manually delete to reclaim the disk space. Another task failed with error 9 today. The host had previously run 7 of the last batch successfully and 1 more since without a problem. The host has been running the same projects continuously with 8 - 11 GB free memory according to top, with roughly 70GB of disk space available. I believe a wrapper is being used to run the oifs tasks, is that correct ? Regarding the entries that are put into the stderr.txt file whilst the 'task' is running, is that just by the actual real application, by the wrapper, or both ? These entries for example: 04:51:49 STEP 1440 H= 360:00 +CPU= 25.486 The child process terminated with status: 0 Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGh7zg+001440 Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHh7zg+001440 Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAh7zg+001440 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166594/ICMSHh7zg+001392 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166594/ICMUAh7zg+001416 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166594/ICMUAh7zg+001404 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166594/ICMSHh7zg+001416 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166594/ICMSHh7zg+001344 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166594/ICMGGh7zg+001356 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166594/ICMGGh7zg+001344 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166594/ICMGGh7zg+001428 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166594/ICMGGh7zg+001380 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166594/ICMSHh7zg+001368 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166594/ICMGGh7zg+001404 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166594/ICMUAh7zg+001392 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166594/ICMSHh7zg+001440 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166594/ICMUAh7zg+001368 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166594/ICMUAh7zg+001440 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166594/ICMUAh7zg+001344 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166594/ICMSHh7zg+001356 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166594/ICMUAh7zg+001356 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166594/ICMGGh7zg+001416 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166594/ICMSHh7zg+001428 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166594/ICMGGh7zg+001368 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166594/ICMUAh7zg+001428 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166594/ICMGGh7zg+001440 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166594/ICMGGh7zg+001392 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166594/ICMUAh7zg+001380 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166594/ICMSHh7zg+001380 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166594/ICMSHh7zg+001404 Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a05k_2016092300_15_949_12166594_0_r812719879_14.zip Uploading the final file: upload_file_14.zip Uploading trickle at timestep: 1295100 04:55:23 (1414522): called boinc_finish(0) The code in app_control.cpp for the client that is handling the error 9 reporting is this section:
if (WIFEXITED(stat)) {
result->exit_status = WEXITSTATUS(stat);
double x;
char buf[256];
bool is_notice;
int e;
if (temporary_exit_file_present(x, buf, is_notice)) {
handle_temporary_exit(will_restart, x, buf, is_notice);
} else {
if (log_flags.task_debug) {
msg_printf(result->project, MSG_INFO,
"[task] process exited with status %d\n",
result->exit_status
);
}
if (result->exit_status) {
set_task_state(PROCESS_EXITED, "handle_exited_app");
snprintf(err_msg, sizeof(err_msg),
"process exited with code %d (0x%x, %d)",
result->exit_status, result->exit_status,
(~0xff)|result->exit_status
);
gstate.report_result_error(*result, err_msg);
} else {
if (finish_file_present(e)) {
set_task_state(PROCESS_EXITED, "handle_exited_app");
} else {
handle_premature_exit(will_restart);
}
}
}
} else if (WIFSIGNALED(stat)) {
int got_signal = WTERMSIG(stat);
if (log_flags.task_debug) {
msg_printf(result->project, MSG_INFO,
"[task] process got signal %d", got_signal
);
}
if (WIFEXITED(stat)) { <=== This evaluates to TRUE, the exit was NORMAL result->exit_status = WEXITSTATUS(stat); <==== This sets the result's exit_status flag to be that for the child, in this case 9 if (result->exit_status) { <==== Because non-zero we do this section of code set_task_state(PROCESS_EXITED, "handle_exited_app"); snprintf(err_msg, sizeof(err_msg), "process exited with code %d (0x%x, %d)", result->exit_status, result->exit_status, (~0xff)|result->exit_status ); gstate.report_result_error(*result, err_msg); So that is why I'm asking if a wrapper is being used and it is the wrapper code that I need to go and look at. Is the wrapper code being used a standard BOINC flavour or have you made your own version that is not available to the public ? It would seem to me that the application/model that is doing all the work has done a splendid job and got all the right answers and sent them to the server but at the very last moment the wrapper has indicated a problem of its own that marks all that work and the task done as invalid. I've seen it mentioned several times here that error 9 is signal 9 but it isn't. There is a separate bit of code, at the end of that posted above, that handles signals and you get a different message in the log. I'm solely looking at those tasks that get an error 9 after completing the work correctly as best I can tell. You could check that by requiring a quorum to validate against but then you are duplicating every task rather than just having those that fail get repeated by a second or third host. Linux says Error 9 is EBADF - It is displayed in case of bad file descriptor I have been looking for any remnants in the slot or project directories that would indicate something went amiss but have found no such evidence. I enabled slot_debug that showed approximately 1,950 files got cleared out of the slot upon termination without a problem. The temporary task directory under the projects folder is gone. The task job files (jf_*) have gone. To get any further with the cause of this problem will require a better understanding of what is happening right at the end. If it is the wrapper code that is returning an error code of 9 right at the end then that is where we need to look to determine the cause. |
|
34)
Message boards :
Number crunching :
OpenIFS Discussion
(Message 66825)
Posted 8 Dec 2022 by PDW
Post: I don't think the error/success difference can be attributed to the model code or data, but must be down to something in the host environment. Not sure that I can agree with that completely yet. Another task completed the application successfully but failed with error 9. I had turned task_debug on so have more info in the event log but that'll have to wait till tomorrow, though it is still no clearer of the exact problem. Linux says Error 9 is EBADF - It is displayed in case of bad file descriptor. Having observed the ending of the task it is the process of 'tidying up' after the last zip file has been uploaded when the problem occurred. It took 1 minute 45 seconds between completing the zip upload to the status 9 being reported. All the files, and there are a lot of them, need to be cleared away. No idea if Boinc does that by just trying to delete the slot directory or has a list of the files/directories to remove. Perhaps something in that process either loses the plot and doesn't track what is being asked to be deleted with what has been deleted or asks for something to be deleted that is no longer there. Maybe it is during the process of creating the "what to tell the server" dialog but that makes it up to the server so seems unlikely. The host has lots of free memory. It is returning about 1,200 tasks daily, the only one that fails with this error is the oifs task. The other project's server shows the host has about 700 tasks pending validation, 16,000 valid tasks, 0 invalid and 3 error tasks ( those are 3 cancelled by the server), so no problem seen there. The host has approximately 1.75 hyperthreads free according to top and uses an SSD drive. Some oifs tasks work fine and some don't and return error 9. Is there a better Event Log flag option that will look even deeper into the abyss and determine what actually is causing the 9 to be returned ? I think the time gaps before 'finish' and 'report' are entirely normal and deliberate The report time is most likely delayed because within Boinc it knows it is somewhere along the 3,636 second delay the project requested. If you actually hit Update the task is reported immediately. Having the report_results_immediately flag set in cc_config.xml makes no difference. |
|
35)
Message boards :
Number crunching :
OpenIFS Discussion
(Message 66807)
Posted 7 Dec 2022 by PDW
Post: The task page says a task failed because... Outcome Computation error Client state Compute error Exit status 9 (0x00000009) Unknown error code But the end of the stderr log says all okay... 11:20:07 STEP 2951 H=2951:00 +CPU= 18.626 11:20:37 STEP 2952 H=2952:00 +CPU= 29.460 The child process terminated with status: 0 Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhpi1+002952 Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhpi1+002952 Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhpi1+002952 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163983/ICMGGhpi1+002928 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163983/ICMGGhpi1+002952 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163983/ICMSHhpi1+002940 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163983/ICMUAhpi1+002952 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163983/ICMUAhpi1+002940 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163983/ICMUAhpi1+002928 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163983/ICMGGhpi1+002940 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163983/ICMSHhpi1+002952 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163983/ICMSHhpi1+002928 Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0894_2021050100_123_945_12163983_1_r1746892727_122.zip Uploading the final file: upload_file_122.zip Uploading trickle at timestep: 10623600 11:24:30 (825570): called boinc_finish(0) </stderr_txt> The event log had no errors/problems either showing this task completed and a new task started. There was no master executable left to kill. The run/CPU time for the task looked correct. Yet it is claimed it is a failure ! |
|
36)
Message boards :
Number crunching :
OpenIFS Discussion
(Message 66753)
Posted 3 Dec 2022 by PDW
Post: Had a task fail after 12 hours with error: free(): unaligned chunk detected in tcache 2 Have completed 5 okay so far, running the same additional project all the time so no change in memory usage. No suspending of work (manually or by Boinc). The master process for the aborted task was still running so terminated it, it showed the same slot as the new task that had started up. |
|
37)
Message boards :
Number crunching :
Error while computing???
(Message 59466)
Posted 18 Jan 2019 by PDW
Post: Why don't they cancel the batch from the server and cancel the units from the machines ? |
|
38)
Message boards :
Number crunching :
Why no credit in Statistics?
(Message 58705)
Posted 6 Sep 2018 by PDW
Post: And I think that the news at present is bad. Does he have to spend the month before the term starts getting ready ? |
|
39)
Message boards :
Number crunching :
Nearly there
(Message 58291)
Posted 26 Jun 2018 by PDW
Post: When is the results database going to be updated ? I have a load of tasks showing as in progress in my website account that I have already run, completed and returned. Also credit is down by about 1.7 million from what it had been. |
|
40)
Message boards :
Number crunching :
Stuck upload issue
(Message 58007)
Posted 29 Mar 2018 by PDW
Post: I still have one task that is stuck uploading to upload6, it has sent 6.66/104.92MB so far... This one has finally uploaded successfully ! |
©2024 climateprediction.net