climateprediction.net (CPDN) home page
Thread 'OpenIFS Discussion'

Thread 'OpenIFS Discussion'

Message boards : Number crunching : OpenIFS Discussion
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 21 · 22 · 23 · 24 · 25 · 26 · 27 . . . 32 · Next

AuthorMessage
ProfileDave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4541
Credit: 19,039,635
RAC: 18,944
Message 68305 - Posted: 13 Feb 2023, 21:11:56 UTC - in response to Message 68303.  

Not a problem, it was just an observation.
ID: 68305 · Report as offensive     Reply Quote
[SG]Felix

Send message
Joined: 4 Oct 15
Posts: 34
Credit: 9,075,151
RAC: 374
Message 68307 - Posted: 13 Feb 2023, 23:00:23 UTC
Last modified: 13 Feb 2023, 23:02:50 UTC

I have a small Problem with those BL WUs:

Had no Problems with PS WUs.

Here a Link to one of my WUs:
https://www.cpdn.org/result.php?resultid=22311544
Mo 13 Feb 2023 23:47:02 CET | climateprediction.net | Computation for task oifs_43r3_bl_a26b_2016092300_15_991_12209606_0 finished
Mo 13 Feb 2023 23:47:06 CET | climateprediction.net | Output file oifs_43r3_bl_a26b_2016092300_15_991_12209606_0_r1758202807_7.zip for task oifs_43r3_bl_a26b_2016092300_15_991_12209606_0 absent
Mo 13 Feb 2023 23:47:06 CET | climateprediction.net | Output file oifs_43r3_bl_a26b_2016092300_15_991_12209606_0_r1758202807_8.zip for task oifs_43r3_bl_a26b_2016092300_15_991_12209606_0 absent
Mo 13 Feb 2023 23:47:06 CET | climateprediction.net | Output file oifs_43r3_bl_a26b_2016092300_15_991_12209606_0_r1758202807_9.zip for task oifs_43r3_bl_a26b_2016092300_15_991_12209606_0 absent
Mo 13 Feb 2023 23:47:06 CET | climateprediction.net | Output file oifs_43r3_bl_a26b_2016092300_15_991_12209606_0_r1758202807_10.zip for task oifs_43r3_bl_a26b_2016092300_15_991_12209606_0 absent
Mo 13 Feb 2023 23:47:06 CET | climateprediction.net | Output file oifs_43r3_bl_a26b_2016092300_15_991_12209606_0_r1758202807_11.zip for task oifs_43r3_bl_a26b_2016092300_15_991_12209606_0 absent
Mo 13 Feb 2023 23:47:06 CET | climateprediction.net | Output file oifs_43r3_bl_a26b_2016092300_15_991_12209606_0_r1758202807_12.zip for task oifs_43r3_bl_a26b_2016092300_15_991_12209606_0 absent
Mo 13 Feb 2023 23:47:06 CET | climateprediction.net | Output file oifs_43r3_bl_a26b_2016092300_15_991_12209606_0_r1758202807_13.zip for task oifs_43r3_bl_a26b_2016092300_15_991_12209606_0 absent
Mo 13 Feb 2023 23:47:06 CET | climateprediction.net | Output file oifs_43r3_bl_a26b_2016092300_15_991_12209606_0_r1758202807_14.zip for task oifs_43r3_bl_a26b_2016092300_15_991_12209606_0 absent
Mo 13 Feb 2023 23:47:06 CET | climateprediction.net | Starting task oifs_43r3_bl_a2nz_2016092300_15_991_12210242_0
Mo 13 Feb 2023 23:53:18 CET | climateprediction.net | Computation for task oifs_43r3_bl_a2nz_2016092300_15_991_12210242_0 finished
Mo 13 Feb 2023 23:53:18 CET | climateprediction.net | Output file oifs_43r3_bl_a2nz_2016092300_15_991_12210242_0_r1856183219_0.zip for task oifs_43r3_bl_a2nz_2016092300_15_991_12210242_0 absent
Mo 13 Feb 2023 23:53:18 CET | climateprediction.net | Output file oifs_43r3_bl_a2nz_2016092300_15_991_12210242_0_r1856183219_1.zip for task oifs_43r3_bl_a2nz_2016092300_15_991_12210242_0 absent
Mo 13 Feb 2023 23:53:18 CET | climateprediction.net | Output file oifs_43r3_bl_a2nz_2016092300_15_991_12210242_0_r1856183219_2.zip for task oifs_43r3_bl_a2nz_2016092300_15_991_12210242_0 absent
Mo 13 Feb 2023 23:53:18 CET | climateprediction.net | Output file oifs_43r3_bl_a2nz_2016092300_15_991_12210242_0_r1856183219_3.zip for task oifs_43r3_bl_a2nz_2016092300_15_991_12210242_0 absent
Mo 13 Feb 2023 23:53:18 CET | climateprediction.net | Output file oifs_43r3_bl_a2nz_2016092300_15_991_12210242_0_r1856183219_4.zip for task oifs_43r3_bl_a2nz_2016092300_15_991_12210242_0 absent
Mo 13 Feb 2023 23:53:18 CET | climateprediction.net | Output file oifs_43r3_bl_a2nz_2016092300_15_991_12210242_0_r1856183219_5.zip for task oifs_43r3_bl_a2nz_2016092300_15_991_12210242_0 absent
Mo 13 Feb 2023 23:53:18 CET | climateprediction.net | Output file oifs_43r3_bl_a2nz_2016092300_15_991_12210242_0_r1856183219_6.zip for task oifs_43r3_bl_a2nz_2016092300_15_991_12210242_0 absent
Mo 13 Feb 2023 23:53:18 CET | climateprediction.net | Output file oifs_43r3_bl_a2nz_2016092300_15_991_12210242_0_r1856183219_7.zip for task oifs_43r3_bl_a2nz_2016092300_15_991_12210242_0 absent
Mo 13 Feb 2023 23:53:18 CET | climateprediction.net | Output file oifs_43r3_bl_a2nz_2016092300_15_991_12210242_0_r1856183219_8.zip for task oifs_43r3_bl_a2nz_2016092300_15_991_12210242_0 absent
Mo 13 Feb 2023 23:53:18 CET | climateprediction.net | Output file oifs_43r3_bl_a2nz_2016092300_15_991_12210242_0_r1856183219_9.zip for task oifs_43r3_bl_a2nz_2016092300_15_991_12210242_0 absent
Mo 13 Feb 2023 23:53:18 CET | climateprediction.net | Output file oifs_43r3_bl_a2nz_2016092300_15_991_12210242_0_r1856183219_10.zip for task oifs_43r3_bl_a2nz_2016092300_15_991_12210242_0 absent
Mo 13 Feb 2023 23:53:18 CET | climateprediction.net | Output file oifs_43r3_bl_a2nz_2016092300_15_991_12210242_0_r1856183219_11.zip for task oifs_43r3_bl_a2nz_2016092300_15_991_12210242_0 absent
Mo 13 Feb 2023 23:53:18 CET | climateprediction.net | Output file oifs_43r3_bl_a2nz_2016092300_15_991_12210242_0_r1856183219_12.zip for task oifs_43r3_bl_a2nz_2016092300_15_991_12210242_0 absent
Mo 13 Feb 2023 23:53:18 CET | climateprediction.net | Output file oifs_43r3_bl_a2nz_2016092300_15_991_12210242_0_r1856183219_13.zip for task oifs_43r3_bl_a2nz_2016092300_15_991_12210242_0 absent
Mo 13 Feb 2023 23:53:18 CET | climateprediction.net | Output file oifs_43r3_bl_a2nz_2016092300_15_991_12210242_0_r1856183219_14.zip for task oifs_43r3_bl_a2nz_2016092300_15_991_12210242_0 absent
Mo 13 Feb 2023 23:53:18 CET | climateprediction.net | Starting task oifs_43r3_bl_a2nv_2016092300_15_991_12210238_0
Mo 13 Feb 2023 23:53:22 CET | climateprediction.net | Computation for task oifs_43r3_bl_a2nv_2016092300_15_991_12210238_0 finished
Mo 13 Feb 2023 23:53:22 CET | climateprediction.net | Output file oifs_43r3_bl_a2nv_2016092300_15_991_12210238_0_r31801520_0.zip for task oifs_43r3_bl_a2nv_2016092300_15_991_12210238_0 absent
Mo 13 Feb 2023 23:53:22 CET | climateprediction.net | Output file oifs_43r3_bl_a2nv_2016092300_15_991_12210238_0_r31801520_1.zip for task oifs_43r3_bl_a2nv_2016092300_15_991_12210238_0 absent
Mo 13 Feb 2023 23:53:22 CET | climateprediction.net | Output file oifs_43r3_bl_a2nv_2016092300_15_991_12210238_0_r31801520_2.zip for task oifs_43r3_bl_a2nv_2016092300_15_991_12210238_0 absent
Mo 13 Feb 2023 23:53:22 CET | climateprediction.net | Output file oifs_43r3_bl_a2nv_2016092300_15_991_12210238_0_r31801520_3.zip for task oifs_43r3_bl_a2nv_2016092300_15_991_12210238_0 absent
Mo 13 Feb 2023 23:53:22 CET | climateprediction.net | Output file oifs_43r3_bl_a2nv_2016092300_15_991_12210238_0_r31801520_4.zip for task oifs_43r3_bl_a2nv_2016092300_15_991_12210238_0 absent
Mo 13 Feb 2023 23:53:22 CET | climateprediction.net | Output file oifs_43r3_bl_a2nv_2016092300_15_991_12210238_0_r31801520_5.zip for task oifs_43r3_bl_a2nv_2016092300_15_991_12210238_0 absent
Mo 13 Feb 2023 23:53:22 CET | climateprediction.net | Output file oifs_43r3_bl_a2nv_2016092300_15_991_12210238_0_r31801520_6.zip for task oifs_43r3_bl_a2nv_2016092300_15_991_12210238_0 absent
Mo 13 Feb 2023 23:53:22 CET | climateprediction.net | Output file oifs_43r3_bl_a2nv_2016092300_15_991_12210238_0_r31801520_7.zip for task oifs_43r3_bl_a2nv_2016092300_15_991_12210238_0 absent
Mo 13 Feb 2023 23:53:22 CET | climateprediction.net | Output file oifs_43r3_bl_a2nv_2016092300_15_991_12210238_0_r31801520_8.zip for task oifs_43r3_bl_a2nv_2016092300_15_991_12210238_0 absent
Mo 13 Feb 2023 23:53:22 CET | climateprediction.net | Output file oifs_43r3_bl_a2nv_2016092300_15_991_12210238_0_r31801520_9.zip for task oifs_43r3_bl_a2nv_2016092300_15_991_12210238_0 absent
Mo 13 Feb 2023 23:53:22 CET | climateprediction.net | Output file oifs_43r3_bl_a2nv_2016092300_15_991_12210238_0_r31801520_10.zip for task oifs_43r3_bl_a2nv_2016092300_15_991_12210238_0 absent
Mo 13 Feb 2023 23:53:22 CET | climateprediction.net | Output file oifs_43r3_bl_a2nv_2016092300_15_991_12210238_0_r31801520_11.zip for task oifs_43r3_bl_a2nv_2016092300_15_991_12210238_0 absent
Mo 13 Feb 2023 23:53:22 CET | climateprediction.net | Output file oifs_43r3_bl_a2nv_2016092300_15_991_12210238_0_r31801520_12.zip for task oifs_43r3_bl_a2nv_2016092300_15_991_12210238_0 absent
Mo 13 Feb 2023 23:53:22 CET | climateprediction.net | Output file oifs_43r3_bl_a2nv_2016092300_15_991_12210238_0_r31801520_13.zip for task oifs_43r3_bl_a2nv_2016092300_15_991_12210238_0 absent
Mo 13 Feb 2023 23:53:22 CET | climateprediction.net | Output file oifs_43r3_bl_a2nv_2016092300_15_991_12210238_0_r31801520_14.zip for task oifs_43r3_bl_a2nv_2016092300_15_991_12210238_0 absent
Mo 13 Feb 2023 23:53:22 CET | climateprediction.net | Starting task oifs_43r3_bl_a31n_2016092300_15_991_12210734_0
Mo 13 Feb 2023 23:53:26 CET | climateprediction.net | Computation for task oifs_43r3_bl_a31n_2016092300_15_991_12210734_0 finished
Mo 13 Feb 2023 23:53:26 CET | climateprediction.net | Output file oifs_43r3_bl_a31n_2016092300_15_991_12210734_0_r1142312962_0.zip for task oifs_43r3_bl_a31n_2016092300_15_991_12210734_0 absent
Mo 13 Feb 2023 23:53:26 CET | climateprediction.net | Output file oifs_43r3_bl_a31n_2016092300_15_991_12210734_0_r1142312962_1.zip for task oifs_43r3_bl_a31n_2016092300_15_991_12210734_0 absent
Mo 13 Feb 2023 23:53:26 CET | climateprediction.net | Output file oifs_43r3_bl_a31n_2016092300_15_991_12210734_0_r1142312962_2.zip for task oifs_43r3_bl_a31n_2016092300_15_991_12210734_0 absent
Mo 13 Feb 2023 23:53:26 CET | climateprediction.net | Output file oifs_43r3_bl_a31n_2016092300_15_991_12210734_0_r1142312962_3.zip for task oifs_43r3_bl_a31n_2016092300_15_991_12210734_0 absent
Mo 13 Feb 2023 23:53:26 CET | climateprediction.net | Output file oifs_43r3_bl_a31n_2016092300_15_991_12210734_0_r1142312962_4.zip for task oifs_43r3_bl_a31n_2016092300_15_991_12210734_0 absent
Mo 13 Feb 2023 23:53:26 CET | climateprediction.net | Output file oifs_43r3_bl_a31n_2016092300_15_991_12210734_0_r1142312962_5.zip for task oifs_43r3_bl_a31n_2016092300_15_991_12210734_0 absent
Mo 13 Feb 2023 23:53:26 CET | climateprediction.net | Output file oifs_43r3_bl_a31n_2016092300_15_991_12210734_0_r1142312962_6.zip for task oifs_43r3_bl_a31n_2016092300_15_991_12210734_0 absent
Mo 13 Feb 2023 23:53:26 CET | climateprediction.net | Output file oifs_43r3_bl_a31n_2016092300_15_991_12210734_0_r1142312962_7.zip for task oifs_43r3_bl_a31n_2016092300_15_991_12210734_0 absent
Mo 13 Feb 2023 23:53:26 CET | climateprediction.net | Output file oifs_43r3_bl_a31n_2016092300_15_991_12210734_0_r1142312962_8.zip for task oifs_43r3_bl_a31n_2016092300_15_991_12210734_0 absent
Mo 13 Feb 2023 23:53:26 CET | climateprediction.net | Output file oifs_43r3_bl_a31n_2016092300_15_991_12210734_0_r1142312962_9.zip for task oifs_43r3_bl_a31n_2016092300_15_991_12210734_0 absent
Mo 13 Feb 2023 23:53:26 CET | climateprediction.net | Output file oifs_43r3_bl_a31n_2016092300_15_991_12210734_0_r1142312962_10.zip for task oifs_43r3_bl_a31n_2016092300_15_991_12210734_0 absent
Mo 13 Feb 2023 23:53:26 CET | climateprediction.net | Output file oifs_43r3_bl_a31n_2016092300_15_991_12210734_0_r1142312962_11.zip for task oifs_43r3_bl_a31n_2016092300_15_991_12210734_0 absent
Mo 13 Feb 2023 23:53:26 CET | climateprediction.net | Output file oifs_43r3_bl_a31n_2016092300_15_991_12210734_0_r1142312962_12.zip for task oifs_43r3_bl_a31n_2016092300_15_991_12210734_0 absent
Mo 13 Feb 2023 23:53:26 CET | climateprediction.net | Output file oifs_43r3_bl_a31n_2016092300_15_991_12210734_0_r1142312962_13.zip for task oifs_43r3_bl_a31n_2016092300_15_991_12210734_0 absent
Mo 13 Feb 2023 23:53:26 CET | climateprediction.net | Output file oifs_43r3_bl_a31n_2016092300_15_991_12210734_0_r1142312962_14.zip for task oifs_43r3_bl_a31n_2016092300_15_991_12210734_0 absent
Mo 13 Feb 2023 23:53:26 CET | climateprediction.net | Starting task oifs_43r3_bl_a316_2016092300_15_991_12210717_0
Mo 13 Feb 2023 23:53:30 CET | climateprediction.net | Computation for task oifs_43r3_bl_a316_2016092300_15_991_12210717_0 finished
Mo 13 Feb 2023 23:53:30 CET | climateprediction.net | Output file oifs_43r3_bl_a316_2016092300_15_991_12210717_0_r815290613_0.zip for task oifs_43r3_bl_a316_2016092300_15_991_12210717_0 absent
Mo 13 Feb 2023 23:53:30 CET | climateprediction.net | Output file oifs_43r3_bl_a316_2016092300_15_991_12210717_0_r815290613_1.zip for task oifs_43r3_bl_a316_2016092300_15_991_12210717_0 absent
Mo 13 Feb 2023 23:53:30 CET | climateprediction.net | Output file oifs_43r3_bl_a316_2016092300_15_991_12210717_0_r815290613_2.zip for task oifs_43r3_bl_a316_2016092300_15_991_12210717_0 absent
Mo 13 Feb 2023 23:53:30 CET | climateprediction.net | Output file oifs_43r3_bl_a316_2016092300_15_991_12210717_0_r815290613_3.zip for task oifs_43r3_bl_a316_2016092300_15_991_12210717_0 absent
Mo 13 Feb 2023 23:53:30 CET | climateprediction.net | Output file oifs_43r3_bl_a316_2016092300_15_991_12210717_0_r815290613_4.zip for task oifs_43r3_bl_a316_2016092300_15_991_12210717_0 absent
Mo 13 Feb 2023 23:53:30 CET | climateprediction.net | Output file oifs_43r3_bl_a316_2016092300_15_991_12210717_0_r815290613_5.zip for task oifs_43r3_bl_a316_2016092300_15_991_12210717_0 absent
Mo 13 Feb 2023 23:53:30 CET | climateprediction.net | Output file oifs_43r3_bl_a316_2016092300_15_991_12210717_0_r815290613_6.zip for task oifs_43r3_bl_a316_2016092300_15_991_12210717_0 absent
Mo 13 Feb 2023 23:53:30 CET | climateprediction.net | Output file oifs_43r3_bl_a316_2016092300_15_991_12210717_0_r815290613_7.zip for task oifs_43r3_bl_a316_2016092300_15_991_12210717_0 absent
Mo 13 Feb 2023 23:53:30 CET | climateprediction.net | Output file oifs_43r3_bl_a316_2016092300_15_991_12210717_0_r815290613_8.zip for task oifs_43r3_bl_a316_2016092300_15_991_12210717_0 absent
Mo 13 Feb 2023 23:53:30 CET | climateprediction.net | Output file oifs_43r3_bl_a316_2016092300_15_991_12210717_0_r815290613_9.zip for task oifs_43r3_bl_a316_2016092300_15_991_12210717_0 absent
Mo 13 Feb 2023 23:53:30 CET | climateprediction.net | Output file oifs_43r3_bl_a316_2016092300_15_991_12210717_0_r815290613_10.zip for task oifs_43r3_bl_a316_2016092300_15_991_12210717_0 absent
Mo 13 Feb 2023 23:53:30 CET | climateprediction.net | Output file oifs_43r3_bl_a316_2016092300_15_991_12210717_0_r815290613_11.zip for task oifs_43r3_bl_a316_2016092300_15_991_12210717_0 absent
Mo 13 Feb 2023 23:53:30 CET | climateprediction.net | Output file oifs_43r3_bl_a316_2016092300_15_991_12210717_0_r815290613_12.zip for task oifs_43r3_bl_a316_2016092300_15_991_12210717_0 absent
Mo 13 Feb 2023 23:53:30 CET | climateprediction.net | Output file oifs_43r3_bl_a316_2016092300_15_991_12210717_0_r815290613_13.zip for task oifs_43r3_bl_a316_2016092300_15_991_12210717_0 absent
Mo 13 Feb 2023 23:53:30 CET | climateprediction.net | Output file oifs_43r3_bl_a316_2016092300_15_991_12210717_0_r815290613_14.zip for task oifs_43r3_bl_a316_2016092300_15_991_12210717_0 absent
Mo 13 Feb 2023 23:53:30 CET | climateprediction.net | Starting task oifs_43r3_bl_a3hk_2016092300_15_991_12211307_0
Mo 13 Feb 2023 23:53:34 CET | climateprediction.net | Computation for task oifs_43r3_bl_a3hk_2016092300_15_991_12211307_0 finished
Mo 13 Feb 2023 23:53:34 CET | climateprediction.net | Output file oifs_43r3_bl_a3hk_2016092300_15_991_12211307_0_r921471673_0.zip for task oifs_43r3_bl_a3hk_2016092300_15_991_12211307_0 absent
Mo 13 Feb 2023 23:53:34 CET | climateprediction.net | Output file oifs_43r3_bl_a3hk_2016092300_15_991_12211307_0_r921471673_1.zip for task oifs_43r3_bl_a3hk_2016092300_15_991_12211307_0 absent
Mo 13 Feb 2023 23:53:34 CET | climateprediction.net | Output file oifs_43r3_bl_a3hk_2016092300_15_991_12211307_0_r921471673_2.zip for task oifs_43r3_bl_a3hk_2016092300_15_991_12211307_0 absent
Mo 13 Feb 2023 23:53:34 CET | climateprediction.net | Output file oifs_43r3_bl_a3hk_2016092300_15_991_12211307_0_r921471673_3.zip for task oifs_43r3_bl_a3hk_2016092300_15_991_12211307_0 absent
Mo 13 Feb 2023 23:53:34 CET | climateprediction.net | Output file oifs_43r3_bl_a3hk_2016092300_15_991_12211307_0_r921471673_4.zip for task oifs_43r3_bl_a3hk_2016092300_15_991_12211307_0 absent
Mo 13 Feb 2023 23:53:34 CET | climateprediction.net | Output file oifs_43r3_bl_a3hk_2016092300_15_991_12211307_0_r921471673_5.zip for task oifs_43r3_bl_a3hk_2016092300_15_991_12211307_0 absent
Mo 13 Feb 2023 23:53:34 CET | climateprediction.net | Output file oifs_43r3_bl_a3hk_2016092300_15_991_12211307_0_r921471673_6.zip for task oifs_43r3_bl_a3hk_2016092300_15_991_12211307_0 absent
Mo 13 Feb 2023 23:53:34 CET | climateprediction.net | Output file oifs_43r3_bl_a3hk_2016092300_15_991_12211307_0_r921471673_7.zip for task oifs_43r3_bl_a3hk_2016092300_15_991_12211307_0 absent
Mo 13 Feb 2023 23:53:34 CET | climateprediction.net | Output file oifs_43r3_bl_a3hk_2016092300_15_991_12211307_0_r921471673_8.zip for task oifs_43r3_bl_a3hk_2016092300_15_991_12211307_0 absent
Mo 13 Feb 2023 23:53:34 CET | climateprediction.net | Output file oifs_43r3_bl_a3hk_2016092300_15_991_12211307_0_r921471673_9.zip for task oifs_43r3_bl_a3hk_2016092300_15_991_12211307_0 absent
Mo 13 Feb 2023 23:53:34 CET | climateprediction.net | Output file oifs_43r3_bl_a3hk_2016092300_15_991_12211307_0_r921471673_10.zip for task oifs_43r3_bl_a3hk_2016092300_15_991_12211307_0 absent
Mo 13 Feb 2023 23:53:34 CET | climateprediction.net | Output file oifs_43r3_bl_a3hk_2016092300_15_991_12211307_0_r921471673_11.zip for task oifs_43r3_bl_a3hk_2016092300_15_991_12211307_0 absent
Mo 13 Feb 2023 23:53:34 CET | climateprediction.net | Output file oifs_43r3_bl_a3hk_2016092300_15_991_12211307_0_r921471673_12.zip for task oifs_43r3_bl_a3hk_2016092300_15_991_12211307_0 absent
Mo 13 Feb 2023 23:53:34 CET | climateprediction.net | Output file oifs_43r3_bl_a3hk_2016092300_15_991_12211307_0_r921471673_13.zip for task oifs_43r3_bl_a3hk_2016092300_15_991_12211307_0 absent
Mo 13 Feb 2023 23:53:34 CET | climateprediction.net | Output file oifs_43r3_bl_a3hk_2016092300_15_991_12211307_0_r921471673_14.zip for task oifs_43r3_bl_a3hk_2016092300_15_991_12211307_0 absent
Mo 13 Feb 2023 23:53:34 CET | climateprediction.net | Starting task oifs_43r3_bl_a3i0_2016092300_15_991_12211323_0
Mo 13 Feb 2023 23:53:39 CET | climateprediction.net | Computation for task oifs_43r3_bl_a3i0_2016092300_15_991_12211323_0 finished
Mo 13 Feb 2023 23:53:39 CET | climateprediction.net | Output file oifs_43r3_bl_a3i0_2016092300_15_991_12211323_0_r1613172252_0.zip for task oifs_43r3_bl_a3i0_2016092300_15_991_12211323_0 absent
Mo 13 Feb 2023 23:53:39 CET | climateprediction.net | Output file oifs_43r3_bl_a3i0_2016092300_15_991_12211323_0_r1613172252_1.zip for task oifs_43r3_bl_a3i0_2016092300_15_991_12211323_0 absent
Mo 13 Feb 2023 23:53:39 CET | climateprediction.net | Output file oifs_43r3_bl_a3i0_2016092300_15_991_12211323_0_r1613172252_2.zip for task oifs_43r3_bl_a3i0_2016092300_15_991_12211323_0 absent
Mo 13 Feb 2023 23:53:39 CET | climateprediction.net | Output file oifs_43r3_bl_a3i0_2016092300_15_991_12211323_0_r1613172252_3.zip for task oifs_43r3_bl_a3i0_2016092300_15_991_12211323_0 absent
Mo 13 Feb 2023 23:53:39 CET | climateprediction.net | Output file oifs_43r3_bl_a3i0_2016092300_15_991_12211323_0_r1613172252_4.zip for task oifs_43r3_bl_a3i0_2016092300_15_991_12211323_0 absent
Mo 13 Feb 2023 23:53:39 CET | climateprediction.net | Output file oifs_43r3_bl_a3i0_2016092300_15_991_12211323_0_r1613172252_5.zip for task oifs_43r3_bl_a3i0_2016092300_15_991_12211323_0 absent
Mo 13 Feb 2023 23:53:39 CET | climateprediction.net | Output file oifs_43r3_bl_a3i0_2016092300_15_991_12211323_0_r1613172252_6.zip for task oifs_43r3_bl_a3i0_2016092300_15_991_12211323_0 absent
Mo 13 Feb 2023 23:53:39 CET | climateprediction.net | Output file oifs_43r3_bl_a3i0_2016092300_15_991_12211323_0_r1613172252_7.zip for task oifs_43r3_bl_a3i0_2016092300_15_991_12211323_0 absent
Mo 13 Feb 2023 23:53:39 CET | climateprediction.net | Output file oifs_43r3_bl_a3i0_2016092300_15_991_12211323_0_r1613172252_8.zip for task oifs_43r3_bl_a3i0_2016092300_15_991_12211323_0 absent
Mo 13 Feb 2023 23:53:39 CET | climateprediction.net | Output file oifs_43r3_bl_a3i0_2016092300_15_991_12211323_0_r1613172252_9.zip for task oifs_43r3_bl_a3i0_2016092300_15_991_12211323_0 absent
Mo 13 Feb 2023 23:53:39 CET | climateprediction.net | Output file oifs_43r3_bl_a3i0_2016092300_15_991_12211323_0_r1613172252_10.zip for task oifs_43r3_bl_a3i0_2016092300_15_991_12211323_0 absent
Mo 13 Feb 2023 23:53:39 CET | climateprediction.net | Output file oifs_43r3_bl_a3i0_2016092300_15_991_12211323_0_r1613172252_11.zip for task oifs_43r3_bl_a3i0_2016092300_15_991_12211323_0 absent
Mo 13 Feb 2023 23:53:39 CET | climateprediction.net | Output file oifs_43r3_bl_a3i0_2016092300_15_991_12211323_0_r1613172252_12.zip for task oifs_43r3_bl_a3i0_2016092300_15_991_12211323_0 absent
Mo 13 Feb 2023 23:53:39 CET | climateprediction.net | Output file oifs_43r3_bl_a3i0_2016092300_15_991_12211323_0_r1613172252_13.zip for task oifs_43r3_bl_a3i0_2016092300_15_991_12211323_0 absent
Mo 13 Feb 2023 23:53:39 CET | climateprediction.net | Output file oifs_43r3_bl_a3i0_2016092300_15_991_12211323_0_r1613172252_14.zip for task oifs_43r3_bl_a3i0_2016092300_15_991_12211323_0 absent
Mo 13 Feb 2023 23:53:39 CET | climateprediction.net | Starting task oifs_43r3_bl_a3hx_2016092300_15_991_12211320_0
Mo 13 Feb 2023 23:53:43 CET | climateprediction.net | Computation for task oifs_43r3_bl_a3hx_2016092300_15_991_12211320_0 finished
Mo 13 Feb 2023 23:53:43 CET | climateprediction.net | Output file oifs_43r3_bl_a3hx_2016092300_15_991_12211320_0_r1513650664_0.zip for task oifs_43r3_bl_a3hx_2016092300_15_991_12211320_0 absent
Mo 13 Feb 2023 23:53:43 CET | climateprediction.net | Output file oifs_43r3_bl_a3hx_2016092300_15_991_12211320_0_r1513650664_1.zip for task oifs_43r3_bl_a3hx_2016092300_15_991_12211320_0 absent
Mo 13 Feb 2023 23:53:43 CET | climateprediction.net | Output file oifs_43r3_bl_a3hx_2016092300_15_991_12211320_0_r1513650664_2.zip for task oifs_43r3_bl_a3hx_2016092300_15_991_12211320_0 absent
Mo 13 Feb 2023 23:53:43 CET | climateprediction.net | Output file oifs_43r3_bl_a3hx_2016092300_15_991_12211320_0_r1513650664_3.zip for task oifs_43r3_bl_a3hx_2016092300_15_991_12211320_0 absent
Mo 13 Feb 2023 23:53:43 CET | climateprediction.net | Output file oifs_43r3_bl_a3hx_2016092300_15_991_12211320_0_r1513650664_4.zip for task oifs_43r3_bl_a3hx_2016092300_15_991_12211320_0 absent
Mo 13 Feb 2023 23:53:43 CET | climateprediction.net | Output file oifs_43r3_bl_a3hx_2016092300_15_991_12211320_0_r1513650664_5.zip for task oifs_43r3_bl_a3hx_2016092300_15_991_12211320_0 absent
Mo 13 Feb 2023 23:53:43 CET | climateprediction.net | Output file oifs_43r3_bl_a3hx_2016092300_15_991_12211320_0_r1513650664_6.zip for task oifs_43r3_bl_a3hx_2016092300_15_991_12211320_0 absent
Mo 13 Feb 2023 23:53:43 CET | climateprediction.net | Output file oifs_43r3_bl_a3hx_2016092300_15_991_12211320_0_r1513650664_7.zip for task oifs_43r3_bl_a3hx_2016092300_15_991_12211320_0 absent
Mo 13 Feb 2023 23:53:43 CET | climateprediction.net | Output file oifs_43r3_bl_a3hx_2016092300_15_991_12211320_0_r1513650664_8.zip for task oifs_43r3_bl_a3hx_2016092300_15_991_12211320_0 absent
Mo 13 Feb 2023 23:53:43 CET | climateprediction.net | Output file oifs_43r3_bl_a3hx_2016092300_15_991_12211320_0_r1513650664_9.zip for task oifs_43r3_bl_a3hx_2016092300_15_991_12211320_0 absent
Mo 13 Feb 2023 23:53:43 CET | climateprediction.net | Output file oifs_43r3_bl_a3hx_2016092300_15_991_12211320_0_r1513650664_10.zip for task oifs_43r3_bl_a3hx_2016092300_15_991_12211320_0 absent
Mo 13 Feb 2023 23:53:43 CET | climateprediction.net | Output file oifs_43r3_bl_a3hx_2016092300_15_991_12211320_0_r1513650664_11.zip for task oifs_43r3_bl_a3hx_2016092300_15_991_12211320_0 absent
Mo 13 Feb 2023 23:53:43 CET | climateprediction.net | Output file oifs_43r3_bl_a3hx_2016092300_15_991_12211320_0_r1513650664_12.zip for task oifs_43r3_bl_a3hx_2016092300_15_991_12211320_0 absent
Mo 13 Feb 2023 23:53:43 CET | climateprediction.net | Output file oifs_43r3_bl_a3hx_2016092300_15_991_12211320_0_r1513650664_13.zip for task oifs_43r3_bl_a3hx_2016092300_15_991_12211320_0 absent
Mo 13 Feb 2023 23:53:43 CET | climateprediction.net | Output file oifs_43r3_bl_a3hx_2016092300_15_991_12211320_0_r1513650664_14.zip for task oifs_43r3_bl_a3hx_2016092300_15_991_12211320_0 absent


Update:
And this one did work at the beginning:
https://www.cpdn.org/result.php?resultid=22309804
ID: 68307 · Report as offensive     Reply Quote
ProfileDave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4541
Credit: 19,039,635
RAC: 18,944
Message 68308 - Posted: 14 Feb 2023, 7:50:52 UTC - in response to Message 68307.  
Last modified: 14 Feb 2023, 8:31:34 UTC

Here a Link to one of my WUs:
https://www.cpdn.org/result.php?resultid=22311544


Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12211307.zip
/var/lib/boinc-client/slots/1/159l_2/o3chem_l91/O3CHEM06_V2.9  bad CRC 4c885ac3  (should be 54ab7da4)
..Unzipping the climate data file failed


Is where the problem is. None of your tasks have completed their second attempt yet so I am holding off on posting any guesses as to the reason. It looks as if you have wisely decided not to try your second Linux box which is much lower on memory with these but it may be worth letting it try just one at a time to see what happens.

Edit: Just had a reply from Glen about a failed task on the testing branch. It may be worth doing a project reset. It may be a corrupt file for the model type downloaded.

Edit2: Looking at the results in so far as of about 0100UTC when it was updated, 4 out of the 106 fails were file read/copy/unzip fails. About 70 were other but I haven't yet tried looking to see if there is anything linking enough of them to give them their own category.
ID: 68308 · Report as offensive     Reply Quote
AndreyOR

Send message
Joined: 12 Apr 21
Posts: 318
Credit: 14,961,979
RAC: 10,538
Message 68309 - Posted: 14 Feb 2023, 9:34:55 UTC

Had a failure on a newer Ryzen PC with the following code, but there's nothing in the error log, at least that I could pick up on, indicating a problem. Task: https://www.cpdn.org/result.php?resultid=22312560
process exited with code 9 (0x9, -247)
ID: 68309 · Report as offensive     Reply Quote
Glenn Carver

Send message
Joined: 29 Oct 17
Posts: 1049
Credit: 16,614,827
RAC: 12,088
Message 68310 - Posted: 14 Feb 2023, 11:03:25 UTC - in response to Message 68309.  
Last modified: 14 Feb 2023, 11:05:35 UTC

Errors

Had a failure on a newer Ryzen PC with the following code, but there's nothing in the error log, at least that I could pick up on, indicating a problem. Task: https://www.cpdn.org/result.php?resultid=22312560
process exited with code 9 (0x9, -247)
That's one of those annoying tasks where the model ran just fine and then the task either goes wrong in the boinc client (in the boinc_finish function) or maybe as it exits. I know it has something to do with file locks or files not being cleared but exactly what files (whether task or boinc related) we don't (yet) know.

Bad CRC
[SG] Felix, as Dave has already said, this is a file corruption. Either something went wrong in downloading the file, or when it was copied into the slot dir. There's nothing the task code can do about this. My advice if it keeps happening would be to reset the project but this will Abort your current tasks so only do it if things keep going wrong.

Model crashes. There will also be tasks that fail in the model because this experiment is designed to push the limits of what the model can do. If you have a failed task with a very long code traceback in the output, that'll be why. Credit is based on trickles, so credit will be given for work done.

I had a look at the batch stats, over 1000 tasks completed already and the error stats show the memory corruption issue is much reduced (less than 5%); the model crashes by comparison are ~10%.
ID: 68310 · Report as offensive     Reply Quote
ProfileDave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4541
Credit: 19,039,635
RAC: 18,944
Message 68311 - Posted: 14 Feb 2023, 11:23:49 UTC


Bad CRC

[SG] Felix, as Dave has already said, this is a file corruption. Either something went wrong in downloading the file, or when it was copied into the slot dir. There's nothing the task code can do about this. My advice if it keeps happening would be to reset the project but this will Abort your current tasks so only do it if things keep going wrong.
I have posted a question in the BOINC fora about this. I really feel BOINC should be doing CRC checks on all downloads and if the check doesn't match, re-downloading the file. I am awaiting Richard who has responded doing some digging before deciding whether to put in a feature request to git-hub.
ID: 68311 · Report as offensive     Reply Quote
Richard Haselgrove

Send message
Joined: 1 Jan 07
Posts: 1061
Credit: 36,748,059
RAC: 5,647
Message 68312 - Posted: 14 Feb 2023, 11:43:26 UTC - in response to Message 68311.  
Last modified: 14 Feb 2023, 12:32:06 UTC

Richard who has responded doing some digging before deciding whether to put in a feature request to git-hub.
It may take me a while. My suspicion is - following similar errors at other projects - that sometimes BOINC introduces errors if a long download of a large file is interrupted, and tries to restart from an intermediate "OK so far" point. I think BOINC loses count of how much has been downloaded so far, and how much remains to be downloaded.

But that usually results in a byte count mismatch. This case is different: the file in question is a zipped climate data file (clim_data_12211307.zip) identified as belonging to a single specific workunit - WU 12211307 is the source for task 22311544. And the CRC error was detected during unzipping, not downloading. I'd also expect a single-file error like that to affect a single task, not a whole sequence of them (*). It's possible that the file storage system on that host has developed a more significant fault, affecting multiple files.

I'll try pulling out a network cable (**) during a download to test the interrupted/lost count theory, but others may have other explanations worth exploring.

(*) Edit: I'd also expect a zip CRC error to be detected when a task starts its first run, not on a midway restart.

(**) Edit2: Argh - all the tasks have been snapped up. The network cable will have to wait for the next batch.
ID: 68312 · Report as offensive     Reply Quote
Richard Haselgrove

Send message
Joined: 1 Jan 07
Posts: 1061
Credit: 36,748,059
RAC: 5,647
Message 68313 - Posted: 14 Feb 2023, 11:55:32 UTC - in response to Message 68310.  

Credit is based on trickles, so credit will be given for work done.
Is that the last upload file received, or the tiny administrative trickle sent to the scheduler?

I'm only seeing administrative trickles at timesteps 864000 and 1295100 (associated with upload zips 9 and 14) for these tasks. If a task fails after upload 8, does it get any credit? I suppose we'll find out on Sunday morning ...
ID: 68313 · Report as offensive     Reply Quote
Jean-David Beyer

Send message
Joined: 5 Aug 04
Posts: 1120
Credit: 17,202,915
RAC: 2,154
Message 68314 - Posted: 14 Feb 2023, 12:09:31 UTC
Last modified: 14 Feb 2023, 13:02:02 UTC

My first ever double free or corruption

Task 22308784
Name oifs_43r3_bl_a1ei_2016092300_15_991_12208605_0
Workunit 12208605

Error tasks for computer 1511241
22308784 12208605 13 Feb 2023, 15:24:42 UTC 14 Feb 2023, 10:48:08 UTC Error while computing 19,858.70 19,708.54 --- OpenIFS 43r3 Baroclinic Lifecycle v1.11
x86_64-pc-linux-gnu

End of stderr ...

Uploading the intermediate file: upload_file_10.zip
04:46:49 STEP 1057 H= 264:15 +CPU= 19.396
04:47:06 STEP 1058 H= 264:30 +CPU= 17.089
double free or corruption (!prev)

</stderr_txt>

Would this help?
https://linuxhint.com/double-free-corruption-error/
ID: 68314 · Report as offensive     Reply Quote
Jean-David Beyer

Send message
Joined: 5 Aug 04
Posts: 1120
Credit: 17,202,915
RAC: 2,154
Message 68319 - Posted: 14 Feb 2023, 17:40:45 UTC - in response to Message 68318.  

Here is one. Not recent.

22250807 	12166550 	19 Dec 2022, 2:23:58 UTC 	19 Dec 2022, 9:23:21 UTC 	Completed 	22,360.51 	22,031.03 	1,232.00 	OpenIFS 43r3 Baroclinic Lifecycle v1.07
x86_64-pc-linux-gnu

ID: 68319 · Report as offensive     Reply Quote
Glenn Carver

Send message
Joined: 29 Oct 17
Posts: 1049
Credit: 16,614,827
RAC: 12,088
Message 68320 - Posted: 14 Feb 2023, 18:51:13 UTC - in response to Message 68314.  

My first ever double free or corruption
It's a badge of honour, wear it with pride ;)

Would this help? https://linuxhint.com/double-free-corruption-error/
Not really. I know what's causing the problem but catching/reproducing the fails on my machines so I can debug them is nigh impossible. So it means examining the code very carefully (I didn't write it). There have been a number of fixes recently that have gone in. The wrapper process now running includes most of them and I'm glad to say memory issues now account for ~5% of fails. The development code version includes a couple more fixes. It's priority to get these sorted as it's holding up putting out the higher resolution configurations of OpenIFS (which I thought I'd be able to do last year!).
ID: 68320 · Report as offensive     Reply Quote
Jean-David Beyer

Send message
Joined: 5 Aug 04
Posts: 1120
Credit: 17,202,915
RAC: 2,154
Message 68321 - Posted: 14 Feb 2023, 20:13:56 UTC - in response to Message 68320.  

My first ever double free or corruption

It's a badge of honour, wear it with pride ;)


Giggle.

Would this help? https://linuxhint.com/double-free-corruption-error/

Not really. I know what's causing the problem but catching/reproducing the fails on my machines so I can debug them is nigh impossible. So it means examining the code very carefully (I didn't write it).


In the old days, when I worked for a large regulated monopoly we used a program called lint written by Steve Johnson, to examine C (and later, C++) code for all kinds of subtle coding errors (e.g., using uninitialized variables (especially pointers)). It cured lots of problems before they bit us. The main limitation is that it is a static analysis, but since so many errors were detectable this way it was very useful.

I tried to find lint for use these days, but it is difficult to find one in which I have confidence. It may be as much trouble getting a version of it working as to ignore it and try using a debugger such as gdb that you may already be using.

There have been a number of fixes recently that have gone in. The wrapper process now running includes most of them and I'm glad to say memory issues now account for ~5% of fails. The development code version includes a couple more fixes. It's priority to get these sorted as it's holding up putting out the higher resolution configurations of OpenIFS (which I thought I'd be able to do last year!).

ID: 68321 · Report as offensive     Reply Quote
AndreyOR

Send message
Joined: 12 Apr 21
Posts: 318
Credit: 14,961,979
RAC: 10,538
Message 68323 - Posted: 14 Feb 2023, 21:05:39 UTC - in response to Message 68314.  

My first ever double free or corruption

Pretty cool run, it's like your PC is custom built for CPDN, sprinkled with a bit of luck and good resource usage.

You're also the only person that I've seen who uses RHEL. I wonder if Glenn has seen any correlations between failure rates and Linux distros?
ID: 68323 · Report as offensive     Reply Quote
AndreyOR

Send message
Joined: 12 Apr 21
Posts: 318
Credit: 14,961,979
RAC: 10,538
Message 68324 - Posted: 14 Feb 2023, 21:10:31 UTC - in response to Message 68310.  
Last modified: 14 Feb 2023, 21:11:28 UTC

Model crashes. There will also be tasks that fail in the model because this experiment is designed to push the limits of what the model can do. If you have a failed task with a very long code traceback in the output, that'll be why. Credit is based on trickles, so credit will be given for work done.

Is this one of them or is it something else? https://www.cpdn.org/result.php?resultid=22311226. I'm wondering as it was easy to tell with Hadley models when failures were due to this reason, not sure what it looks like with OIFS yet.
ID: 68324 · Report as offensive     Reply Quote
Jean-David Beyer

Send message
Joined: 5 Aug 04
Posts: 1120
Credit: 17,202,915
RAC: 2,154
Message 68325 - Posted: 14 Feb 2023, 22:24:02 UTC - in response to Message 68323.  

Pretty cool run, it's like your PC is custom built for CPDN, sprinkled with a bit of luck and good resource usage.

You're also the only person that I've seen who uses RHEL. I wonder if Glenn has seen any correlations between failure rates and Linux distros?


My machine was not custom-built for anything. It is a Dell Precision 5820 Tower that came with an Intel Xeon Processor W-2245 (8 C 3.9GHz 4.7GHz Turbo HT 16.5M B, 155W DDR4-2933), 32GB 2x16GB DDR4 2933MHz RDIM M ECC Memory that I doubled to 64 GBytes anticipating large RAM requirements of Oifs tasks coming up. It has a M.2 512GB PCIe NVMe Class 40 Solid State Drive mainly for system stuff, and two 4 TByte spinning hard drives. One of those hard drives has a partition just for Boinc stuff: 524 GB — 508 GB free (3.1% full). I started being glad it was so big when it started filling up with trickles during the period where the upload server was not working properly.

I suppose I am one of the few using RHEL because it costs money (around $1/day). CentOS has an almost identical system that used to be free and probably still is. Even Fedora is similar. The main differences are that RHEL releases are supported for 10 years. I suspect CentOS is supported for less time, and Fedora is more interested at being the latest and greatest, so it changes a lot (something I do not want to be bothered with).
ID: 68325 · Report as offensive     Reply Quote
ProfileDave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4541
Credit: 19,039,635
RAC: 18,944
Message 68326 - Posted: 14 Feb 2023, 22:34:17 UTC

My last four of latest batch now finished. I just have to wait another five hours for the last three to finish uploading before they can report.
ID: 68326 · Report as offensive     Reply Quote
Glenn Carver

Send message
Joined: 29 Oct 17
Posts: 1049
Credit: 16,614,827
RAC: 12,088
Message 68327 - Posted: 14 Feb 2023, 22:49:57 UTC - in response to Message 68324.  

Model crashes. There will also be tasks that fail in the model because this experiment is designed to push the limits of what the model can do. If you have a failed task with a very long code traceback in the output, that'll be why. Credit is based on trickles, so credit will be given for work done.
Is this one of them or is it something else? https://www.cpdn.org/result.php?resultid=22311226. I'm wondering as it was easy to tell with Hadley models when failures were due to this reason, not sure what it looks like with OIFS yet.
Yes, that's one of them. There's a line that says
forrtl: error (72): floating overflow
'forrtl' is 'Fortran runtime library'. Floating overflow means a real number became too big to be represented in 64 bits. Further down there's a reference to a subroutine called 'turbulence layer'. In essence the model blew up because the vertical turbulence became too strong. As this is likely with this configuration the model timestep was reduced from its normal 60 mins to 20mins, however, extreme perturbations can still blow the model.
ID: 68327 · Report as offensive     Reply Quote
mikey

Send message
Joined: 18 Nov 18
Posts: 21
Credit: 6,635,794
RAC: 2,524
Message 68329 - Posted: 15 Feb 2023, 1:37:20 UTC - in response to Message 68292.  

Is there much of a chance we will ever see more of the apps being made to run on Windows pc's? As you know currently only the WAH2 apps run on Windows pc's.


I'm working on a Windows version of OpenIFS now. The bulk of the model code is fine, it's the lower level calls to operating system functions which need time to work around because linux is very different to windows in that respect. I can't give you an estimate.


Thank you VERY much, working on it means it will work at some point and that's the whole idea, to get even more pc's in the mix to help some some of these calculations.
ID: 68329 · Report as offensive     Reply Quote
Yeti

Send message
Joined: 5 Aug 04
Posts: 178
Credit: 19,243,550
RAC: 36,469
Message 68331 - Posted: 15 Feb 2023, 12:40:58 UTC - in response to Message 68327.  

Yes, that's one of them. There's a line that says
forrtl: error (72): floating overflow
'forrtl' is 'Fortran runtime library'.
This WU will go the same way: https://www.cpdn.org/workunit.php?wuid=12207576
Supporting BOINC, a great concept !
ID: 68331 · Report as offensive     Reply Quote
ProfileDave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4541
Credit: 19,039,635
RAC: 18,944
Message 68332 - Posted: 15 Feb 2023, 13:22:58 UTC

Two resends one reported, one waiting to report today from the same user. I suspect they both ran into memory problems. Could the push notifications be used to ask users to limit the number of tasks they run concurrently?
this is the computer in question.
ID: 68332 · Report as offensive     Reply Quote
Previous · 1 . . . 21 · 22 · 23 · 24 · 25 · 26 · 27 . . . 32 · Next

Message boards : Number crunching : OpenIFS Discussion

©2024 cpdn.org