Task 15691665

Name	hadcm3n_4ifw_1980_40_008340504_0
Workunit	8491365
Created	29 Mar 2013, 2:32:26 UTC
Sent	29 Mar 2013, 11:36:58 UTC
Report deadline	28 Jun 2013, 19:04:09 UTC
Received	20 Apr 2013, 23:05:38 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-226 (0xFFFFFF1E) ERR_TOO_MANY_EXITS
Computer ID	1091586
Run time	15 days 20 hours 19 min 51 sec
CPU time	13 days 4 hours 58 min 59 sec
Validate state	Invalid
Credit	4,665.60
Device peak FLOPS	1.77 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> too many exit(0)s </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11900, iMonCtr=1 Model crash detected, will try to restart... C19:10:59 (10280): No heartbeat from core client for 30 sec - exiting 19:11:00 (10280): No heartbeat from core client for 30 sec - exiting 19:11:01 (10280): No heartbeat from core client for 30 sec - exiting 19:11:02 (10280): No heartbeat from core client for 30 sec - exiting 19:11:03 (10280): No heartbeat from core client for 30 sec - exiting 19:11:05 (10280): No heartbeat from core client for 30 sec - exiting 19:11:06 (10280): No heartbeat from core client for 30 sec - exiting 19:11:07 (10280): No heartbeat from core client for 30 sec - exiting 19:11:08 (10280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7000, iMonCtr=1 Model crash detected, will try to restart... 22:35:38 (11172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:12:53 (10928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:21:52 (11812): No heartbeat from core client for 30 sec - exiting 14:21:53 (11812): No heartbeat from core client for 30 sec - exiting 14:21:54 (11812): No heartbeat from core client for 30 sec - exiting 14:21:55 (11812): No heartbeat from core client for 30 sec - exiting 14:21:56 (11812): No heartbeat from core client for 30 sec - exiting 14:21:57 (11812): No heartbeat from core client for 30 sec - exiting 14:21:58 (11812): No heartbeat from core client for 30 sec - exiting 14:21:59 (11812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11640, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 06:32:18 (11528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:32:19 (11528): No heartbeat from core client for 30 sec - exiting 06:32:20 (11528): No heartbeat from core client for 30 sec - exiting 06:32:21 (11528): No heartbeat from core client for 30 sec - exiting 06:32:22 (11528): No heartbeat from core client for 30 sec - exiting 06:32:23 (11528): No heartbeat from core client for 30 sec - exiting 06:32:24 (11528): No heartbeat from core client for 30 sec - exiting 06:32:25 (11528): No heartbeat from core client for 30 sec - exiting 06:32:26 (11528): No heartbeat from core client for 30 sec - exiting 06:32:27 (11528): No heartbeat from core client for 30 sec - exiting 06:32:28 (11528): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 22:53:23 (11916): No heartbeat from core client for 30 sec - exiting 22:53:25 (11916): No heartbeat from core client for 30 sec - exiting 22:53:26 (11916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/4ifwko.pjj1c10 Error converting file to netcdf: dataout/4ifwko.pij1c10 Error converting file to netcdf: dataout/4ifwko.pfj1c10 Error converting file to netcdf: dataout/4ifwka.phj1c10 Error converting file to netcdf: dataout/4ifwka.pgj1c10 Error converting file to netcdf: dataout/4ifwka.pej1c10 Error converting file to netcdf: dataout/4ifwka.pdj1c10 Suspended CPDN Monitor - Suspend request from BOINC... 20:37:55 (9336): No heartbeat from core client for 30 sec - exiting 20:37:57 (9336): No heartbeat from core client for 30 sec - exiting 20:37:58 (9336): No heartbeat from core client for 30 sec - exiting 20:37:59 (9336): No heartbeat from core client for 30 sec - exiting 20:38:00 (9336): No heartbeat from core client for 30 sec - exiting 20:38:01 (9336): No heartbeat from core client for 30 sec - exiting 20:38:02 (9336): No heartbeat from core client for 30 sec - exiting 20:38:03 (9336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold 17:27:36 (11928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:19:07 (11944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 Apr 2013 06:58:17	1091586	15691665	hadcm3n_4ifw_1980_40_008340504_0	388,800	1,095,209	2.8169
19 Apr 2013 02:26:34	1091586	15691665	hadcm3n_4ifw_1980_40_008340504_0	362,880	1,022,155	2.8168
17 Apr 2013 16:31:09	1091586	15691665	hadcm3n_4ifw_1980_40_008340504_0	336,960	948,431	2.8147
15 Apr 2013 15:46:09	1091586	15691665	hadcm3n_4ifw_1980_40_008340504_0	311,040	875,055	2.8133
13 Apr 2013 04:27:01	1091586	15691665	hadcm3n_4ifw_1980_40_008340504_0	285,120	800,747	2.8085
12 Apr 2013 03:02:20	1091586	15691665	hadcm3n_4ifw_1980_40_008340504_0	259,200	725,228	2.7979
11 Apr 2013 02:11:58	1091586	15691665	hadcm3n_4ifw_1980_40_008340504_0	233,280	650,375	2.7880
09 Apr 2013 07:09:24	1091586	15691665	hadcm3n_4ifw_1980_40_008340504_0	207,360	575,990	2.7777
08 Apr 2013 03:04:57	1091586	15691665	hadcm3n_4ifw_1980_40_008340504_0	181,440	502,505	2.7695
07 Apr 2013 00:09:57	1091586	15691665	hadcm3n_4ifw_1980_40_008340504_0	155,520	426,201	2.7405
06 Apr 2013 01:15:13	1091586	15691665	hadcm3n_4ifw_1980_40_008340504_0	129,600	355,280	2.7414
03 Apr 2013 15:20:41	1091586	15691665	hadcm3n_4ifw_1980_40_008340504_0	103,680	291,137	2.8080
02 Apr 2013 08:58:41	1091586	15691665	hadcm3n_4ifw_1980_40_008340504_0	77,760	225,198	2.8961
31 Mar 2013 16:36:09	1091586	15691665	hadcm3n_4ifw_1980_40_008340504_0	51,840	151,899	2.9302
30 Mar 2013 11:55:25	1091586	15691665	hadcm3n_4ifw_1980_40_008340504_0	25,920	74,916	2.8903