Name | famous_u1kl_1599_200_006634880_5 |
Workunit | 6838252 |
Created | 22 Jul 2010, 14:46:34 UTC |
Sent | 22 Jul 2010, 15:46:40 UTC |
Report deadline | 21 Oct 2010, 23:13:51 UTC |
Received | 11 Sep 2010, 15:15:56 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 22 (0x00000016) Unknown error code |
Computer ID | 1297049 |
Run time | 7 days 1 hours 7 min |
CPU time | 7 days 4 hours 16 min 30 sec |
Validate state | Invalid |
Credit | 3,057.37 |
Device peak FLOPS | 1.69 GFLOPS |
Application version | UK Met Office FAMOUS v6.11 i686-pc-linux-gnu |
Stderr | <core_client_version>6.10.56</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=19704, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=19704, iMonCtr=1 Model crash detected, will try to restart... (19704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=21630, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=21630, iMonCtr=1 Model crash detected, will try to restart... (23040): Can't acquire lockfile (-154) - waiting 35s (21630): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=23040, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=23040, iMonCtr=1 Model crash detected, will try to restart... (23040): No heartbeat from core client for 30 sec - exiting (24491): Can't acquire lockfile (-154) - waiting 35s CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=24491, iMonCtr=1 Model crash detected, will try to restart... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=24491, iMonCtr=1 Model crash detected, will try to restart... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 (24491): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=25908, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=25908, iMonCtr=1 Model crash detected, will try to restart... (28698): Can't acquire lockfile (-154) - waiting 35s (25908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... (28698): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=30580, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=30580, iMonCtr=1 Model crash detected, will try to restart... (9414): Can't acquire lockfile (-154) - waiting 35s (30580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9414, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9414, iMonCtr=1 Model crash detected, will try to restart... (11089): Can't acquire lockfile (-154) - waiting 35s Suspended CPDN Monitor - Suspend request from BOINC... (11089): Can't acquire lockfile (-154) - exiting (11098): Can't acquire lockfile (-154) - waiting 35s Suspended CPDN Monitor - Suspend request from BOINC... (11098): Can't acquire lockfile (-154) - exiting (9414): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... (11201): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=13061, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=13061, iMonCtr=1 Model crash detected, will try to restart... (14420): Can't acquire lockfile (-154) - waiting 35s (13061): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=14420, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=14420, iMonCtr=1 Model crash detected, will try to restart... (17639): Can't acquire lockfile (-154) - waiting 35s (14420): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17639, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17639, iMonCtr=1 Model crash detected, will try to restart... (17639): No heartbeat from core client for 30 sec - exiting (25589): Can't acquire lockfile (-154) - waiting 35s CPDN Monitor - No 'heartbeat' from BOINC... (25589): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (25589): No heartbeat from core client for 30 sec - exiting (25589): No heartbeat from core client for 30 sec - exiting (25589): No heartbeat from core client for 30 sec - exiting (25589): No heartbeat from core client for 30 sec - exiting (25589): No heartbeat from core client for 30 sec - exiting (25589): No heartbeat from core client for 30 sec - exiting (25589): No heartbeat from core client for 30 sec - exiting (25589): No heartbeat from core client for 30 sec - exiting (25589): No heartbeat from core client for 30 sec - exiting (25589): No heartbeat from core client for 30 sec - exiting (25589): No heartbeat from core client for 30 sec - exiting (25589): No heartbeat from core client for 30 sec - exiting (25589): No heartbeat from core client for 30 sec - exiting (25589): No heartbeat from core client for 30 sec - exiting (25589): No heartbeat from core client for 30 sec - exiting (25589): No heartbeat from core client for 30 sec - exiting (25589): No heartbeat from core client for 30 sec - exiting (25589): No heartbeat from core client for 30 sec - exiting (25589): No heartbeat from core client for 30 sec - exiting (25589): No heartbeat from core client for 30 sec - exiting (25589): No heartbeat from core client for 30 sec - exiting (25589): No heartbeat from core client for 30 sec - exiting (10686): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... (10787): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... (7099): Can't acquire lockfile (-154) - waiting 35s (7099): Can't acquire lockfile (-154) - exiting (23779): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... (7202): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... (19009): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=13884, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=13884, iMonCtr=1 Model crash detected, will try to restart... (15431): Can't acquire lockfile (-154) - waiting 35s (13884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... (15431): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... (4818): Can't acquire lockfile (-154) - waiting 35s (2394): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... (17474): Can't acquire lockfile (-154) - waiting 35s (16070): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... (17474): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... (30030): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (30030): No heartbeat from core client for 30 sec - exiting (30030): No heartbeat from core client for 30 sec - exiting (30030): No heartbeat from core client for 30 sec - exiting (30030): No heartbeat from core client for 30 sec - exiting (30030): No heartbeat from core client for 30 sec - exiting (30030): No heartbeat from core client for 30 sec - exiting (30030): No heartbeat from core client for 30 sec - exiting (30030): No heartbeat from core client for 30 sec - exiting (30030): No heartbeat from core client for 30 sec - exiting (30030): No heartbeat from core client for 30 sec - exiting (30030): No heartbeat from core client for 30 sec - exiting (30030): No heartbeat from core client for 30 sec - exiting (30030): No heartbeat from core client for 30 sec - exiting (30030): No heartbeat from core client for 30 sec - exiting (30030): No heartbeat from core client for 30 sec - exiting (30030): No heartbeat from core client for 30 sec - exiting (30030): No heartbeat from core client for 30 sec - exiting (30030): No heartbeat from core client for 30 sec - exiting (30030): No heartbeat from core client for 30 sec - exiting (30030): No heartbeat from core client for 30 sec - exiting (30030): No heartbeat from core client for 30 sec - exiting (30030): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... (32275): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... (23412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting (23412): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... (23638): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (23638): No heartbeat from core client for 30 sec - exiting (23638): No heartbeat from core client for 30 sec - exiting (23638): No heartbeat from core client for 30 sec - exiting (23638): No heartbeat from core client for 30 sec - exiting (23638): No heartbeat from core client for 30 sec - exiting (23638): No heartbeat from core client for 30 sec - exiting (23638): No heartbeat from core client for 30 sec - exiting (23638): No heartbeat from core client for 30 sec - exiting (23638): No heartbeat from core client for 30 sec - exiting (23638): No heartbeat from core client for 30 sec - exiting (23638): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... (12766): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... (17124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... (7410): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (5743): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (5771): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (5988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (6003): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (6223): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (6248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (6266): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (6612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (6628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (2318): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... (12300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (12524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (12553): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 21 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 21 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 21 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 21 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 21 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 21 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy Sorry, too many model crashes! :-( (12989): called boinc_finish </stderr_txt> ]]> |
Latest Trickles Received | ||||||
---|---|---|---|---|---|---|
Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
11 Sep 2010 14:17:47 | 779809 | 11613533 | famous_u1kl_1599_200_006634880_5 | 926,666 | 619,422 | 0.6684 |
11 Sep 2010 12:31:08 | 779809 | 11613533 | famous_u1kl_1599_200_006634880_5 | 917,306 | 613,196 | 0.6685 |
11 Sep 2010 10:43:19 | 779809 | 11613533 | famous_u1kl_1599_200_006634880_5 | 907,946 | 606,974 | 0.6685 |
11 Sep 2010 08:59:48 | 779809 | 11613533 | famous_u1kl_1599_200_006634880_5 | 898,586 | 600,749 | 0.6685 |
11 Sep 2010 07:17:14 | 779809 | 11613533 | famous_u1kl_1599_200_006634880_5 | 889,226 | 594,539 | 0.6686 |
11 Sep 2010 05:31:38 | 779809 | 11613533 | famous_u1kl_1599_200_006634880_5 | 879,866 | 588,338 | 0.6687 |
11 Sep 2010 03:50:30 | 779809 | 11613533 | famous_u1kl_1599_200_006634880_5 | 870,506 | 582,129 | 0.6687 |
11 Sep 2010 02:03:34 | 779809 | 11613533 | famous_u1kl_1599_200_006634880_5 | 861,146 | 575,920 | 0.6688 |
11 Sep 2010 00:20:32 | 779809 | 11613533 | famous_u1kl_1599_200_006634880_5 | 851,786 | 569,700 | 0.6688 |
10 Sep 2010 22:19:28 | 779809 | 11613533 | famous_u1kl_1599_200_006634880_5 | 842,426 | 563,444 | 0.6688 |
10 Sep 2010 20:34:13 | 779809 | 11613533 | famous_u1kl_1599_200_006634880_5 | 833,066 | 557,185 | 0.6688 |
10 Sep 2010 18:50:23 | 779809 | 11613533 | famous_u1kl_1599_200_006634880_5 | 823,706 | 550,909 | 0.6688 |
10 Sep 2010 17:03:16 | 779809 | 11613533 | famous_u1kl_1599_200_006634880_5 | 814,346 | 544,641 | 0.6688 |
10 Sep 2010 15:19:51 | 779809 | 11613533 | famous_u1kl_1599_200_006634880_5 | 804,986 | 538,358 | 0.6688 |
10 Sep 2010 13:35:16 | 779809 | 11613533 | famous_u1kl_1599_200_006634880_5 | 795,626 | 532,066 | 0.6687 |
10 Sep 2010 11:51:32 | 779809 | 11613533 | famous_u1kl_1599_200_006634880_5 | 786,266 | 525,819 | 0.6688 |
10 Sep 2010 10:07:12 | 779809 | 11613533 | famous_u1kl_1599_200_006634880_5 | 776,906 | 519,573 | 0.6688 |
10 Sep 2010 08:20:03 | 779809 | 11613533 | famous_u1kl_1599_200_006634880_5 | 767,546 | 513,312 | 0.6688 |
10 Sep 2010 06:37:18 | 779809 | 11613533 | famous_u1kl_1599_200_006634880_5 | 758,186 | 507,055 | 0.6688 |
10 Sep 2010 04:51:55 | 779809 | 11613533 | famous_u1kl_1599_200_006634880_5 | 748,826 | 500,801 | 0.6688 |
©2024 climateprediction.net