climateprediction.net (CPDN) home page
Thread 'transient HTTP error'

Thread 'transient HTTP error'

Message boards : Number crunching : transient HTTP error
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · Next

AuthorMessage
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 59299 - Posted: 4 Jan 2019, 20:34:20 UTC

"cam"s (Central America), and "sam"s (South America), all go to a server in Mexico, which has become a bit notorious for having problems with their upload server.
ID: 59299 · Report as offensive     Reply Quote
ProfileDave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4541
Credit: 19,039,635
RAC: 18,944
Message 59301 - Posted: 5 Jan 2019, 15:07:21 UTC

All my backlog has uploaded. Now to see how long it takes to get credit :)


My credit has updated since this morning so I suspect you will have yours now. Some time earlier about 1.5GB of data finished uploading and cleared my backlog.
ID: 59301 · Report as offensive     Reply Quote
ProfileSteve Dodd

Send message
Joined: 28 Oct 11
Posts: 15
Credit: 9,965,794
RAC: 4,251
Message 59311 - Posted: 6 Jan 2019, 0:25:51 UTC - in response to Message 59301.  
Last modified: 6 Jan 2019, 0:27:11 UTC

Indeed, I have received credit. I'm a happy camper right now :) (although it hasn't shown up on the stats site yet (BOINCStats). I've looked for a permissions check box or something in case of GDPR, but I can't find one.)
ID: 59311 · Report as offensive     Reply Quote
ProfileAlan K

Send message
Joined: 22 Feb 06
Posts: 491
Credit: 31,264,479
RAC: 15,556
Message 59312 - Posted: 6 Jan 2019, 0:35:59 UTC

Looks like there is a problem with upload3:-

05/01/2019 22:47:47 | climateprediction.net | [http] [ID#27] Info: Connection #49 to host upload3.cpdn.org left intact
05/01/2019 22:47:47 | climateprediction.net | [error] Error reported by file upload server: Server is out of disk space
05/01/2019 22:47:47 | climateprediction.net | Temporarily failed upload of wah2_global_e0ry_208812_145_769_011662878_1_r1636291340_125.zip: transient upload error
ID: 59312 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 59313 - Posted: 6 Jan 2019, 1:30:19 UTC

Apparently the relevant people know within 10 minutes if there's a server problem, from various alarms. But getting someone to the problem is a different matter.

I think this one requires someone on site to move large chunks of data from one server to another. After they find one that's not already full.
ID: 59313 · Report as offensive     Reply Quote
ProfileAlan K

Send message
Joined: 22 Feb 06
Posts: 491
Credit: 31,264,479
RAC: 15,556
Message 59314 - Posted: 6 Jan 2019, 10:47:36 UTC - in response to Message 59313.  

Upload11 has the same problem. I would have thought that since they are "virtual" servers they have been allocated a quota of space on an NAS and that increasing this quota is simply a case of logging on to the server and changing a couple of values. But there again that depends on the support staff having remote access.......
ID: 59314 · Report as offensive     Reply Quote
ProfileDave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4541
Credit: 19,039,635
RAC: 18,944
Message 59323 - Posted: 7 Jan 2019, 8:36:12 UTC

And now getting transient upload error on 2 SAFR zips. Project informed.
ID: 59323 · Report as offensive     Reply Quote
ProfileAlan K

Send message
Joined: 22 Feb 06
Posts: 491
Credit: 31,264,479
RAC: 15,556
Message 59337 - Posted: 7 Jan 2019, 23:11:03 UTC - in response to Message 59323.  
Last modified: 7 Jan 2019, 23:14:39 UTC

It looks as if trickles are getting through and being recorded on the database. I have 3 SAFR moels and a PNW model that have registered them today. Does this help:

07/01/2019 22:51:03 | climateprediction.net | [http] [ID#214] Received header from server: <html><head>
07/01/2019 22:51:03 | climateprediction.net | [http] [ID#214] Received header from server: <title>504 Gateway Timeout</title>
07/01/2019 22:51:03 | climateprediction.net | [http] [ID#214] Received header from server: </head><body>
07/01/2019 22:51:03 | climateprediction.net | [http] [ID#214] Received header from server: <h1>Gateway Timeout</h1>
07/01/2019 22:51:03 | climateprediction.net | [http] [ID#214] Received header from server: <p>The gateway did not receive a timely response
07/01/2019 22:51:03 | climateprediction.net | [http] [ID#214] Received header from server: from the upstream server or application.</p>
07/01/2019 22:51:03 | climateprediction.net | [http] [ID#214] Received header from server: <hr>
07/01/2019 22:51:03 | climateprediction.net | [http] [ID#214] Received header from server: <address>Apache/2.4.20 (Unix) Server at upload11.cpdn.org Port 80</address>
07/01/2019 22:51:03 | climateprediction.net | [http] [ID#214] Received header from server: </body></html>
07/01/2019 22:51:03 | climateprediction.net | [http] [ID#214] Info: Closing connection 548
07/01/2019 22:51:03 | climateprediction.net | [file_xfer] http op done; retval -184 (transient HTTP error)
ID: 59337 · Report as offensive     Reply Quote
Tom Miller

Send message
Joined: 26 Mar 18
Posts: 1
Credit: 2,704,020
RAC: 0
Message 59338 - Posted: 7 Jan 2019, 23:44:05 UTC

I have about 15 or 20 tasks across six machines trying to upload.

Does this happen often?
ID: 59338 · Report as offensive     Reply Quote
ProfileIain Inglis
Volunteer moderator

Send message
Joined: 16 Jan 10
Posts: 1084
Credit: 7,884,997
RAC: 4,577
Message 59339 - Posted: 8 Jan 2019, 0:03:58 UTC - in response to Message 59338.  

I have about 15 or 20 tasks across six machines trying to upload.

Does this happen often?


There does seem to be a bit of a problem at the moment because the size of the upload files has increased dramatically. However, it’s not as if the project could not do the maths when releasing a batch and anticipate the storage requirements. So the demand bit I understand but the failure of the supply bit I don’t.
ID: 59339 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 59341 - Posted: 8 Jan 2019, 2:44:36 UTC

Just to clear up the matter of trickles:

These are NOT zip files, and don't appear in the Transfers tab.
They are an RPC.

Also they have their own upload server, which is at Oxford, which is why they (usually), always appear, even when the data files are stuck.
They are the only visible record that an upload has occurred, as there's no list of the more than a dozen zip Upload servers, a lot of which are at Oxford, but also scattered around the planet.
ID: 59341 · Report as offensive     Reply Quote
ccpmc

Send message
Joined: 9 Nov 18
Posts: 3
Credit: 1,117,025
RAC: 0
Message 59342 - Posted: 8 Jan 2019, 3:11:45 UTC
Last modified: 8 Jan 2019, 3:15:58 UTC

I turn the http_debug option on and get the logs as follow:

2019/01/08 11:01:03 | climateprediction.net | Started upload of wah2_safr50_c60f_201212_16_777_011702055_0_r2141728672_5.zip
2019/01/08 11:01:04 | climateprediction.net | [http] [ID#11771] Info: Trying 130.246.191.84...
2019/01/08 11:01:05 | climateprediction.net | [http] [ID#11771] Info: Connected to upload11.cpdn.org (130.246.191.84) port 80 (#3802)
2019/01/08 11:01:05 | climateprediction.net | [http] [ID#11771] Sent header to server: POST /cgi-bin/file_upload_handler HTTP/1.1
2019/01/08 11:01:05 | climateprediction.net | [http] [ID#11771] Sent header to server: Host: upload11.cpdn.org
2019/01/08 11:01:05 | climateprediction.net | [http] [ID#11771] Sent header to server: User-Agent: BOINC client (windows_x86_64 7.14.2)
2019/01/08 11:01:05 | climateprediction.net | [http] [ID#11771] Sent header to server: Accept: */*
2019/01/08 11:01:05 | climateprediction.net | [http] [ID#11771] Sent header to server: Accept-Encoding: deflate, gzip
2019/01/08 11:01:05 | climateprediction.net | [http] [ID#11771] Sent header to server: Content-Type: application/x-www-form-urlencoded
2019/01/08 11:01:05 | climateprediction.net | [http] [ID#11771] Sent header to server: Accept-Language: zh_CN
2019/01/08 11:01:05 | climateprediction.net | [http] [ID#11771] Sent header to server: Content-Length: 313
2019/01/08 11:01:05 | climateprediction.net | [http] [ID#11771] Sent header to server:
2019/01/08 11:01:05 | climateprediction.net | [http] [ID#11771] Info: We are completely uploaded and fine
2019/01/08 11:01:05 | climateprediction.net | [http] [ID#11771] Received header from server: HTTP/1.1 200 OK
2019/01/08 11:01:05 | climateprediction.net | [http] [ID#11771] Received header from server: Date: Tue, 08 Jan 2019 03:01:07 GMT
2019/01/08 11:01:05 | climateprediction.net | [http] [ID#11771] Received header from server: Server: Apache/2.4.20 (Unix)
2019/01/08 11:01:05 | climateprediction.net | [http] [ID#11771] Received header from server: Transfer-Encoding: chunked
2019/01/08 11:01:05 | climateprediction.net | [http] [ID#11771] Received header from server: Content-Type: text/plain
2019/01/08 11:01:05 | climateprediction.net | [http] [ID#11771] Received header from server:
2019/01/08 11:01:05 | climateprediction.net | [http] [ID#11771] Received header from server: 64
2019/01/08 11:01:05 | climateprediction.net | [http] [ID#11771] Received header from server: <data_server_reply>
2019/01/08 11:01:05 | climateprediction.net | [http] [ID#11771] Received header from server: <status>0</status>
2019/01/08 11:01:05 | climateprediction.net | [http] [ID#11771] Received header from server: <file_size>13385728</file_size>
2019/01/08 11:01:05 | climateprediction.net | [http] [ID#11771] Received header from server: </data_server_reply>
2019/01/08 11:01:05 | climateprediction.net | [http] [ID#11771] Received header from server:
2019/01/08 11:01:05 | climateprediction.net | [http] [ID#11771] Received header from server: 0
2019/01/08 11:01:05 | climateprediction.net | [http] [ID#11771] Received header from server:
2019/01/08 11:01:05 | climateprediction.net | [http] [ID#11771] Info: Connection #3802 to host upload11.cpdn.org left intact
2019/01/08 11:01:06 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle set
2019/01/08 11:01:06 | climateprediction.net | [http] [ID#11771] Info: Found bundle for host upload11.cpdn.org: 0x49d2860 [can pipeline]
2019/01/08 11:01:06 | climateprediction.net | [http] [ID#11771] Info: Re-using existing connection! (#3802) with host upload11.cpdn.org
2019/01/08 11:01:06 | climateprediction.net | [http] [ID#11771] Info: Connected to upload11.cpdn.org (130.246.191.84) port 80 (#3802)
2019/01/08 11:01:06 | climateprediction.net | [http] [ID#11771] Sent header to server: POST /cgi-bin/file_upload_handler HTTP/1.1
2019/01/08 11:01:06 | climateprediction.net | [http] [ID#11771] Sent header to server: Host: upload11.cpdn.org
2019/01/08 11:01:06 | climateprediction.net | [http] [ID#11771] Sent header to server: User-Agent: BOINC client (windows_x86_64 7.14.2)
2019/01/08 11:01:06 | climateprediction.net | [http] [ID#11771] Sent header to server: Accept: */*
2019/01/08 11:01:06 | climateprediction.net | [http] [ID#11771] Sent header to server: Accept-Encoding: deflate, gzip
2019/01/08 11:01:06 | climateprediction.net | [http] [ID#11771] Sent header to server: Content-Type: application/x-www-form-urlencoded
2019/01/08 11:01:06 | climateprediction.net | [http] [ID#11771] Sent header to server: Accept-Language: zh_CN
2019/01/08 11:01:06 | climateprediction.net | [http] [ID#11771] Sent header to server: Content-Length: 88159713
2019/01/08 11:01:06 | climateprediction.net | [http] [ID#11771] Sent header to server: Expect: 100-continue
2019/01/08 11:01:06 | climateprediction.net | [http] [ID#11771] Sent header to server:
2019/01/08 11:01:06 | climateprediction.net | [http] [ID#11771] Received header from server: HTTP/1.1 100 Continue
2019/01/08 11:07:06 | climateprediction.net | [http] [ID#11771] Info: We are completely uploaded and fine
2019/01/08 11:07:06 | climateprediction.net | [http] [ID#11771] Received header from server: HTTP/1.1 200 OK
2019/01/08 11:07:06 | climateprediction.net | [http] [ID#11771] Received header from server: Date: Tue, 08 Jan 2019 03:01:08 GMT
2019/01/08 11:07:06 | climateprediction.net | [http] [ID#11771] Received header from server: Server: Apache/2.4.20 (Unix)
2019/01/08 11:07:06 | climateprediction.net | [http] [ID#11771] Received header from server: Transfer-Encoding: chunked
2019/01/08 11:07:06 | climateprediction.net | [http] [ID#11771] Received header from server: Content-Type: text/plain
2019/01/08 11:07:06 | climateprediction.net | [http] [ID#11771] Received header from server:
2019/01/08 11:07:06 | climateprediction.net | [http] [ID#11771] Received header from server: 8a
2019/01/08 11:07:06 | climateprediction.net | [http] [ID#11771] Received header from server: <data_server_reply>
2019/01/08 11:07:06 | climateprediction.net | [http] [ID#11771] Received header from server: <status>1</status>
2019/01/08 11:07:06 | climateprediction.net | [http] [ID#11771] Received header from server: <message>EOF on socket read : asked for 262144, got 132524
2019/01/08 11:07:06 | climateprediction.net | [http] [ID#11771] Received header from server: </message>
2019/01/08 11:07:06 | climateprediction.net | [http] [ID#11771] Received header from server: </data_server_reply>
2019/01/08 11:07:06 | climateprediction.net | [http] [ID#11771] Received header from server:
2019/01/08 11:07:06 | climateprediction.net | [http] [ID#11771] Received header from server: 0
2019/01/08 11:07:06 | climateprediction.net | [http] [ID#11771] Received header from server:
2019/01/08 11:07:06 | climateprediction.net | [http] [ID#11771] Info: Connection #3802 to host upload11.cpdn.org left intact
2019/01/08 11:07:07 | climateprediction.net | [error] Error reported by file upload server: EOF on socket read : asked for 262144, got 132524
2019/01/08 11:07:07 | climateprediction.net | Temporarily failed upload of wah2_safr50_c60f_201212_16_777_011702055_0_r2141728672_5.zip: transient upload error
ID: 59342 · Report as offensive     Reply Quote
Jean-David Beyer

Send message
Joined: 5 Aug 04
Posts: 1120
Credit: 17,202,915
RAC: 2,154
Message 59343 - Posted: 8 Jan 2019, 3:56:34 UTC - in response to Message 59341.  
Last modified: 8 Jan 2019, 3:57:29 UTC

Just to clear up the matter of trickles:

These are NOT zip files, and don't appear in the Transfers tab.
They are an RPC.


Mine, on Linux, look like zip files and they DO appear in the Transfers tab:

Mon 07 Jan 2019 06:32:57 PM EST | climateprediction.net | Temporarily failed upload of hadcm3s_x5300_190012_60_771_011668342_2_r376222488_1.zip: transient HTTP error
Mon 07 Jan 2019 06:32:57 PM EST | climateprediction.net | Backing off 01:35:41 on upload of hadcm3s_x5300_190012_60_771_011668342_2_r376222488_1.zip
Mon 07 Jan 2019 06:32:58 PM EST | | Internet access OK - project servers may be temporarily down.

Talking about this work unit:

Name hadcm3s_x5300_190012_60_771_011668342_2
Workunit 11668342

It has successfully uploaded the only trickle I am likely to get, but it does not realize it and continually tries to upload it over and over. It has now spent 2:04:22 uploading this file over and over again.
ID: 59343 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 59346 - Posted: 8 Jan 2019, 5:58:22 UTC

You're talking about a zip file, not a trickle_up.
Two different things.

The only place that you can see the trickle_up files is in the climateprediction.net folder on the computer.
And even then, you have to turn off Network access in the BOINC manager, and wait until there's a message in the Projects tab, saying: Trickle_up waiting. Then go and look in the folder, where you'll see both Trickle_ups and zip files.
(I think that the file type for Trickle_ups is "xml".)

And the project people are well aware of the upload problem, and are moving data off the Upload servers to various NASs.
ID: 59346 · Report as offensive     Reply Quote
Jean-David Beyer

Send message
Joined: 5 Aug 04
Posts: 1120
Credit: 17,202,915
RAC: 2,154
Message 59350 - Posted: 8 Jan 2019, 13:18:47 UTC - in response to Message 59346.  

You're talking about a zip file, not a trickle_up.
Two different things.


Then what is that zip file? Until a zip file like that is uploaded, typically after about 24 hours are expended by the application, I get no credits for trickles, but pretty soon after one of those zip files gets uploaded, I do get credit.

And in the last year or so, I get only one of those files produced and uploaded, even though the work units continue to run for about a week. They use lots more CPU time, but get no more trickles and no more of these uploads. In 2018, I have received only hadcm3s work units, and some fail and some complete.
ID: 59350 · Report as offensive     Reply Quote
ProfileIain Inglis
Volunteer moderator

Send message
Joined: 16 Jan 10
Posts: 1084
Credit: 7,884,997
RAC: 4,577
Message 59351 - Posted: 8 Jan 2019, 13:36:29 UTC - in response to Message 59350.  

You're talking about a zip file, not a trickle_up.
Two different things.


Then what is that zip file? ...


Here, for example, are the first two trickle-related upload files from a current model of mine, and the "restart" and "out" files (which are also Zip files):

trickle_up_wah2_sas50_a010_200812_22_778_011703060_0_1546886195.xml
trickle_up_wah2_sas50_a010_200812_22_778_011703060_0_1546901393.xml
...

wah2_sas50_a010_200812_22_778_011703060_0_r1212817215_1.zip
wah2_sas50_a010_200812_22_778_011703060_0_r1212817215_2.zip
...

wah2_sas50_a010_200812_22_778_011703060_0_r1212817215_out.zip
wah2_sas50_a010_200812_22_778_011703060_0_r1212817215_restart.zip


There will be 22 of the pair of trickle-related files (XML and Zip) for that particular model type.

As far as I know, the credits are awarded for the "trickle" files - i.e. the XML files - but the science for all current model types is in the trickle-related Zip files and the final restart file.
ID: 59351 · Report as offensive     Reply Quote
ProfileDave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4541
Credit: 19,039,635
RAC: 18,944
Message 59352 - Posted: 8 Jan 2019, 13:55:35 UTC

And in the last year or so, I get only one of those files produced and uploaded, even though the work units continue to run for about a week. They use lots more CPU time, but get no more trickles and no more of these uploads. In 2018, I have received only hadcm3s work units, and some fail and some complete.


There is an issue with the trickle up files for HADCM3S tasks which is one of the many issues being worked on by the project. (There is I believe a plan to rewrite some/all of this code to address this and other issues.) Only the first trickle up shows on the web site. The trickle up messages are still being sent according to my event log but not showing on website and credit not being granted as far as I can see.

Fortunately, I am not particularly bothered about credit other than as part of checking things are working.
ID: 59352 · Report as offensive     Reply Quote
ccpmc

Send message
Joined: 9 Nov 18
Posts: 3
Credit: 1,117,025
RAC: 0
Message 59354 - Posted: 8 Jan 2019, 15:33:11 UTC

When will the problem be fixed?
ID: 59354 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 59355 - Posted: 8 Jan 2019, 15:36:55 UTC

The future is a strange place. You'll just have to wait and see.
ID: 59355 · Report as offensive     Reply Quote
ProfileDave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4541
Credit: 19,039,635
RAC: 18,944
Message 59357 - Posted: 8 Jan 2019, 17:03:45 UTC - in response to Message 59355.  
Last modified: 8 Jan 2019, 17:06:58 UTC

And HADCM3S zips also failing at 100%. Was vaguely hoping they were going somewhere else so would get through. Obviously not!

Edit:Setting both machines to suspend network activity till tomorrow.
ID: 59357 · Report as offensive     Reply Quote
Previous · 1 · 2 · 3 · 4 · Next

Message boards : Number crunching : transient HTTP error

©2024 cpdn.org