climateprediction.net home page
Errors with team description encoding

Errors with team description encoding

Message boards : Number crunching : Errors with team description encoding
Message board moderation

To post messages, you must log in.

AuthorMessage
Profile old_user555153

Send message
Joined: 30 Jan 09
Posts: 2
Credit: 12,087
RAC: 0
Message 37747 - Posted: 14 Aug 2009, 9:28:21 UTC
Last modified: 14 Aug 2009, 9:47:14 UTC

after todays database errors,

http://climateapps2.oucs.ox.ac.uk/cpdnboinc/team_display.php?teamid=188

Добро пожаловать в Украинскую Команду Распределенных Вычислений!

Заходите на наш командный форум по проекту ClimatePrediction.net !
ID: 37747 · Report as offensive     Reply Quote
Profile mo.v
Volunteer moderator
Avatar

Send message
Joined: 29 Sep 04
Posts: 2363
Credit: 14,611,758
RAC: 0
Message 37767 - Posted: 14 Aug 2009, 21:06:02 UTC

The CPDN programmers have been informed. Thank you for reporting this problem.
Cpdn news
ID: 37767 · Report as offensive     Reply Quote
Profile Milo Thurston
Volunteer moderator
Volunteer developer

Send message
Joined: 2 Mar 06
Posts: 253
Credit: 363,646
RAC: 0
Message 37821 - Posted: 18 Aug 2009, 12:47:39 UTC

Thanks, this should now be fixed:

Добро пожаловать в Украинскую Команду Распределенных Вычислений!

If any more examples of incorrect encoding appear please let me know.
ID: 37821 · Report as offensive     Reply Quote
Profile mo.v
Volunteer moderator
Avatar

Send message
Joined: 29 Sep 04
Posts: 2363
Credit: 14,611,758
RAC: 0
Message 37832 - Posted: 18 Aug 2009, 22:34:49 UTC

Milo, I\'m afraid that profiles still don\'t all display properly. For example, Chinese characters display in this member\'s name here but not in his profile.

Similarly with a Russian member here in the list and in his profile
Cpdn news
ID: 37832 · Report as offensive     Reply Quote
Profile Milo Thurston
Volunteer moderator
Volunteer developer

Send message
Joined: 2 Mar 06
Posts: 253
Credit: 363,646
RAC: 0
Message 37838 - Posted: 19 Aug 2009, 10:35:02 UTC - in response to Message 37832.  
Last modified: 19 Aug 2009, 10:44:31 UTC

Milo, I\'m afraid that profiles still don\'t all display properly.


To fix the last one I had to track down the particular table/column at fault and convert from latin1->binary->utf8. Apparently there were around 500 records there that needed converting. I\'ll track down and convert these ones as well.

Edit: Hmmm... It seems that this Russian one is corrupted by some other means, although the Chinese was convertable.
ID: 37838 · Report as offensive     Reply Quote
DJStarfox

Send message
Joined: 27 Jan 07
Posts: 300
Credit: 3,288,263
RAC: 26,370
Message 37854 - Posted: 20 Aug 2009, 14:11:29 UTC - in response to Message 37838.  

To fix the last one I had to track down the particular table/column at fault and convert from latin1->binary->utf8. Apparently there were around 500 records there that needed converting. I\'ll track down and convert these ones as well.
Edit: Hmmm... It seems that this Russian one is corrupted by some other means, although the Chinese was convertable.


Milo:
UTF-8 should work. Did you see this article? I hope it helps you.
http://www.oreillynet.com/onlamp/blog/2006/01/turning_mysql_data_in_latin1_t.html
ID: 37854 · Report as offensive     Reply Quote
Profile Milo Thurston
Volunteer moderator
Volunteer developer

Send message
Joined: 2 Mar 06
Posts: 253
Credit: 363,646
RAC: 0
Message 37860 - Posted: 21 Aug 2009, 9:45:52 UTC - in response to Message 37854.  


UTF-8 should work. Did you see this article? I hope it helps you.
http://www.oreillynet.com/onlamp/blog/2006/01/turning_mysql_data_in_latin1_t.html


There\'s a problem with that article\'s method - it would involve another complete dump of the database which would require shutting it down overnight. So, if I can find a way to change the few remaining entries without doing that it would be greatly preferable. I haven\'t found it yet.
ID: 37860 · Report as offensive     Reply Quote
DJStarfox

Send message
Joined: 27 Jan 07
Posts: 300
Credit: 3,288,263
RAC: 26,370
Message 37863 - Posted: 21 Aug 2009, 14:14:20 UTC - in response to Message 37860.  

There\'s a problem with that article\'s method - it would involve another complete dump of the database which would require shutting it down overnight. So, if I can find a way to change the few remaining entries without doing that it would be greatly preferable. I haven\'t found it yet.


I think that it would be crazy to dump the DB again. The only way I would know how to fix them would be to have the old database up (isolated of course), and then re-convert the affected records individually with a custom SQL script.

Hopefully, if the moderators can find a good sample of the people affected, you\'ll be able to find a perfect WHERE clause to identify them. Then, hopefully, you\'ll be able to write One Script To fix Them All.

There were some additional user comments on that article, such as configuration for the php.ini, httpd.conf, etc that would be good to verify, so this shouldn\'t be an issue going forward.

I\'m sure this was many, many hours to pull off an encoding change, so I congratulate you on only having these minor issues.
ID: 37863 · Report as offensive     Reply Quote
old_user1
Avatar

Send message
Joined: 5 Aug 04
Posts: 907
Credit: 299,864
RAC: 0
Message 37865 - Posted: 21 Aug 2009, 22:17:22 UTC - in response to Message 37863.  

It also depends on the web browser encoding someone uses, for example some of those links work if I select \"Cyrillic - Windows\" on my Mac Safari browser, but the \"Default\" encoding (unicode I guess) it is gibberish.
ID: 37865 · Report as offensive     Reply Quote
Profile Milo Thurston
Volunteer moderator
Volunteer developer

Send message
Joined: 2 Mar 06
Posts: 253
Credit: 363,646
RAC: 0
Message 37881 - Posted: 23 Aug 2009, 17:44:37 UTC - in response to Message 37865.  

It also depends on the web browser encoding someone uses, for example some of those links work if I select \"Cyrillic - Windows\" on my Mac Safari browser, but the \"Default\" encoding (unicode I guess) it is gibberish.


Indeed - it looks like some entries have gone in in CP1251 (or others) and been re-encoded into latin1 by MySQL. The ideal solution would be to track these down and convert them individually, which I have done for some entries. The ones that remain are ones that I have not been able to successfully convert yet. The particular Russian one discussed here seems resistant to being corrected within the database, although it may be viewed in the browser as you describe.
ID: 37881 · Report as offensive     Reply Quote

Message boards : Number crunching : Errors with team description encoding

©2024 climateprediction.net