Wierdly lost utf8 characters when inserting into the database

[eluser]Nameless One[/eluser]
I'm using CodeIgniter's ActiveRecord with MySQL database. My project uses utf8 character encoding. My $config['global_xss_filtering'] is set to TRUE.

When I enter the character 'š' in one of my forms, it is strangely lost when the data is inserted into the database. Even worse, if 'š' is in the middle of the string, the character and everything after it is lost. Character 'č' is correctly inserted, but I've noticed that it is converted into č html entity. When I output the post data with $this->input->post(), it displays correctly both with 'č' and 'š'.

Does anyone know what causes this? Is it the global xss filtering or ActiveRecord? The only difference between 'č' and 'š' that I know of is that 'š' exists in ASCII and 'č' doesn't. Is there any need for this kind of filtering, anyway? How does a non-AScII character make your application vulnerable if it is not converted to a html entity?

[eluser]TheFuzzy0ne[/eluser]
What's your database table encoding set to?

[eluser]Nameless One[/eluser]
The collation in the database is utf8_general_ci. In config/database.php, I have set:

Code:

$db['admin']['char_set'] = "utf8";
$db['admin']['dbcollat'] = "utf8_general_ci";

I hope this automatically does SET NAMES 'utf8'. I've been using utf8_general_ci and SET NAMES 'utf8' for years without problems. I also tried to turn off the global xss filtering and it didn't help, so I guess ActiveRecord is the problem.

[eluser]TheFuzzy0ne[/eluser]
It's unlikely that AR is the cause of the problem, as the only thing it does to your data, is escape it before inserting it into the database. Can you confirm it's actually stored incorrectly in the database, or is it just that it's not displaying correctly in the output page?

If you add this to your controller:

Code:

$this->output->enable_profiler(TRUE);

You will get a printout at the bottom of each page showing you what queries have been fired at your database. This won't work, however, if you redirect to another page after the insert.

[eluser]Nameless One[/eluser]
[quote author="TheFuzzy0ne" date="1239039293"]It's unlikely that AR is the cause of the problem, as the only thing it does to your data, is escape it before inserting it into the database. Can you confirm it's actually stored incorrectly in the database, or is it just that it's not displaying correctly in the output page?[/quote]

I already mentioned in my first post that I checked the database.

[quote author="TheFuzzy0ne" date="1239039293"]
If you add this to your controller:

Code:

$this->output->enable_profiler(TRUE);

You will get a printout at the bottom of each page showing you what queries have been fired at your database. This won't work, however, if you redirect to another page after the insert.[/quote]

Thank's for the suggestion. I read a little about the profiler but I had no idea it actually outputs the queries that were ran. The strings with 'š' seem to be correct in the query. The query output by the profiler works correctly in phpMyAdmin. Now I have no idea what's wrong with my application

[eluser]Nameless One[/eluser]
Ok, thanks for help Fuzzy. It seems there was a problem with the db server which has been resolved recently. ActiveRecord is ok.

[eluser]TheFuzzy0ne[/eluser]
Woohoo!

[eluser]deadelvis[/eluser]
I'm encountering the exact same problem. Can you tell me what was the "problem with the db server which has been resolved"? I might have to hunt down the same issue with my host.