BUG: UTF-8 does NOT work!

[eluser]AzizLight[/eluser]
Hello everybody,
I am developing an extremely simple website using CodeIgniter. First of all I'd like to say that I did all the things listed in Phil Sturgeon's blog post.

Now there are two things that are not working properly:

1) First, UTF-8 is not detected by CodeIgniter. This means that despite all the steps described in Phil Sturgeon's blog post, data retrieved from the database are not treated as UTF-8.
2) The database system in codeigniter fails to retrieve data in UTF-8.

I will explain how I came to those two conclusions:

Here is a method that I use in one of my models:

Code:

public function get_one($id)
{
    $this->db->select('idTexte, titre_fr AS titre, texte_fr AS texte, intro_fr AS intro')
             ->from($this->_table)
             ->where($this->_primary_key, $id)
             ->limit(1);
    $query = $this->db->get();
    
    if ($query->num_rows() == 1)
        return $query->row();
    else
        return FALSE;
} // End of get_one

And here is the result I get from the database (result displayed with var_dump()):

Code:

object(stdClass)#19 (4) {
  ["idTexte"]=>
  string(7) "aPropos"
  ["titre"]=>
  string(42) "À Propos - à propos Présentation"
  ["texte"]=>
  string(156) "<p> c'est cool, c'est super!!!!!! HéHé!</p>
<p> Ãƒâ‚¬ Propos - à propos</p>
<p>À Propos - à propos</p>"
  ["intro"]=>
  string(13) "<p> </p>"
}

Now look at the equivalent method using a new mysqli connection:

Code:

public function get_one($id)
{
    $mysqli = new mysqli('localhost', 'root', 'root', 'the_database');
    
    $_q = 'SELECT titre_fr AS titre, texte_fr AS texte, intro_fr AS intro FROM table_textes';
    
    if ($result = $mysqli->query($_q))
    {
        while ($row = $result->fetch_object())
        {
            var_dump($row); // DEBUG <-
            die();
        }
    }
} // End of get_one

And here is the var_dump() output:

Code:

object(stdClass)#19 (3) {
  ["titre"]=>
  string(35) "À Propos - à propos Présentation"
  ["texte"]=>
  string(142) "<p> c'est beau, c'est cool, c'est super!!!!!! HéHé!</p>
<p> Ã€ Propos - à propos</p>
<p>À Propos - à propos</p>"
  ["intro"]=>
  string(13) "<p> </p>"
}

There is clearly something wrong with the CodeIgniter database system. The difference between the two var_dump outputs is that if I use PHP's utf8_decode on the retrieved data, the mysqli output is displayed correctly:

CodeIgniter's output:

Code:

object(stdClass)#19 (4) {
  ["idTexte"]=>
  string(7) "aPropos"
  ["titre"]=>
  string(35) "Ã? Propos - à propos Présentation"
  ["texte"]=>
  string(142) "<p> c'est beau, c'est cool, c'est super!!!!!! HéHé!</p>
<p> Ã? Propos - à propos</p>
<p>Ã? Propos - à propos</p>"
  ["intro"]=>
  string(13) "<p> </p>"
}

MySQLi's output:

Code:

object(stdClass)#19 (3) {
  ["titre"]=>
  string(32) "À Propos - à propos Présentation"
  ["texte"]=>
  string(136) "<p> c'est beau, c'est cool, c'est super!!!!!! HéHé!</p>
<p> À Propos - à propos</p>
<p>À Propos - à propos</p>"
  ["intro"]=>
  string(13) "<p> </p>"
}

These two last code snippets prove my first point: CodeIgniter doesn't recognize UTF-8:the use of utf8_decode() should not be necessary for MySQLi's output to get displayed correctly.

Now, I prepended the title of this post with BUG because it's the most likely explenation. Also, I downloaded a fresh version of CodeIgniter and reran all the tests and I got the same results, which proves once again that those two problems I have are actually bugs.

Can anyone confirm what I am saying, and help me solve the problem please?

[eluser]renatomoya[/eluser]
Hello,

It's weird you're having this issue. I've developed several sites with CodeIgniter and no problems with UTF-8.

Can you tell me something.. what IDE are you using? Please tell me you're not using Dreamweaver.

Regards.

[eluser]AzizLight[/eluser]
I use Textmate. (I will never use Dreamweaver, or pay for it for that matter...)

Also I am sure that this is not the issue anyway. I created a small script that is independent and not connected to CodeIgniter in any way. All this script does is connect to the database to retrieve data. I didn't have any issue with UTF-8. Hence, the problem comes from CodeIgniter.

[eluser]renatomoya[/eluser]
Okay, I remember having issues with accents / encoding. And the problam was the way I was saving the scripts as. If you save your scripts encoded in ANSI, 100% sure you'll have issues with accents and data retrieved from a database.

I use Notepad++, this awesome opensource IDE has a menu called: "Format" which allows you to convert any text file into UTF-8 without BOM.

All your scripts must be saved to this format in order to get rid of all the symbols, and of course add the meta tag with utf-8 encoding.

What collation are your database tables? I use utf8_general_ci for spanish accents.

Regards.

[eluser]umefarooq[/eluser]
First of all your database char set should be set for UTF-8 and just set you html meta header as UTF-8 and it works fine, now a days working on multi language site which has English and Arabic for the site, and its working fine just setting header UTF-8 and database accepting UTF-8 char.

[eluser]eoinmcg[/eluser]
I've had a similar experience as @umefarooq; setting the db char set and html header to uTF8 does the trick.

BTW, the post name 'BUG: UTF-8 does NOT work!' is a bit excessive, no? Seems like you're doing sth wrong...

[eluser]AzizLight[/eluser]
Thanks for all your replies. The reason I called this a "BUG" is because accents don't appear correctly even though my db has a UTF-8 charset, I have have a meta tag that sets the content to UTF-8, I used php to set the content type to UTF-8 too (using the header() function), so it should work, but it doesn't. The weirdest thing is that if I take the same data from the same DB using plain php/mysqli, everything works fine. But if I use it withing a codeigniter install, it does't work at all...

I also tried to create a new CI project and connect to the same database but it still didn't work.

About the texteditor, as I said before, I use Textmate, and the file encoding is set to UTF-8 in the preferences, so there shouldn't be any problems...

At this point I have no clue what the problems is, but my logic says that it comes from CodeIgniter...

[eluser]keiichi[/eluser]
Try to send yourself the encoding,

Code:

header('Content-Type:text/html; charset=UTF-8');

Other solution is to use (if you can) PHP ini directive per directories

create a php.ini file in the directory where the view resides en put this code:

Code:

ini_set('default_charset', 'UTF-8');

If nothing works, then maybe the problem is the encoding of apache, you can try:

Create a .htaccess file to change the character encoding and put this code:

Code:

AddCharset UTF-8 .php

You can set the extension to whatever yo need .html, .htm etc.

Hope this helps.

[eluser]Vasu Adiga[/eluser]
How did you resolve this finally? I am facing the same issue as you...plain php/mysqli works fine, but codeigniter doesn't. Note that the data was inserted into DB by java program (not php).

[quote author="AzizLight" date="1267642573"]Thanks for all your replies. The reason I called this a "BUG" is because accents don't appear correctly even though my db has a UTF-8 charset, I have have a meta tag that sets the content to UTF-8, I used php to set the content type to UTF-8 too (using the header() function), so it should work, but it doesn't. The weirdest thing is that if I take the same data from the same DB using plain php/mysqli, everything works fine. But if I use it withing a codeigniter install, it does't work at all...

I also tried to create a new CI project and connect to the same database but it still didn't work.

About the texteditor, as I said before, I use Textmate, and the file encoding is set to UTF-8 in the preferences, so there shouldn't be any problems...

At this point I have no clue what the problems is, but my logic says that it comes from CodeIgniter...[/quote]

[eluser]AzizLight[/eluser]
I didn't resolve the issue...

I just retrieved what I needed using MySQLi, it's the only solution I found. Until I find a real solution I'm not using CodeIgniter for client projects anymore.