*idea* - Translation system *idea*

[eluser]Silviu[/eluser]
Hello,

I'm not sure if this question belongs here, but I'm not sure where to ask it.

It is about a "feature" that I "met" once in a custom framework, and I'm curious what the community has to say about it:

This feature is about a translation system.


Basically, when you are developing your views/controllers using CI, whenever you need to use a string from a specific language, you call

Code:

lang('This is the string you need to translate');

The system works pretty well, for my taste, but when using it for large sites, it gets cumbersome.

In this unnamed framework, I found it was using a helper function similar to the one used by CI:

Code:

translate('This is the string you need to translate');

However, instead of searching for a corresponding key in the language array (as in CI), it was pulling values from the database. If a corresponding value was not found, one was created and the admin section of the site flagged the new values as ready for translation.

That gave my an idea, and I'm asking the community for opinions. Maybe it's worth following it, maybe it's not.

Here it goes:

Instead of using only the language file or only the database table, we use a language file and a database table with a structure like id, string(must be unique), lang_en, lang_fr, lang_de....
Whenever a requested string is not found in the language file, the helper function attempts to insert a new value in the "strings" table in the database, and then recreates the language file (according to the values present in the table) and in the end it returns the very string we used as a parameter to have a text in that place.

This means that for every unknown string it encounters, a new entry is created and the language file is rewritten. This operation happens only once, on the next page reload the string is present in the language file so there will be no extra database activity.

Pros:
- You don't have to work in two different places when developing, you just write the string in the code and it gets added to the table and to the language file
- You can see at any moment if/what strings are untranslated (and for which language)
- (if you create and admin module for this system) you can change strings for every language at once, instead of working on multiple files
- The end user can change strings without the need for a programmer to do it for him
- With a proper coded language switching mechanism, the user can create a new language for the site on the fly (maybe this is not a "pro" item, as it means more work and less future money for the developers :-) )
- The system falls back to the very string you used as parameter, so there will never be empty strings because that key was not defined in the language file
- The system will also fall back to this string if there is no value for that particular language the site visitor has requested

Cons:
- You need to implement an admin module for this "feature" to work 100% (extra work - quite a lot)
- The strings get added to the table as required, so if you give the user a blank table, the people who visit the website can sometimes find "untranslated" strings in the site
- The "strings" table might be populated with leftover values during development, which will prompt the admin to translate them, even if they are no longer used. This might prompt the developer to clear the strings table, leading to the above point.
- You need to have the language files writable on the server

Well, that's about it. I find this system quite interesting, but, as I said in the beginning, I'm looking for community feedback.

Pros? Cons? Comments? They are all welcome!

[eluser]vitoco[/eluser]
it sounds pretty nice, however my only concern is performance.

I think the id in the table, must be the SHA1 of the string, cause it get a define length to use as index in the language file ( also CHAR it's faster that VARCHAR ), avoid crashes in the db and it's cleaner.
Another thing, to the user it's better to find the default lang string instead of "untranslated", so the string/lang must have a "translated" flag.

Apart from that, it's a simple and clean way to translate.

[eluser]Silviu[/eluser]
The idea is that the strings are read from a language file (as CI does now). Only when a string is missing from that file the database is updated and the language files are overwritten with the new database content.

Second, by using translate('some string'), you will give a text to be translated. This text will also act as fallback if either the text is missing or if the translation for a particular language is missing.

I was thinking on checking against a null value to see if a text was translated into a particular language, I'm not sure if a "translated" flag is needed for a language (easy to add though)

[eluser]vitoco[/eluser]
do you wanna code this as a group project ?....to see what came out

i'm not used to work with people, so it will be fun to try it.

[eluser]Twisted1919[/eluser]
Well, the idea is not bad, but performance wise, you'll suffer, just because of the database look-ups.
I remember when i worked with drupal(i think drupal was), it had something similar:
t('string to be translated'); which is great on small sites, BUT, on large sites with many languages it really is a pain.

What i see as being a great implementation is not using files at all, it makes no sense to combine database translations and file translation, just pick one of them and go with it till the end. I would pick database just because of the flexibility.

THIS IS JUST AN IDEA, NOTHING TESTED
The table will have fields like: language_id | category | original_text | translated_text
The language_id would be set globally by the framework, so you don't have to worry about it.
Next, you would translate the messages as:

Code:

Translate::t('category','string');

Note, you will need a class for this, because the implementation needs some class vars in where you can keep track of the loaded categories, of the categories coming from cache etc etc.

The t() method will keep track of the loaded messages.
When messages from a category are requested, we'll try to fetch them from cache, if the cache is not available, we'll fetch from database then write into the cache.
Beside this, you will need a way to store messages that you define during runtime and at the end of the request merge those messages with the ones received from cache/database.

I wrote a class that stores website settings in database based on categories and uses cache + lazy loading.
You can alter the class easily and make it work as you want.
If you want to take a look, the class is here: http://pastebin.com/mFnqZ36U

[eluser]vitoco[/eluser]
i think the aproach it's similar, cache v/s file , and both use db look-ups when it doesn't get the string translation directly. So, it can be an option, built the cache from the database, or create language files from the db.

And as you 2 proposed helper and library, i will propose a model, for the built-in db access .

@Twisted1919 , i don't get the functionality of the 'category' param.

[eluser]Twisted1919[/eluser]
The category param is used mainly to get multiple translatable strings together.
Say you have your application built on modules, each module can be a category, and in that module you only load the translation category with same name as the module.
Also, using lazy loading here helps allot, because you don't load all the items from a category at the begin of the request, instead you load them only when an item from that category is requested first time and you store them in memory during runtime so that, the next call to an intem from a loaded category will serve the translation from memory rather than checking the database or the file system once again.
Makes sense ?

[eluser]vitoco[/eluser]
it makes sense, but i will put it as a second ( optional ) param.

[eluser]Twisted1919[/eluser]
Don't put it just optional, at least give it a default category, but don't let the option so that empty categories can be passed to the method.

[eluser]vitoco[/eluser]
ok...agreed, for me category = '' is for you category='default/global'