uploading and processing large files into database

[eluser]coolgeek[/eluser]
Nothing you can write in PHP/CI is going to match the performance of native MySQL processing. So, to start with, loading into a temp table is the better option than trying to parse the file yourself.

The next thing you want to do is try to limit the amount of data you have to import. Where is the data coming from? Is it timestamped, on either import, update, or both?