Extract an Excel in background (for large files)

When handling big files, you are on right track, the best solution is to upload a file, and handle it with different process behind the scenes.

It's up to you to figure out the best solution for you, but it really comes down to only three parts - a worker script, a notification system that worker script can query to see if it needs to do something or not and a way to notify user that their file has been processed.

For worker script, your need to make a controller that you can call from command line:

Code:

php /your-app-folder/index.php controller/method

Make sure you add command line check to your controller (https://www.codeigniter.com/user_guide/g...cli#is_cli). This is so no-one can launch processor heavy tasks by simply going to right URL in their browser.

PHP Code:

class Worker extends CI_Controller
{
    public function __construct()
    {
        parent::__construct();
        if (!is_cli()) {
            echo "Unauthorised access.";
            exit;
        }
    }

    public function process()
    {
        // check if there are any files to process, get one queued file

        // if not, exit

        // lock returned file so next process can't start processing same file

        // process file

        // ...
    }
} 


When file is uploaded, assuming you move it to a folder within your project structure, you can either add necessary meta-data with additional .json file or DB entry.

The thing to look out here is you want to set it up in a way that you can lock certain files as being processed. You could have state field in your DB, so you fetch 1 row from "pending" state, but not "in progress", or if you go with meta-file solution, you can scan your special folder for .json files, but rename it to .locked while you process the file.

It's something called "race condition", so lets say it'll take 5 minutes for you to process a file, and you check for new uploads once a minute, you don't want to end up in loop where same file is processed multiple times before removed.

Once you get the code working by calling it manually from command line, you need to set up cron job on your server that calls same command once a minute.

Because all this is running behind the scenes, I suggest logging requests, at least at first, so you know the crons are running properly.