Welcome Guest, Not a member yet? Register   Sign In
MySQL Mass Input via CSV
#1

[eluser]Unknown[/eluser]
Hello and Good Day!!
I am having some trouble that I have not come across before with CI. I am trying to import csv data into MySQL via CI. Some of the csv files have up to 180k lines with 40+/- table fields. I am starting small for testing and can not import over 20k lines with the expected results...their is no html output, but the queries process without fail. As you will see in the code, I am importing 4 csv files. If I leave out the first file, everything works fine, but as soon as I the first file, I receive no html output results, only a blank page. I also receive the same results if I try the first file by its self. I had to modify the first file to less than 10k lines to get it to work for testing.

Code:
<h1>Header Line One</h1>
<h2>Header Line Two</h2>

&lt;?

$Choices = ARRAY('info1'=>ARRAY('table'=>'TABLE_ONE','file'=>'File_One.csv'),/*69338 lines*/
        'info2'=>ARRAY('table'=>'TABLE_TWO','file'=>'File_Two.csv'),/*7317 lines*/
        'info3'=>ARRAY('table'=>'TABLE_ONE','file'=>'File_Three.csv'),/*1441 lines*/
        'info4'=>ARRAY('table'=>'TABLE_TWO','file'=>'File_Four.csv')/*2129 lines*/
        );


$this->db->truncate('TABLE_ONE');
$this->db->truncate('TABLE_TWO');

FOREACH($Choices AS $to_import){
    $linecount = 0;
    $this->csvimporter->open_file($to_import['file'],TRUE,',',4096,'"');
    
    ECHO '<h3>Importing File: '.$to_import['file'].'</h3><br />';
    WHILE($CSVdata = $this->csvimporter->get(1000)){ /*gets 1000 lines from the csv at a time*/
        IF(!EMPTY($CSVdata)){
            FOREACH($CSVdata AS $CSVDataLine){
                $linecount++;
                $this->db->insert($to_import['table'], $CSVDataLine);
                
                IF($linecount % 1000 == 0){
                    ECHO 'Success Import: '.$linecount.'<br />';
                }
            }
        }
    }
    $this->csvimporter->close_file();
    ECHO '<p>File import complete.<br />
        Total Line Count: '.$linecount.'</p>';
}

?&gt;

$this->csvimporter->* is a custom class and has been ruled out as any problems. I have used several ways to import the data, including manually making the connection and queries. This is a variation of one of my original script built for CI as I am trying to move the scripts to CI. This is my first problem that I could not solve myself.

I am at a loss to why there is not html output even though the import works. I do not find anything about a max buffer, time limit or anything of the sorts that would keep it from outputting the results. The output page is blank and states "done" by the browser before the queries are complete. Please help, this is driving me nuts!!




Theme © iAndrew 2016 - Forum software by © MyBB