+- CodeIgniter Forums (https://forum.codeigniter.com)
+-- Forum: Development (https://forum.codeigniter.com/forumdisplay.php?fid=6)
+--- Forum: CodeIgniter 3.x (https://forum.codeigniter.com/forumdisplay.php?fid=17)
+--- Thread: How about a max_insert_batch method ? (/showthread.php?tid=70871)
How about a max_insert_batch method ? - devaicom - 06-12-2018
Hi,
I deal with lots of data, that is several 100k rows x several websites every day.
Data is not always clean (external origin), so insert_batch sometimes does not insert all of it, and it's ok.
But i'd like to maximize the amount of data really inserted in database, so i was thinking about extending DB_query_builder to add a "max_insert_batch" method, which would do almost the same as insert_batch, except this :
each time you call _insert_batch, you verify the number of affected_rows
if it's different from the number of lines sent (which is batch_size, except for the last batch), you recursively call insert_batch with the same block of data you just used, but with a batch_size twice as small (unless of course batch_size = 1)
Your opinion ?
Thanks
RE: How about a max_insert_batch method ? - InsiteFX - 06-12-2018
There is also a limit that is set on how many rows can be inserted in MySQL it's self.