Welcome Guest, Not a member yet? Register   Sign In
Import csv long file into sqlite database takes too long
#2

1. First, that is a big import and your server may well have limits on the size of upload or a time limit on the processing

2. To speed it up, create your entire multi dimensional array first, then do a single database call with batch_insert. Otherwise you are doing 2500 database queries, which is ridiculous.

3. Allow partial updates and limit the file upload size. So first do a delete using where_in and an array of row identifiers from the CSV, then insert the new data.

4. Stuff on a local device is always going to be much quicker than on a web server, that not only has to do all the work your local machine does, it is probably having to do a whole lot more. Server environments can be very complicated.
Reply


Messages In This Thread
RE: Import csv long file into sqlite database takes too long - by PaulD - 08-19-2016, 12:54 PM



Theme © iAndrew 2016 - Forum software by © MyBB