Query Builder: regex error when passing large array to where_in - Printable Version +- CodeIgniter Forums (https://forum.codeigniter.com) +-- Forum: Development (https://forum.codeigniter.com/forumdisplay.php?fid=6) +--- Forum: Issues (https://forum.codeigniter.com/forumdisplay.php?fid=19) +--- Thread: Query Builder: regex error when passing large array to where_in (/showthread.php?tid=72820) |
Query Builder: regex error when passing large array to where_in - objecttothis - 02-15-2019 CI 3.1.9 PHP 5.6.38 FreeBSD 11.2 Apache 2.4 MySQL 5.7 Problem: passing a large array to where_in generates the php error and the update fails. Code: preg_match(): Compilation failed: regular expression is too large at offset 106661 /usr/local/www/public_html/pos_alpha/vendor/codeigniter/framework/system/database/DB_query_builder.php 2418 Background: I need to be able to generate this SQL: Code: UPDATE ospos_attribute_values Query builder cannot use db->join() with db->update(), so my solution is to use query builder and break the query into two. One to pull the attribute_id's that share the definition_id that I want and put them in an array. and the second is to update the attribute_values table by setting the attribute_value to the attribute_datetime. PHP Code: $this->db->trans_start(); The problem comes in that this array in particular contains large amounts of data (12,000+ items). I'm guessing that since MySQL and QueryBuilder were never designed to have WHERE IN clauses contain more than a small array (that's what joins in updates are for) that my problem is being caused by this. Should I just abandon Query Builder and use query() instead to make my join or is there a good solution that includes query builder that I'm just missing? RE: Query Builder: regex error when passing large array to where_in - php_rocs - 02-15-2019 @objecttothis, Why is your Where clause so big? Can you make it smaller? RE: Query Builder: regex error when passing large array to where_in - kilishan - 02-15-2019 I would chunk the data into smaller pieces. Should work fine then. RE: Query Builder: regex error when passing large array to where_in - objecttothis - 02-18-2019 (02-15-2019, 11:48 AM)php_rocs Wrote: @objecttothis, My WHERE clause isn't large. The WHERE IN clause is large however because of the number of attribute_ids that need to have the data 'moved' from one column to the next. It can be chunked into smaller segments, but there is still a significant performance hit. This is why JOIN with UPDATE exists in SQL. RE: Query Builder: regex error when passing large array to where_in - objecttothis - 02-18-2019 (02-15-2019, 11:52 AM)kilishan Wrote: I would chunk the data into smaller pieces. Should work fine then. Yes, it doesn't error if I put the UPDATE into the foreach loop but that means that I am doing 12,000+ hits against the database which takes longer than if the WHERE_IN array could handle a 12k long array. In the end I abandoned Query Builder for this transaction in favor of db->query() due to the limitations of not being able to do a join with an update or get an efficient solution by breaking the query into two. RE: Query Builder: regex error when passing large array to where_in - kilishan - 02-18-2019 (02-18-2019, 04:01 AM)objecttothis Wrote:(02-15-2019, 11:52 AM)kilishan Wrote: I would chunk the data into smaller pieces. Should work fine then. Well you wouldn't chunk one row at a time, just someone your server can easier handle - like 1000 rows or 2000? Drastically cuts the number of queries down, and still protects your server's memory usage as the size of the dataset grows. |