Datamapper memory consumption ? 16Mb exhausted on small query

[eluser]junkwax[/eluser]
Hi, I just installed Datamapper 1.6.0.

I do:

Code:

$c = new MyClass();
$c->select( 'id' )->limit( 2000 )->get();

and I get the PHP memory error:
Fatal error: Allowed memory size of 16777216 bytes exhausted (tried to allocate 40 bytes) in.........datamapper.php.

When I load a smaller number of rows there is no problem:

Code:

$c = new MyClass();
$c->select( 'id' )->limit( 500 )->get();

So I looked a what's in $c and I found that $c grows proportionnaly
with the number of rows I fetch. In fact $c is 8KB X number of rows,
so for 16Mb of allowed memory, I can only fetch 16Mb / 8Kb rows (~about 2,000 rows).

If I do :

Code:

foreach( $c->all as $obj )
{
   print_r($obj);
}

I see that each $obj is an instance of MyClass containing 8K of stuff that eat memory: lang string, reference to validation objects...etc. So $c->all is a big object containing thousands of 8KB MyClass instances.

It is a bug of there's something I do horribly wrong ?

[eluser]OverZealous[/eluser]
Out of curiosity why are you trying to load 2000 objects into PHP?? PHP is not the correct place to be managing that many objects.

If you are trying to manipulate them, perhaps there is a way to run the update using a query.

If you need them in PHP for some reason (meaning, you need to output the objects), you could try loading them in inside a loop, as thus:

Code:

for($i=0; $i<2000; $i+=100) {
    $myObj->get($i, 100);
    // process 100 objects here
}

[eluser]junkwax[/eluser]
No manipulation here, I just want to fetch and display 2000 Id's on a web page.

Code:

$c = new MyClass();
$c->select( 'id' )->limit( 2000 )->get();

foreach( $c->all as $x )
{
  echo $x->id;
}

What's wrong with that code? It eats 16MB of PHP memory.

How should I procede to display the 2000 ids on a page in an optimized way ?

[eluser]OverZealous[/eluser]
Well, there is nothing technically wrong with that code. It's just that DataMapper isn't optimized for displaying a lot of individual objects like that - it's more optimized for manipulating objects.

The reason it uses so much RAM is that each MyClass requires a complete object, with a lot of properties and overhead. It isn't noticeable for a few hundred objects, but 2000 is a lot.

Anyway, did you try the code I mentioned above? It should reduce your peak RAM usage. Otherwise, for such a special case, I would be tempted to just run a normal query (for this instance). You aren't really doing anything in your example code that makes using DM seem worth the overhead.

[eluser]junkwax[/eluser]
Ok thanks for the answer.

I use DM to simplify my life, but if I have to switch between 2 ways
of querying my db, DM losts its shine.

Honestly, if I have to use a normal query for such a simple case,
(after all it's a small get() ), it means that DM is not fit for
production use and DM code must be improved.

[eluser]pistolPete[/eluser]
[quote author="junkwax" date="1236657025"]it means that DM is not fit for production use and DM code must be improved.[/quote]
Why do you need to display 2000 ids?
Which individual can handle that much information on one page?

[eluser]junkwax[/eluser]
It's not the real query. And you certainly understand that "MyClass" is not the real class name.

My point is that exhausting 16MB for selecting ~2000 numeric IDs is horrific.

So far the solutions provided are:

1) "Use a normal query instead of DM"
2) "Don't display 2000 ids to your users".

I think that it's not my UI requirements that must adapt to DM, but DM that must handle my (small) UI requirements. The fact is that eating 8KB per row fetched is a mega-waste.

For now I hacked a little bit DM to not eat 8KB per row, so now I can select 35,000 rows ( 17 times more ). So my problem is fixed, but my patch certainly broken something somewhere else.

[eluser]OverZealous[/eluser]
Instead of just being critical, do you think you could explain what you did to reduce the RAM consumption? DataMapper is used by more than a few people, and if your solution is feasible, I'm sure that it would be looked at how it could be integrated into the core distribution.

You aren't loading 16MB for "~2000 numeric IDs". You are loading 16MB for 2000 in-memory objects.

And I never said don't show 2000 ids to your customer. I gave you a solution that let you run multiple queries, so you weren't loading all the objects into RAM at once.

[eluser]junkwax[/eluser]
The modification is suitable for my needs only, DONT DO THAT at home,
as it *certainly* breaks many features of DM.

I modified the _to_object() function in datamapper.php to add $item_clean :

Code:

function _to_object($result, $model)
    {
        $items = array();

        foreach ($result as $row)
        {
            $item = new $model();
            $item_clean = new stdClass;

            foreach ($row as $field => $value)
            {
                    $item->{$field} = $value;
                    $item_clean->{$field} = $value;
            }

            $item->_refresh_stored_values();
            $items[$item_clean->id] = $item_clean;
        }
        return $items;
    }

@Overzealous: I know you didnt say that, it's pistolPete's comment.

[eluser]jacobc[/eluser]
Could you explain why you need to display over 2000 ids?

I don't think an ORM system should be expected to handle so many. Each row is a new object so obviously it is going to take up space.

ORM's are supposed to simplify the relationships between models... obviously. Which DataMapper does well.

If you are displaying a list of 2000 objects on one page surely that seems like way too much.