Oh, as usual I figure things out an hour after posting in a forum.
My conclusions : although the core is a little bit slower than version 3, the real bottleneck in my case was caused by the PCONNECT option in the database.php settings. By setting PCONNECT to TRUE, my page load times came back down to their usual speeds.
Hope this helps somebody else.
