Honestly, the QB class doesn't do enough to actually have a significant impact on performance unless you're in a situation that requires extreme performance. Anyone could write poorly-performing SQL using either approach.
If you need to generate your SQL in your application based on a number of conditions, it's possible that your method of building the SQL could end up being less efficient (or more error-prone) than it would have been using Query Builder. Alternatively, you could build something more efficient and less error-prone for your particular situation, simply because Query Builder has to be more flexible than your own solution.
Portability between database engines is probably going to have more to do with your database design and how you generate your tables than the majority of the SQL statements in your code. Query Builder can only do so much to prevent you from having to rewrite some of your SQL if you have to change database engines.
As with any question about performance, the answer is completely dependent on your situation. In most cases, the best thing to do is to take the approach that will cost you the least amount of time in development (with an eye towards being able to change things if necessary), then worry about performance when you can actually measure performance and determine what (if anything) is causing a problem. I've never run into a situation in which XDebug's profiler showed me that moving from Query Builder to a hand-built query would be the best thing I could do to improve performance.
