My two cents (as I've been having some "completely redesign the application" thoughts this morning)... And spent some time reading about "design patterns".
Compared to you, Thyrosis, I don't use many libraries in a project, (mostly) just for stuff, that I really now I will reuse and is application independent - like email, ACL for user based applications (as I use my own), for example a "config" library (used for user changeable configuration)...
And as far as I understand (correct me if i'm wrong), basic MVC pattern is a "funny" beast, as it is pretty old (before the web, modern applications...), so the M is nowadays not used just for data retrieval but for a lot more.
I use the CI Model in this way:
- To retrieve the data and return it to the controllers
- To manipulate/change/convert the data so it's usage is simple later on
- To check consistency or "logic" like so: "if row X in A does not exist, do not insert into row X into B" or "if X exists then update Z,Y,W" and at the same time (where constraints are not set in the DB) to check and keep consistency of data
- And in some cases, I call some library's methods and functions to again, return data which can be simply used in controllers (and later, views).
- I do try to keep controllers as slim(er)

(but when I started with CI, this was not the case, I did everything in controllers and used models exclusively for data retrieval)
I have to say, your approach, using mainly libraries, has some benefits...
Anyways, as for the libraries, I second Rufnex - they should be written in a way, that they are depended only on "themselves", so no models needed...