Using a model in a plugin

[eluser]TheServant[/eluser]
I apologise if this has already been discussed, but I cannot find it.

My structure is this:
I have one Controller which will take a page request, run a corresponding plugin (a unique on for each page) and then serve the view returned by the plugin.
I used a plugin instead of a library because a library should be stand alone and generally a collection of functions rather than one.

However, I have realised that the plugin cannot use models loaded from the controller? I also cannot load a model in the plugin? Am I just doing something wrong or are plugins not meant to be used this way?