[eluser]drewbee[/eluser]
Yes, it is highly scalable. You have to remember, a database is nothing more then files with data in them, organized for optimization, permissions, users etc.
This is just skipping that part going straight to files.
There are a few limitations based on the operating system. The maximum index size on a 32 bit system is 2 GB (2^31 - 1). This is not a fault of the index, but of the operating system; this is the case for any linux 32 bit system. If you need a bigger index, 64 bit can provide a maximum index (file size) of (2^63 - 1) 8,589,934,592 GB. lol, you should be fine.
It uses segments to store the indexes, and each index can be merged using the optimize() method.
I used it as a spider for one of my test projects, and it worked just fine (including the slow speed of PHP crawling pages, adding and parsing data) etc.
Henry is correct as well. I tend to use lucene for even basic searches (IE tags, blogs, forums etc etc).
The only downside to this is it takes twice as much diskspace (database side and Lucene Index); As I was describing in another thread though, the less I can hit my DB server, the better.
