Welcome Guest, Not a member yet? Register   Sign In
Testing: Database refresh and migrations
#1

I've run into a particular issue a few times and I'd like the community's input on it. Here's the scenario:

You have an e-commerce project. It's time to write your feature tests but in order for most pages to load you need a semi-complete set of data so you the framework tools for creating a fake set of test data. The catch is this requires dozens of users, hundreds of items, shopping carts, images, groups, privileges, etc etc etc and by the time you hit "go" every one of your 240 feature tests takes 30 seconds to run.
Well, no problem - how about you group feature tests into compatible classes and turn off database refresh between tests? Well now you don't get migrations unless they were run prior to your test case, so the seeder starts failing.

Sounds pretty specific but I've had to do acrobatics in a couple projects to get this working. My opinion is that `CIDatabaseTestCase::$refresh` should control wiping the database between each test, but if migrations are specified to run then they should run regardless of `$refresh`, same as seeders.

Anyone else running project tests care to weigh in?
Reply
#2

Though on a much smaller scale, I encountered this kind of problem - database queries slow down tests quite noticeable. So I do something similar, try to do tests sequentially - without wiping DB - where possible. I have very little experience with unit and integration testing, so I guess I will just watch and learn.
Reply
#3

That’s exactly the case I’m talking about. Since you don’t wipe the database between tests, let me ask: how are you setting up the database initially?
Reply
#4

(07-03-2020, 03:30 PM)MGatner Wrote: That’s exactly the case I’m talking about. Since you don’t wipe the database between tests, let me ask: how are you setting up the database initially?

I think I might not interpreted your question as you intended. I do a complete initialization on the database once for a test cycle of about ~ 420 individual tests as of now. So my input here might not be relevant. 

If you meant that database should be fully initialized once, and then somehow restore it to its original state between each run considerably faster than a full initialization, then I would research DB restoration tools or methods of some sort. Like Point In Time recovery with PgSQL, which might be faster, depending on how many alterations happen during the tests. But it is cumbersome and might not even be possible in other than development / test environment.

Anyways, I do this as a test itself like this (no surprise):
PHP Code:
class InitDatabaseTest extends CIDatabaseTestCase
{
    protected $refresh true;
    protected $seed 'InitDatabaseSeeder';
    protected $basePath TESTPATH  .'../app/Database';
    protected $namespace 'App';

    public function testSeedUsers()
    {
        $this->seeInDatabase('user', ['username' => 'john.doe']);
    }

  
where InitDatabaseSeeder contains all the INSERT statements to fill the DB with initial data.
Reply
#5

It would be a reasonable change, I like this idea.

There is one thing I'm afraid though. People may start writing tests in such a way that they tests will depend on the result of other tests that have been done before - just to "save" some time. But I guess we can't do anything about it.
Reply
#6

I agree that is a possibility, but honestly at this point I am more invested in the framework encouraging people to write tests than in enforcing proper unit testing protocol. With our current configuration I think it is quite possible someone would read "refresh database between each test?", set $refresh to false, and then have an error when their Seeder failed because the testing database hadn't migrated. That seems to me a bigger deterrent to successful test writing than interdependent tests.
Reply
#7

Here’s what I came up with:

https://github.com/codeigniter4/CodeIgniter4/pull/3221
Reply
#8

And it was merged in. Just a couple of thoughts since I didn't get to respond here prior to seeing it on GitHub.

SQLite in-memory databases make a huge performance difference, and I believe the handler is compatible in almost all cases with CI core database functionality - even foreign keys if you have a new enough version. Granted - if you need MySQL or Postgres -specific queries, you're boned. Smile

I've also seen it done where you wrap everything within each test in a transaction and rollback the transaction at the end of the test. That does speed things up, but I've ran into issues testing things in the database within the tests itself since the transaction was never finalized. So, not something I love but it can have it's uses.
Reply
#9

Thanks for the response Lonnie. I've been using the updated class, and it is an improvement but still not perfect. I think down the road CIDatabaseTestCase should go the way of FeatureTestCase: turn it into a stub class implementing a trait that we expose for developers, and the existing DB methods be spread out into smaller, optional bites.
Reply




Theme © iAndrew 2016 - Forum software by © MyBB