composer install and direct install

No you can't delete the vendor folder. It contains the actual framework and default files/folder (e.g. App, public, tests, vendor ...)

All files and folders in your root catalog (app, public ...) are a identical copy vendor\codeigniter4\framework (except system folder). System folder contains the framework itself. Everything else to run the application.

That vendor folder gets updated with every version, downloading new copies of e.g. vendor\codeigniter4\framework\app\Config\Mimes.php so that you can copy/merge it with your own file app\Config\Mimes.php.

You should not touch the vendor folder at all. Only modify everything else.