how to use composer when application and system directory are not in webroot?

My first answer was aimed only at handling your very valid concern about "hard-wired" paths. Now I'll talk about the rest of your OP.

IMO, the design assumption that /vendor should be in /application was incorrect. It is common, dare I say a universal industry practice to install Composer locally in the project folder e.g. /var/www/project. The Composer documentation itself makes this recommendation. That directory is also the place that composer.json is expected to be.

By default, Composer creates /vendor in the project directory.  You can configure Composer to put /vendor somewhere else, but CodeIgniter does not do that. If it did you would see something like the following in the distributed composer.json.

Code:

"config": {
    "vendor-dir": "application/vendor"
}

But you don't see that. I have no idea why CI "thinks" the way it does.

So, it's my recommendation to go with standard industry practice. Install Composer to and run it from the project directory and let /vendor be created there too.

Might this introduce any problems with composer? I'd point out that any composer changes, such as installing the packages I described in my OP, would be making adjustments to the composer.json file that exists as part of the basic CodeIgniter install. I worry that this might introduce problems.

I cannot think of what those problems might be. The distributed composer.json file only installs a couple of packages that are useful for developing applications, but none that CodeIgniter itself depends on.
As long as you use Composer to manage the packages it's pretty hard to mess it up. That includes accidentally deleting or replacing composer.json. Just don't loose composer.lock which is a file that should be tracked in git or whatever tool you use for version control.

PHP Code:

$config['composer_autoload'] = str_replace('application/', 'vendor/', APPATH); 


I would point out that this would be problematic if my project had "application/" in its path already. E.g., if my project were called "my-application" instead of "project" and was installed at /var/www/my-application/.

Yes, that's true. But for solving the problem you defined in the OP it works.

PHP Code:

$config['composer_autoload'] = APPATH . 'vendor/'; 


That path, if I'm not mistaken, is where CodeIgniter assumes your autoload file is by default. You could just set it to TRUE and CodeIgniter would look in that spot for the autoload file. This is partly what prompted my question in the first place. If CodeIgniter by default looks for the autoload file at application/vendor/autoload.php, perhaps there is some compelling reason for that location to be used?

Again, you are correct. That is the path CodeIgniter assumes and simply using true would produce the same result. I lost track of that because; 1) I always put /vendor in the project directory; 2) I forgot that CodeIgniter expects a non-standard location for /vendor. I cannot find a compelling reason for that and, as pointed out earlier, several reasons why it's compelling for it to be in the project root.

Also, I prefer *not* to use the autoload feature myself because the vast majority of controller methods I've defined have no need for the libraries I'm installing using composer. There's no point in loading composer's autoloader for the vast majority of pages on my site. For this reason, I prefer to only load the autoloader manually when the libraries are needed like so:

PHP Code:

require_once APPPATH . "vendor/autoload.php"; 


It's still a bit disappointing that I must autoload ALL my composer-installed libraries at once rather than just loading the one I need at a given time. It seems a bit inefficient. I pretty much never need to use sendgrid and google vision SDKs at the same time.

I think you are confusing the CodeIgniter feature  “Auto-load” with the computer programming term "autoloading". In the context of Composer "autoload" refers to the capability of loading and linking code  automatically when needed, so that the programmer is not required to define or include those portions of the program explicitly. In other words, it eliminates the need write and maintain a lot of require and include statements.

The following code does not load all the composer installed libraries at once.

PHP Code:

require_once APPPATH . "vendor/autoload.php"; 


What it does is put in place code that will find the appropriate class file and include (or require) it on demand. For instance, assume you have used Composer to install the "PhpSpreadsheet" package. To use this package in a controller you first need tell the class that you intend to use the class

PHP Code:

<?php
/**
 * Displays some report and export a spreadsheet
 */

use PhpOffice\PhpSpreadsheet\Writer\Xlsx;

defined('BASEPATH') OR exit('No direct script access allowed');

class Reports extends CI_Controller
{
... 


Then when you need to write to the excel file there isn't a call to $this->load->... or include('/path/to/class/file) or any extra lines of code. You simply create an instance like this

PHP Code:

$writer = new Xlsx($workbook); 


And the autoloader will find the file and include it automatically. But the code is not loaded until the call to new.

Just use $config['composer_autoload'] and forget about having to include autoload.php anywhere for the rest of your project lifetime.