Personally, I would use join_fields as you originally intended. This ensures that if you delete all photos, you also delete the primary. The section on join_fields in the manual covers the three basic methods that work with them.
If you want to have multiple relationships instead, then I would do it like this: