Will insulating my garage warm up the cold bathroom that’s above it?

Insulating garage walls won’t help because the garage has no heat source. You’d have to heat the garage to room temperature to make the bathroom upstairs warm and that would be wasteful and bad for your cars.

The only option is a two-pronged campaign to add insulation to the floor, and possibly increasing the amount of heating in the bathroom. You’re fortunate that the garage ceiling frame is accessible. This makes things a lot easier.

I’d recommend spray foam in this application, applied between the overhead joists at least four inches thick. Foam does need to be covered to meet building code, and you could use plywood or drywall. It’s an aesthetic choice. There are many types of foam on the market, but I’ve had good luck with polyurethane in applications like yours.

Spray foam will optimize the insulation situation, but it still might not be enough to make your bathroom as warm as it should be. The best approach is to spray some foam, then see the results this winter. Directing additional heat into the bathroom may be required, but experience will tell for sure.