Is it possible to have the volume control only used on certain outputs

No, but there is a feature request of this:

2 Likes