"it's easy enough for them to leave and do one without dragging me with them. "
No it is not.
If you are the host then other players CANNOT leave until the mission is completed or they loose all the mission rewards.
You are complaining about being forced to do something you dont want to do while at the same time forcing other people to do what they dont want to do.
If you want to run around and loot the small amounts of resources floating in space then set your party to solo and go do free roam.
You can then spend all the time you want roaming around with no people bugging you and you wont be bugging anyone else.