Making this issue so I don't forget.
If a player spectates, prevent them from re-entering the game until a set amount of time has passed. (Defined by the server in seconds.
0?) Enforce this on first join for 1v1s specifically to prevent leaving and re-entering the server as a workaround, but immediately expire the cooldown if no active players are playing to allow for quick join-ins.
Optionally consider adding additional cvars for time to add to that cooldown if the user repeatedly spectates and rejoins, and/or for time without spectating to reset that added penalty.