Don't use client port as default peer port

Fixes using -port and not being able to join servers with connect or -connect without also specifying a port there.