If you want to connect using IP auth, you will need to answer on the port the connects are sent to, this does not apply if you use 'registration'I see this thread talks about changing ports for PJSIP 5061 to fix the same issue
I am not sure if I want to mess with the ports