There are various reasons why you may want to restrict access to your portal- the user left the company or is no longer on the partnerships team, etc. As the Partner account owner, you have the option to Suspend that user.

Log into your portal, and click on the Agents tab. Find the respective Agent and select their name. Near the top right corner, you'll find the status drop-down that should be in the Active status. Click on the drop-down and select the Suspend status.

For more information on Agents' visibility, check out this article.