Every action you make against other players (friendly or not) has the respective effect in your rapport. Your rapport determines if you are friends or enemies with a player, and having bad rapport won't allow you to summon anyone in their Keep or even view it.

Rapport goes down for any aggressive actions against a player, like attacking them, defending against them and of course assaulting, capturing or executing their Nobles.

Rapport goes up with any positive actions, such as Tributes or marriage between your Noble families.