I think it would be almost impossible to determine what triggers a creature to chance its target. Of course, there must be some algorithm behind which is involves math, but in any case, I would rather say based on my experience that it is completely random.
Of course, some monsters are more likely to change targets but it doesn't necessarily means there is something triggering the target change. I do have noticed though, that many monsters tend to change target more often when they are being attacked by other players.
For instance, let's put an example as seen below. This is only an example.
Let's say a giaint spider has 10% chance of changing its target. When attacked by other player it has 15% chance of changing its target. This is how I believe it works.
There are also some very specific actions which could lead to change target like stepping behind a wall that a monster wont pass or when monsters have no possible way to reach their actual target.