The specters only reflect certain elemental damage. For example, Burster Spectre only reflects ice damage. Assuming I understand how everything works correctly, you could attack the spectre with ice damage, they would then reflect that ice damage back to you, then your parry could reflect that damage back to the spectre as neutral damage. Since the damage dealt to the spectre was neutral and not ice, it cannot be reflected back again to you.