Antialiasing is for a Smooth Look. Retro is for an old school look. Smooth retro combines the two Retro and Antialiasing to create an old school smooth look. I know this because I tested it out previously and also if you hover over the settings in your options under graphics then where it says antialiasing mode it will tell you:
"If you select "Antialiasing", the Tibia graphics will appear in a smoother look. However, if you are a fan of retro games you should choose "None" to enjoy a pixelated old-school look. "Smooth Retro" combines the other two modes it is offering a smooth pixelated look."
I tested it with fps and it really makes no difference, for me at least.