To get access to the island itself you have to get shipwrecked there, you do this by traveling by boat from Liberty bay to Thais and let RNG (Random Number Generator) decide when you get shipwrecked. In other words, this is a random occurrence. After getting stranded on Goroma the ship captain will ask you for 30 pieces of wood that are dropped by the island trolls all around the area (make sure to check the holes for the caves filled with trolls and not just the ones on the surface, also bring a shovel and a rope so you don't get trapped). After bringing the captain all of the required pieces of wood you will now be able to sail back to the mainland and also sail from Liberty bay to Goroma anytime you want.
At the west part of the area you get stranded at you will find an energy field that you cannot pass at the moment, to get access past this and also get further access to the other parts of Goroma you need to do The shattered Isles quest.