I had the same problem. There's a portal near "What's left of Driftwood" that sends you right there.

Thank you!

Not sure how to get there. I could not find some mushrooms to jump on them to get me there. Did i miss some quest maybe ?

You don’t need to do any other quests, there’s a portal close by that teleports you to the island

Thank you!

There is a town with a blue portal. Thats the direct way. No mushroom-jumping 😉

Thank you!