The most notable house on the island, Earnest Hemingway house, was built by the man that had the fastest schooner. His most notable claims was the $1M for hand carved Italian marble which was destined for one of the Florida Government buildings. When the Government found out about the auction, they went and outbid everyone. It turns out all that marble in the belly of the ship as ballast weight, made it too deep in the water to get over the coral shoals.
Another famous shipwreck was filled with 1M COB (Cut off the Bar) gold coins. COB coins are known for their irregular shape where they are crudely cut off an ingot then stamped. Each coin is therefore unique. Again, the ballast weight made the boat ride too deep.
The last interesting story Spain had 10+ ships all leave on Friday 13th. Nearly all the ships were ravaged by a storm and littered the Florida east coast. Spain spent the next 3 years recovering the coins. They actually recovered more goods that was on the manifests, since allowing stowaway cargo was normal.