Bute Museum on Stuart Street, just south of the castle, has only two rooms, but they are packed with exhibits on the island's archaeology, history and natural history. It's a good half-hour before or after the castle, and the place to get your bearings before heading out to St Blane's and the island's stone circles and forts.
Opening hours are short and seasonal, so check before you go.