There are a number of sites now where you can see how good your beliefs are - how accurately they predict reality* - with reward or punishment to match!
Sites like Pundit Tracker help you do it for others, but how about for yourself?
Predictionbook: pick your own prediction - about anything - and assign the chance that you think you're right. Are you really right 90% of the time about things you're 90% sure about?
Intrade: a futures market, but not the only one. It's basically a stock exchange for betting on future events - political, science discoveries, you name it. At any point in time, the value of a single share of a bet will be between $0 and $10. If the market thinks that your side of the bet is unlikely to happen, it might cost you only $1 to get in (but you get the full $10 if you're right; the market thinks there's only a 10% chance, so you have 10-to-1 odds). On the other hand, if the market thinks it's likely, you won't make much at all if you try to jump on the bandwagon. Midnight on election night, it was about $9.90 to buy "Obama wins", and if you're right you got back $10. The point is that when you combine the law of large numbers with people suddenly becoming more rational when there are material consequences, you get a tool that converges on good probabilities for events, and rationality is rewarded.
Truth Market: if there's a claim you hear that's crazy ("The world will end in 2012", "Scientists drilled into hell") you can offer a bounty - or better yet, you can go look to see if someone has offered money for something that you can disprove! If you think you can show cell phones are dangerous to our health, and you can prove it, then you're in luck, because that's one of the claims up right now (that they're not).
The Updating Game: A smartphone game where you're asked for your 95% confidence interval on various questions. Play by yourself or with a friend.
None of these will cure irrationality overnight but it's a step in the right direction. It's getting harder to be an inaccurate-belief-having Froot Loop and simultaneously not know that you are such a Froot Loop.
*For people who are too wimpy to actually hold their beliefs about the world up to accountability and would rather believe in falsehoods than behave like grown-ups, well these sites probably aren't for them.
Signaling Group Membership
53 minutes ago