I completely understand the need for Harvard and the NCAA to ban such a substance. It certainly could hinder an athlete's ability to simultaneously be a successful student. They have a special interest in the athlete's success out of the pool that the IOC does not possess.
You may be saying to yourself: "Yeah, that's true, but marijuana is illegal." Is it illegal throughout the world, though? The poor Dutchmen (or athletes in any number of other nations) that want to be compete internationally cannot smoke pot, even though their society deems it a perfectly acceptable practice (if not acceptable, at least not illegal).
As I mentioned before, I am an advocate of ridding the sport of performance-enhancing drugs and I do not personally bow before the bong, but the fact that the IOC uses a valid concern about performance-enhancing drugs to dictate what lifestyles its athletes can lead is simply appalling.
The IOC is not the Pope, it does not function as a societal legislature, it is an organization with the sole purpose of making amateur athletics as fair as possible for the athletes. It serves solely as a protection of athletes from cheating.
If American swimmers want to smoke up, let them take their chances with local police forces and the NCAA, but don't have the IOC tell them how they spend their Saturday nights, please.