I lived in Japan for three years. I wish that I had at least once dragged my body out of bed at 3:00 a.m., taken the one hour train ride to downtown Tokyo and gone to the world famous Tsukiji Fish Market. The market dates back to the 16th century when Tokugawa, the first shogun and builder of Edo (now Tokyo) invited fisherman from Osaka in order to provide seafood to the Edo Castle. This privilege also allowed the fisherman to sell the left-overs, thus creating a market. As the new capital grew, and the population expanded, so grew the demand for fish. The market was reformed and developed into an official wholesale market licensed by the Shogunate.
If you like sushi this is the place to go. The Tsukiji market brings in over 2,500 tons of seafood each day: delicate seaweed, expensive caviar, delicate sea urchin, potentially lethal fugu, gigantic tuna. You'll also find some of the most savvy fish buyers who will drop top dollar (or yen) in seconds for the finest quality (see 60 Minutes story The King of Fish).
While you can no longer go to the market during the wee hours of the morning when the auctions occur it would still be worth the visit. Next time I'm in Japan I will make sure a visit to Tsukiji is on my itinerary.