I can’t believe it! Here is an online rating service for judges in Philadelphia. I had this exact idea years ago, long before I started blogging. In my conception three things were required: (1) a numerical rating system with limited categories, and the numerical rankings would allow one to see how the judge ranked against his or her colleagues; (2) anonymous voting, and (3) limit voting to attorneys admitted before that court.
The method that RatePhillyJudges.com is using only meets the first two requirements. If anyone can vote, then you have to assume anyone will (friends of the judges etc.) and that fact alone taints the results. But I like the fact that the site allows comments (see examples in this post). This allows people to offer suggestions for improvement, and also gives a flavor to the assessment that pure numbers lack. For more information about this site, read the Law.com article.