I think generally speaking you're probably best off going with 100mm and buying the best filter holder that will last you a decade or more, ie a Lee foundation kit, and then if you are on a budget, using the cheaper 100mm filters (hitech and cokin z series are both cheap, as are lee's polyester filters) for lesser used filters and spending as much as you can on your more used filters (NDs and Grad NDs for most people, hitech or lee resin for budget, lee glass if you have a bigger budget).
This way you have a system that can grow as your needs grow, rather than buying a cokin P system, or hitech 85mm, and regretting it 6-12 months down the line and then having to buy it all again.
Most brands seem to consistantly use cheaper methods for their 85mm filters than their 100mm filters, and colour casts seem to be more common.
If you buy a Lee filter holder, then you also have the option of using Lee/Hitech/etc's gels in Lee's polyester filter snap-frames. This makes occasional-use colour correction and colour modification filters a lot cheaper (a sheet of Lee gel is usually around £5 with enough to make a dozen 100x100mm filters). Note that these frames will not fit a hitech or cokin z filter holder due to differences in the allowed filter thickness.
Also bear in mind that you have two alternatives for lens fittings - you CAN buy a mounting ring for each lens you own, or you can buy step-up rings to one common size (some people go with 72 or 77mm, some go with the largest lens they own). Step-up rings are generally cheaper than filter mounting rings, and as a plus side, you then only need one good quality CPL for all of your lenses.
edit:
Also kood make a few square filters that fit the Lee/Hitech/Cokin-Z that run around £20 per filter, so they fit in the 'budget' bracket along with Hitech and Lee polyesters.
