If you don't like your financial adviser, you can just take your assets and walk away, right? Not necessarily, writes Jason Zweig of The Wall Street Journal.

Some advisers charge clients with termination fees for leaving before a set deadline.

Editor’s Note: Retire 10 Years Earlier With These 4 Stocks

"It is appropriate for an adviser to recoup the cost of setting up and administering your account, and perhaps even to deter you from bolting the first time the market dips a little," Zweig says. "But securities lawyers say that termination fees should be directly related to those costs. Otherwise such fees would seem to violate the spirit, if not the letter, of fiduciary duty."

One of those lawyers is Robert Plaze, a partner at Stroock & Stroock & Lavan who formerly regulated investment advisers at the Securities and Exchange Commission.

"If for any reason you don’t trust your adviser anymore, or you don’t like his performance, then terminating the contract is your only real way to protect your interest," he told Zweig. "You shouldn’t be penalized for doing that."

Most advisers don't charge an exit fee, so the best solution may be just to avoid those who do.

Dana Anspach of About.com's Money Over 55 lists six expenses that you should investigate before choosing an investment advisor: the expense ratio of mutual funds; investment management fees; transaction fees; sales charges for mutual funds; exit fees for mutual funds; and annual account fees.

Editor’s Note: Retire 10 Years Earlier With These 4 Stocks

Related Stories:

• Money Manager Sanford: 'Stay Out of Money Market Funds' After SEC Rules
  
• Expenses Strain Profit at BofA's Merrill Wealth Unit