While some investors are selling stocks in a panic as Congress and the White House negotiate to avoid the fiscal cliff, many financial advisers recommend that their clients stand pat.

The worry for investors is that taxes will rise on capital gains and dividends. If the government fails to act before Jan. 1, the top capital gains tax rate will climb to 20 percent from 15 percent currently, and the top dividend tax rate will ascend to 39.6 percent from 15 percent currently.

Still, "I think it's a sucker's game to act on what you think [Congress is] going to do," financial adviser Andrew Millard of Tryon, N.C., tells Smart Money.

Editor's Note: 'It’s Curtains for the US' — Hear Unapologetic Warning from Prophetic Economist.

Investors who sell stocks now to avoid higher capital gains taxes later will still lose a chunk of their booty. And those who sell dividend stocks might have a tough time replacing the income.

As for the impact of higher taxes on wealthy investors, legendary investor Warren Buffett doesn’t think it will be too high.

Suppose an investor comes to you with a great investment idea, he writes in The New York Times. “Would your reply possibly be this? ‘Well, it all depends on what my tax rate will be on the gain. If the taxes are too high, I would rather leave the money in my savings account, earning 0.25 percent.’”

Editor's Note: 'It’s Curtains for the US' — Hear Unapologetic Warning from Prophetic Economist.