A renewed bout of weakness in housing makes it more difficult for the Federal Reserve to think about selling mortgage assets when it decides to begin to withdraw monetary stimulus, a top Fed official said Friday.

James Bullard, president of the St. Louis Federal Reserve Bank, told Reuters on the sidelines of a Fed conference that the U.S. economy was nonetheless still on track for a reasonable economic recovery.

But the health of the housing market, which was at the epicenter of the prolonged financial crisis, will play a particularly key role for the Fed — the U.S. central bank — as it mulls the best way to unwind its emergency policies.

"It's making me nervous," he said of the beleaguered sector. "I will be interested in seeing the (upcoming) data and if that begins to not look good then I'll start to wonder.