The Federal Reserve's massive easing campaign isn't working out too well for many Americans, says former Fed Governor Kevin Warsh.

He and Stanley Druckenmiller, a former colleague of hedge fund legend George Soros and founder of Duquesne Capital, note in a Wall Street Journal opinion piece that U.S. household wealth just reached a record high of $81.8 trillion. That represents a $26 trillion increase during the last five years amid Fed easing.

"No wonder most on Wall Street applaud the Fed's unrelenting balance-sheet recovery strategy," the duo writes. "It's great news for those households and businesses with large asset holdings, high risk tolerances and easy access to credit."

Editor's Note: Seniors Scoop Up Unclaimed $20,500 Checks? (See if You qualify)

But it's not so good for everyone else, Warsh and Druckenmiller maintain.

"It provides little solace for families and small businesses that must rely on their income statements to pay the bills," they argue.

"About half of American households do not own any stocks and more than one-third don't own a residence. Never mind the retirees who are straining to make the most of their golden years on bond returns."

"The country needs an exit from the 2 percent growth trap. There are no shortcuts through Fed-engineered balance-sheet wealth creation," they contend. "The sooner and more predictably the Fed exits its extraordinary monetary accommodation, the sooner businesses can get back to business and labor can get back to work."

Warsh is now a distinguished visiting fellow at the Hoover Institution and Druckenmiller is CEO of the Duquesne Family Office.

Meanwhile, Peter Boockvar, chief market analyst with Lindsey Group, is concerned that the central bank will drop the ball on inflation.

"The Fed is deathly afraid of a rise in rates stomping out the recovery," he told CNBC. "It's clear they will allow inflation to run higher than it should be in order to achieve that."

The Fed's long-term target for inflation is 2 percent. But it has indicated it would accept a rate above that level in the short term to boost the economy.

Editor's Note: Seniors Scoop Up Unclaimed $20,500 Checks? (See if You qualify)

Related Stories:

• CNNMoney: Where's the Recovery on Main Street?
• FT's Luce: Looking for the Missing Recovery? It's the Middle Class, Stupid