A U.S. appeals court ruled on Tuesday that the Securities and Exchange Commission cannot force stock exchanges to conduct a costly experiment to see how the fees they charge and the incentives they offer affect brokers' trading habits.

The SEC's "Transaction Fee Pilot" aimed to shed light on how lucrative rebate payments from exchanges to brokers for stock orders that others can trade against influence the brokers' behavior.

Critics of the practice, including institutional investors managing tens of trillions of dollars, said the payments create conflicts of interest by incentivizing brokers to send customer orders to exchanges that pay the biggest rebates rather than to those that would get the best results for their end clients.

Intercontinental Exchange Inc's New York Stock Exchange, Nasdaq Inc, and Cboe Global Markets Inc sued the SEC over the plan, arguing that rebates are needed to compensate brokers for providing liquidity and that the SEC has not shown they harm the market.

Eliminating rebates would also lead to wider bid-ask spreads, making it more costly to trade, they said.

The U.S. Court of Appeals for the District of Columbia Circuit said the plan "clearly exceeded the SEC's authority under the Exchange Act."

"The Pilot Program emanates from an aimless 'one-off' regulation, i.e., a rule that imposes significant, costly, and disparate regulatory requirements on affected parties merely to allow the Commission to collect data to determine whether there might be a problem worthy of regulation," the court said.

The pilot program, which was recommended by an SEC-appointed committee of market experts as well as the U.S. Treasury, would have tested banning rebates for certain stocks and lowering exchange transaction fees for others.

"We accept the decision of the D.C. Circuit and appreciate the guidance it provides," SEC Chairman Jay Clayton said in a statement.