High gasoline prices are more threatening to the U.S. economy than they appear, as the effects of warm winter weather have cushioned the country from pricey fuels, says billionaire private-equity investor Wilbur Ross.

"What I am worried about is the following: gas prices have been high and the effect on the consumer has been hidden," due to the mild winter, Ross tells MarketWatch.

Demand for heating fuels has been muted due to warm weather, but demand for gasoline is set to rise as the spring and summer approach, when driving increases.

Furthermore, refineries switch to more expensive inputs during the summer, which sends prices even higher.

Some states are averaging over $4 a gallon, and $5 a gallon is possible in some parts of the country.

"If we hit $5 at the gas pump over the summer, that would have a profound effect on the consumer," Ross says.

Crude oil prices have been soaring due to geopolitical tensions in the Middle East, taking gasoline prices up to a nationwide average of $3.84 a gallon, according to the AAA Daily Fuel Gauge Report.

Experts point out that the economy would be stronger if gasoline prices were lower, as consumers are cutting spending elsewhere to fill up their cars.

"It's a thorn in the side of the consumer and businesses," says Chris Christopher, an economist at IHS Global Insight, according to the Associated Press.

The economy this year "would have been better and stronger if we didn't have to deal with this."

Editor's Note: Wall Street Whistleblower Warns of Meltdown, See His Uncensored Interview