Wealthy buyers are pulling back from some of the most expensive housing markets in the U.S., the latest sign that sky-high prices and fears of a recession are weighing on a key sector of the economy.

Toll Brothers Inc., the nation’s largest publicly traded luxury-home builder, said late Tuesday that purchase agreements fell 3% from a year earlier, worse than a decline of less than 1% that was expected by a Bloomberg survey of six analysts. The company’s orders in California, home to some of the priciest markets in the country, tumbled 36% from a year earlier.

The results underscore a shift taking place in the U.S. housing market. The return of low mortgage rates is heating up competition for starter homes and fueling steep price gains in cities that have long been more affordable. Meanwhile, expensive markets, like San Jose and Seattle, as well as the luxury homes that Toll builds, have seen a drop-off in demand.

“Their buyers are a little more sensitive to what’s going on in the broader economy,” said Drew Reading, an analyst at Bloomberg Intelligence. “They’re paying more attention to the stock market.”

Slowdown Signs

Wall Street and Washington have been buzzing this month about the potential for an economic slowdown after U.S. equities suffered one of the deepest sell-offs of the year on Aug. 14 and a key portion of the U.S. Treasury yield curve inverted for the first time in 12 years. New-home sales also were weaker than expected in June.

Homebuilders that cater to entry-level buyers are better positioned to weather shifting demand, Reading said. Shares of D.R. Horton Inc., which focuses on starter homes, have climbed 41% this year, while Toll Brothers is up 12%. The stock slipped 1.9% in premarket trading Wednesday.

The luxury builder’s biggest challenge may be its concentration in California, where its homes under contract had an average price of $1.74 million in the quarter. Chinese buyers have pulled back there and the federal tax overhaul limited deductions for property levies and mortgage interest.

As at other homebuilders, Toll’s profits have gotten squeezed by the need to drive sales with buyer incentives. Fewer California deals, with their relatively fat margins, also didn’t help. Toll’s full-year guidance for adjusted gross margin of 23% was slightly lower than the consensus of 23.2%, based on a Bloomberg survey of analysts.

‘Tailwinds’ Intact

Toll is off to a good start in the current quarter that started Aug. 1, Chief Executive Officer Doug Yearley said in the earnings statement.

“Low mortgage rates, a limited supply of new and existing homes, and a strong employment picture are providing tailwinds,” he said.

The company has been looking to scale up its business in other parts of the country and build less-expensive homes.

“They are diversifying the type of product they’re building, but it’s not a transformation that happens overnight,” said Alex Barron, an analyst with the Housing Research Center in El Paso, Texas. “Toll Brothers is luxury. They’re not going to go from being a Nordstrom to being Walmart.”