The firm, a so-called quantitative hedge fund that uses computer models to drive trading, is shuttering the eight-year old, $1 billion portfolio at the end of the month, according to a letter sent to investors on Tuesday and read to Reuters.

"Due to a lack of investor interest we have decided to wind down the Renaissance Institutional Futures Fund," the letter said.

The fund is off 1.75 percent for the year, but performance was not the key reason for closing it, a person familiar with the firm's thinking said.

The news comes on the heels of Fortress Investment Group's decision to shutter its $2.3 billion macro fund, which is down 17 percent this year.

Renaissance offers three portfolios to outside investors and two of them sport double digit returns, handily beating rivals, as many hedge funds nurse losses and are on track for their worst returns since the 2008 financial crisis.

The firm said it kept open the RIFF fund, launched in 2007, to accommodate investors who wanted a pure futures portfolio, but that there was little demand for such an offering.

Instead investors were far more interested in Renaissance's Institutional Diversified Alpha (RIDA) fund, which invests in futures and equities. The three-year old RIDA fund already oversees $5.5 billion in assets and has returned an average 8 percent a year since its launch. In September it gained 9.1 percent, leaving it up 11.48 percent for the year to date.

The firm's third fund that is available to outsiders is the 10-year-old $10.5 Renaissance Institutional Equities Fund, which trades only in U.S. listed stocks. This year it has gained 11.2 percent after rising 7.61 percent in September.

Since its launch in 2005, it has returned an annualized 9.8 percent, while the broader Standard & Poor's 500 stock index has climbed an average 6.7 percent per year during the same time.

Renaissance's most famous portfolio, the $10 billion Medallion fund, is available only to insiders at the East Setauket, New York-based firm.

Mathematician James Simons founded Renaissance in 1982 and has long ranked among the hedge fund industry's best paid investors. Peter Brown and Robert Mercer now run the firm.