Al-Qaida's affiliate in Syria, the Nusra Front, seized control of large parts of the northwestern city of Idlib from government forces Saturday, sweeping into neighborhoods close to the city center, opposition activists and the group said.

The Nusra Front is leading a group of ultra-conservative rebels in a major offensive that began earlier this week to take Idlib, near the border with Turkey. If the city falls, it would be the second provincial capital and major urban center lost by President Bashar Assad's forces in the four-year-old conflict.

The eastern city of Raqqa fell to rebels, including the Nusra Front, in March 2013. It was subsequently taken over by Nusra's rival, the Islamic State group, which has since declared it as its de facto capital.

The Britain-based Syrian Observatory for Human Rights said rebel fighters seized control of most of Idlib in a push Friday evening and early Saturday after government forces withdrew to their bases and several other buildings in the city.

The group, which relies on an extensive network of activists across Syria, said heavy fighting continued Saturday amid heavy artillery shelling from both sides. The Local Coordination Committees, another opposition activist collective in Syria, also reported the capture of large parts of Idlib by the rebels.

An unnamed Syrian military official quoted by state-run news agency SANA said army forces were fighting "fierce battles" against "armed terrorist groups" to regain control in Idlib, adding that the opposing side suffered heavy losses. Earlier, the agency said army troops directed strikes at fighters who tried to sneak into the area near the national museum and the industrial zone around the eastern entrance of Idlib.

The government claimed earlier this week that "thousands of terrorists" streamed in from Turkey to attack Idlib and its suburbs. Turkey is one of the main backers of the rebels.

A loose alliance of anti-government rebels and members of the Nusra Front control large parts of Idlib province and have tried in the past to capture the provincial capital. The latest attempt appears to be the most successful so far.

On its Twitter account, Nusra said its fighters have taken control of half of the city, posting pictures of the Clock Tower and other landmark locations now under the its control.

In one video, also posted by Nusra on Twitter, a rebel fighter is seen tearing a billboard showing Assad in a suit and tie, with the words "Together, we protect it (Syria)," before trampling it with his feet. The video, which could not be independently confirmed, appeared consistent with AP reporting on events in Idlib.

Also Saturday, U.N. Secretary-General Ban Ki-moon said he was angry and shamed by the failure of the world to stop Syria's raging civil war. He promised to step up diplomatic efforts in comments at a summit of Arab leaders in the Red Sea resort city of Sharm el-Sheikh in Egypt.