Federal troops and National Guardsmen withdrew yesterday from their positions in and around New Haven, and organizers of the demonstrators there blamed the troops for the small attendance at the rally.

More than 20,000 people rallied Friday on the New Haven Green in support of Black Panther Party National Chairman Bobby G. Seale, who is facing charges of murder and kidnapping in the Superior Court in New Haven. About half that number attended a similar rally Saturday.

Organizers later cancelled a "Black Music Festival." originally scheduled to take place yesterday.

More than 2000 National Guardsmen moved into the city Thursday in preparation for the demonstrations, and 4000 Federal troops were flown to Massachusetts and Rhode Island and stood ready to intervene in the event of disorder.

The main rallies were peaceful, although minor skirmishes between guardsmen, police, and small groups of demonstrators took place on the Green after dark. Thirty-nine people were arrested during the incidents, mostly on minor charges.

The New Haven Panther Defense Committee-a predominantly white group which organized the demonstration-later criticized the demonstrators who remained on the Green after dark, saying. "Some of the people who have come to New Haven seem to have no intention of freeing Bobby Seale and the other New Haven Panthers." Douglas F. Miranda, Panther New England Area Captain, called the skirmishers "provocateurs" who are trying to crush the Black Panther Party.

Widespread fear of violence at the demonstration last week led New Haven Mayor Bartholomew F. Guida to call for the Guard. Connecticut Governor John Dempsey later sent a telegram to Attorney General Mitchell asking for the federal troops.

Bruce Allen of the Yale Strike Steering Committee said yesterday that the troop call-ups and the heavy press build-ups had kept the rally small. Demonstration organizers had originally expected between 35,000 and 50,000.