WASHINGTON--Kuwaiti military forces detected the firing of a Silkworm missile yesterday that hit a U.S.-flagged Kuwaiti tanker, and tried but failed to shoot it down with a missile of their own, Pentagon sources said.

The sources, who insisted on anonymity, said Kuwaiti forces had fired a small surface-to-air missile from the island of Faylaka at the Silkworm as it passed nearby on a path toward an oil-loading facility. At least 18 people on the U.S.-flagged tanker, including the ship's American captain, were wounded in the attack.

It was the second missile strike on an American-owned or American-flagged vessel in Kuwaiti waters in two days.

The tanker, the 81,283-ton Sea Isle City, is part of the Kuwaiti fleet reflagged by the United States in July to protect it from attack. Iran claims Kuwait aids Iraq, Iran's enemy in the seven-year Persian Gulf war.

The projectile that hit the Sea Isle City crashed into the crew accommodation quarters as the ship moved slowly toward a refueling pier near Kuwait's Al-Ahmadi oil terminal. The tanker was empty at the time.

One official identified the Kuwaiti missile as a "Strela," the old Western code name for the Soviet Union's SA-7.

According to the International Institute for Strategic Studies, a London-based organization that keeps track of military arsenals around the world, Kuwait is known to have purchased SA-7s from Russia. The SA-7 is a small, shoulder-fired anti-aircraft missile that relies on a heat-seeking warhead.

The White House said, "Iran has committed an outrageous act of aggression" by staging the missile attack. President Reagan declined to discuss "what our future action might be."

According to the Pentagon sources, the Kuwaiti effort to down the missile explained why that Persian Gulf state was so quick to identify the weapon that struck the Sea Isle City as a Silkworm, a Chinese-built antishipping missile acquired by Iran that has a range of about 50 miles.

Secretary of State George Shultz, in Israel to discuss Middle East peace, called the attack a "serious matter."

Kuwait's Cabinet met in emergency session to discuss it.

As the administration weighed its response to the attack, there were indications President Reagan was cancelling his scheduled weekend trip to his Camp David, Md., retreat.

One White House official, declining to be named, indicated that Reagan would not be making his routine weekend foray to the Catoctin Mountains, but would say no more about the President's plans.

Also yesterday, a helicopter carrying a U.S. television crew came under fire from an Iranian gunboat in the southern gulf, shipping sources reported.

Iran did not acknowledge firing the missile. Iranian President Ali Khamenei told worshippers at Tehran University yesterday that an American-flagged ship was "hit by a missile."

The official Islamic Republic News Agency, monitored in Cyprus, quoted the Iranian leader as saying: "Where the missile came from? The almighty knows better."

But, he added, "We have declared that they [the Americans] should not expect us to watch our ships being attacked and other ships remain safe. This is the nature of tension and how it spreads in the region."

At an airport news conference outside Tel Aviv, Shultz noted that the attacks took place in Kuwaiti waters, not the international waters the United States has pledged to protect.

Asked who was responsible, he said: "You probably can make a pretty good guess."