DOCTORS ALWAYS LOOK out of place in somber, oaken courtrooms, as if someone in the TV production office had made a mistake and scheduled a segment of the "Bold Ones" on the wrong set. The men in the white coats either fidget nervously in the witness box, or put on a knowledgeable and worldly air, but the M.D.s feel no better under oath than the lawyer feels under the knife.

Medical authorities often testify as "expert witnesses" for either the prosecution or the defense, but only occasionally do doctors themselves become the center of the court's attention. Last month, however, one of the newest techniques in the medical profession went on trial in Detroit, as a parade of medical experts took the stand to discuss the ethics of psychosurgery.

The trial seemed like an unwritten epilogue to A Clockwork Orange. The 35-year-old man in the witness box could resemble an aged Alex, back in the public spotlight after a few years hidden from scrutiny. But somehow, the romanticized notion of brain transformation that Kubrick served up on the screen and Burgess depicted--to a lesser extent--in his novel seemed far away from the austere chambers of the Wayne County courtroom.

The controversy centered around Mr. L.--the only name used throughout the case--who has spent the last 18 years in Ionia State Hospital, a Michigan mental institution. Mr. L. entered the hospital on the advice of a lawyer after allegedly raping and murdering a 19-year-old Holland, Mich., girl, who worked at Kalamazoo State Hospital.

Mr. L. had himself committed to Ionia under Michigan's criminal psychopath statute, which prevented the state from prosecuting him as a criminal. The Michigan state legislature repealed the law in 1968.

FIVE MONTHS AFTER entering the hospital, Mr. L. killed a nurse and attacked an attendant, and since then he has remained in the mental institution as a criminally sexual psychopath.

In May 1972, Mr. L. received his first chance to reenter society after 18 years of confinement. Dr. E. Gordon Yudashkin, director of the Michigan State Department of Mental Health, approached him and inquired if Mr. L. wanted to participate in experimental psychosurgery.

The experimental program was being carried out by Dr. Ernst Rodin, chief of neurology at the Lafayette Clinic. Rodin's project involved using patients committed to state hospitals who had not responded to therapy. Rodin proposed to treat the patients with an experimental German drug or use psychosurgery.

The only drawback to the German drug--which controls aggressive behavior by inhibiting the production of testosterone--is that it has one rather unfortunate side effect. The drug induces impotence, an extreme emotional price to pay for "curing" violent behavior.

Psychosurgery not only seems less demanding emotionally, the technique has a better track record. About 500 operations have been performed since the technique was developed, and the success rate--depending on which doctors are consulted--is somewhere between 75 and 90 per cent.

Psychosurgery involves the destruction of a small section of brain--known as the amygdala--which supposedly controls aggressive behavior. A number of methods can be used to destroy the tissue, including ultrasound or laser beams. Only a very small section of tissue, about 1/600 of an inch, is damaged in the process.

WHEN PSYCHOSURGERY WAS first developed, many experts feared that the operations would produce vegetable cases like the lobotomies did a few years ago. So far, psychosurgery has proved relatively successful. The question in this trial encompass not only the ethics of psychosurgery, but the legal implications of the operation for the so-called criminally insane.

In Michigan, 18 persons institutionalized under the criminal psychopath statue still await release. Since 1968, when the law was repealed, 258 patients have been released. The state authorities considered the freed "criminals" cured to the extent that they could cope with society. Psychosurgery had not been employed, only traditional therapy methods.

The remaining 18--including Mr. L.--did not respond to treatment, according to state authorities. The organizers of the Lafayette project approached Mr. L. because they thought he had only a slight chance of reacting favorably to treatment.

Last October, Mr. L. agreed to participate in the project, and his parents signed the consent papers. Mr. L. was then transferred to Lafayette Clinic, where electrodes were implanted in his brain to determine the area from which the violence emanated.