DOCTORS ALWAYS LOOK out of place in somber, oaken courtrooms, as if someone in the TV production office had made a mistake and scheduled a segment of the "Bold Ones" on the wrong set. The men in the white coats either fidget nervously in the witness box, or put on a knowledgeable and worldly air, but the M.D.s feel no better under oath than the lawyer feels under the knife.

Medical authorities often testify as "expert witnesses" for either the prosecution or the defense, but only occasionally do doctors themselves become the center of the court's attention. Last month, however, one of the newest techniques in the medical profession went on trial in Detroit, as a parade of medical experts took the stand to discuss the ethics of psychosurgery.

The trial seemed like an unwritten epilogue to A Clockwork Orange. The 35-year-old man in the witness box could resemble an aged Alex, back in the public spotlight after a few years hidden from scrutiny. But somehow, the romanticized notion of brain transformation that Kubrick served up on the screen and Burgess depicted--to a lesser extent--in his novel seemed far away from the austere chambers of the Wayne County courtroom.

The controversy centered around Mr. L.--the only name used throughout the case--who has spent the last 18 years in Ionia State Hospital, a Michigan mental institution. Mr. L. entered the hospital on the advice of a lawyer after allegedly raping and murdering a 19-year-old Holland, Mich., girl, who worked at Kalamazoo State Hospital.

Mr. L. had himself committed to Ionia under Michigan's criminal psychopath statute, which prevented the state from prosecuting him as a criminal. The Michigan state legislature repealed the law in 1968.

FIVE MONTHS AFTER entering the hospital, Mr. L. killed a nurse and attacked an attendant, and since then he has remained in the mental institution as a criminally sexual psychopath.

In May 1972, Mr. L. received his first chance to reenter society after 18 years of confinement. Dr. E. Gordon Yudashkin, director of the Michigan State Department of Mental Health, approached him and inquired if Mr. L. wanted to participate in experimental psychosurgery.

The experimental program was being carried out by Dr. Ernst Rodin, chief of neurology at the Lafayette Clinic. Rodin's project involved using patients committed to state hospitals who had not responded to therapy. Rodin proposed to treat the patients with an experimental German drug or use psychosurgery.

The only drawback to the German drug--which controls aggressive behavior by inhibiting the production of testosterone--is that it has one rather unfortunate side effect. The drug induces impotence, an extreme emotional price to pay for "curing" violent behavior.

Psychosurgery not only seems less demanding emotionally, the technique has a better track record. About 500 operations have been performed since the technique was developed, and the success rate--depending on which doctors are consulted--is somewhere between 75 and 90 per cent.

Psychosurgery involves the destruction of a small section of brain--known as the amygdala--which supposedly controls aggressive behavior. A number of methods can be used to destroy the tissue, including ultrasound or laser beams. Only a very small section of tissue, about 1/600 of an inch, is damaged in the process.

WHEN PSYCHOSURGERY WAS first developed, many experts feared that the operations would produce vegetable cases like the lobotomies did a few years ago. So far, psychosurgery has proved relatively successful. The question in this trial encompass not only the ethics of psychosurgery, but the legal implications of the operation for the so-called criminally insane.

In Michigan, 18 persons institutionalized under the criminal psychopath statue still await release. Since 1968, when the law was repealed, 258 patients have been released. The state authorities considered the freed "criminals" cured to the extent that they could cope with society. Psychosurgery had not been employed, only traditional therapy methods.

The remaining 18--including Mr. L.--did not respond to treatment, according to state authorities. The organizers of the Lafayette project approached Mr. L. because they thought he had only a slight chance of reacting favorably to treatment.

Last October, Mr. L. agreed to participate in the project, and his parents signed the consent papers. Mr. L. was then transferred to Lafayette Clinic, where electrodes were implanted in his brain to determine the area from which the violence emanated.

But before the tests were completed, Gabe Kaiminowitz, a civil liberties lawyer, filed a suit which claimed that the patients transferred from Ionia to Lafayette were being held involuntarily. The three-member court agreed to hear the case, and said it would decide on the constitutionality of Mr. L.'s confinement, and would also rule on the broader issue of psychosurgery.

Opposition to psychosurgery surfaced last year when the Medical Committee on Human Rights, a nationally-based group, accelerated efforts to bring psychosurgery under tight public control, or perhaps prohibit it entirely.

Dr. Peter R. Breggin, a staff member of the Washington School of Psychiatry, stood in the forefront of this effort. Breggin even placed his objections to the technique on public record in the Congressional Record.

UNTIL JUST RECENTLY, psychosurgery was not confined to the so-called criminally insane. One of the pioneers in the field, Harvard neurosurgeon Dr. Vernon H. Mark, has performed surgery only in cases where violent behavior is usually incidental, primarily in instances like epilepsy.

The New York Times said Mark, the co-author of a well-known book on psychosurgery called Violence and the Brain, "has done more than anyone else to popularize the notion that psychosurgery might work."

Medical experts disagree on the ethical aspects of psychosurgery, but the trial of Mr. L. raises legal implications that could have far greater effects. The court may even decide that Mr. L.'s case rests in the hands of the legislative branch, and not in the judiciary.

The three-member panel opened hearings on March 12, and quickly ruled that Mr. L. was indeed being held unconstitutionally, and ordered his release. Mr. L., however, did not feel confident enough to reenter society immediately, and is presently undergoing further treatment at Northville State Hospital.

The court set further hearing on the ethics of psychosurgery for late May, although it is still unclear whether the court has the legal right to rule in this matter. The three judges must basically consider three questions:

*Can the state infinitely detain so-called criminally insane persons if no hope for rehabilitation exists?

*Is an involuntarily hospitalized patient truly able to give "voluntary consent" to experimental procedures?

*If a relapse occurs after release, what measures will be taken?

In an irolic twist, the court must also decide a spin-off of the last question. Can Mr. L. be tried for the murder of 19-year-old Marilyn Kraai, the crime for which he was originally committed, now that he has obtained his release?

No one actually knows, particularly Donald A. Burge, the prosecutor of Kalamazoo County. Burge will hold off on an indictment until the court rules, although it seems clear that those still confined under the repealed law should still be protected. The statue of limitation could also apply.

So Mr. L. lives from one day to the next in a mental institution, and tries to prepare himself for the real world. His life has altered dramatically since he agreed to psychosurgery last October. He told the court March 27 that he changed his mind after reading about the technique. Now, after years of unproductive therapy, he seems to have responded to conventional treatment.

When the three-member court sits down in late May to thrash out the legal and medical complexities of psychosurgery, it will convene without Mr. L. But the medical profession itself will remain on trial in the somewhat eerie backdrop of a Detroit courtroom