WASHINGTON--Congressional hearings on the Iran-Contra affair will begin May 5, and former Reagan administration officials John Poindexter and Oliver North could testify publicly in early June under a tentative agreement with the office of independent counsel Lawrence Walsh, sorces said yesterday.

These sources, speaking on condition they not be identified by name, said Poindexter and North would receive limited grants of immunity before their public testimony, and would be first questioned in private by lawyers for the congressional committees.

At the same time, sources disclosed that House and Senate investigating committees intend to vote today for limited immunity for at least four people in the affair. They include Richard Gadd and Robert Owen, two men who were involved in once-secret efforts to aid the Contra rebels fighting the Nicaraguan government.

On another matter, one source said the Senate committee is expected to begin civil contempt proceedings against retired Air Force Maj. Gen. Richard Secord, who has refused to sign an order directing overseas banks to make available records from any accounts he might control.

Poindexter himself appeared under subpoena yesterday before a House subcommittee which wanted to question him about the Reagan administration's computer-security policy. He refused to answer any questions, citing his constitutional right against self-incrimination, as he had when questioned in December about his involvement in the Iran-Contra affair.

Final details on the compromise involving Poindexter and North were hashed out at a session involving top lawyers for the Senate and House investigating committees and assistants from Walsh's office. The session marked the culmination of an attempt to reach a compromise on how to proceed with questioning of President Reagan's former national security adviser and his former aide on the National Security Council.

Under the plan, subject to ratification by the House and Senate committees and by Walsh himself, Poindexter would receive limited immunity before North, but neither man would appear immediately as a witness at public hearings.

They would be questioned in private by congressional investigators several weeks in advance of their public appearances, and their testimony would be kept secret.

Some lawmakers suggested such an arrangement would extend the time Walsh would have to compile evidence for possible charges against the two men, but they conceded that attorneys for Poindexter and North might contest that.

The timetable appears to represent a victory for Walsh, who had asked Congress for a 90-day delay in any grant of immunity, arguing that acting more swiftly would jeopardize his criminal investigation.

"Walsh essentially got his 90 days," said one congressional source, who asked not to be identified by name.