Although not a soul departed Kennedy library architect I.M. Pei's office Thursday without a smile on his face and a blissful statement about cooperation and convergence on the objectives of the library, all did not go so well at the meeting for Harvard, the city or the community.

For under all the talk about agreement between the various Cantabridgians and library corporation personnel is the knowledge that the Kennedys are getting anxious to pull the whole complex out of Cambridge

A source not far removed from the current negitations said yesterday that the Kennedys are so mad at Harvard for not applying pressure on the community to relent on its museum-must-go demands, that they would not mind taking their money and running to the most advantageous site, probably Cape Cod at this point.

But the University is not totally out of the power scene; Hale Champion, financial vice president, is not about to let the benefits of the complex go down the drain, benefits that include a new Kennedy Institute building, 2.2 acres of property and a $10--million Institute of Politics endowment.

What the University and the city must hope for, and the gnat-like neighborhood groups must dread, is that the Kennedys decide on a middle course--one that may involve slipping in a museum in, or near, the archives. And with Pei's plans only in their infancy, it's a course that could very easily be taken.