Four major Capitol Hill construction projects have cost $180 million more than original estimates because of delays and management problems in the office of the Architect of the Capitol, the General Accounting Office has told Congress.

The GAO -- an auditing and investigative arm of Congress -- said that, because of cost overruns in the construction of the still unfinished Hart Senate Office Building, cheaper materials have had to be used and many planned features omitted. The structure, which originally was projected to cost $48 million in 1972, now is expected to far exceed the $137.7 million already appropriated.

The GAO also cited as sources of huge overruns the construction of the James Madison Memorial building of the Library of Congress, the modification of a former FBI records building so that it can be used as a House office annex, and the improvement and expansion of the Capitol Power Plant.

In its 71-page report, distributed over the weekend, the GAO urged Congress to make sure that realistic cost estimates are made and an adequate funding program adopted before it goes ahead with renovating two older Library of Congress buildings.

It further recommended that the architect's office change contracting procedures used on its two largest recent projects, the Hart and Madison buildings. GAO said it would save money by designing each project fully and award a single contract rather than "phasing" the work, with some of the projects built while other parts still are being designed.

George M. White, architect of the Capitol, whose office was the target of the GAO report, said yesterday he has adopted some of the GAO recommendations, resulting in tightened management controls over projects.

"I didn't disagree with what they said," White declared. But on eliminating phased construction, he said, "I think they took too strong a position. It's a pretty difficult thing to judge . . . particularly when a project involves an existing building that is occupied and must remain occupied."

White, in his formal reply to the GAO, urged it to start using "a less pejorative and, incidentally, more accurate term" than the overrun to describe rising costs. "To view preliminary estimates as though they were contractor's bids can obviously lead to inappropriate conclusions . . . and hence incorrect management decisions on future projects," he said.

White also said he believed his contracting method saved taxpayers $11 million on the Hart Building project. He also noted that inflation currently is causing building costs to rise about 18 percent a year.

The new report echoed findings the GAO reached in 1967 when it criticized cost overruns in constructing the Rayburn House Office Building, which was estimated to cost $64 million but exceeded $125 million. Rayburn was supervised by White's predecessor, the late J. George Stewart.

In its report, the GAO -- which considers Congress its direct boss -- said in a cautious way that the lawmakers themselves must take some of the blame. It noted, and yesterday White agreed, that succeeding generations of senators and representatives and congressional committees with changing memberships often change their minds about what should go into a project. These sometimes cause costly delays, the GAO noted, and occasionally inadequate funds are appropriated.

Here is how the GAO toted up the $180 million in cost overruns from the original estimates on the four recent projects:

* Hart Building: Original appropriation for the full project in 1972, $48 million; total cost to date $137.7 million, inadequate to complete building. Overrun to date, $89.7 million.

* Madison Library: Original authorization in 1965, $75 million; total cost to date, $130.7 million. Overrun to date, $65.7 million, not including $30 million for land and furnishings and a contractor's claim for $23.5 million in damages for construction delays he claims were not his fault.

* Renovation of former FBI records building at the foot of Capitol Hill as a House Office Building annex: Original authorization in 1975, $14.5 million; total cost to date, $27.2 million. Overrun to date, $12.7 million, a figure which is inadequate to meet original renovation specifications.

* Improvements and expansion of Capitol Power Plant: Original appropriation for full estimated cost in 1973, $18.6 million; total cost to date, $30.6 million. Overrun to date, $12 million.

The Hart project, criticized over the years by some members of Congress in both houses as a costly Taj Mahal, will be built with cheaper materials as a result of the tightened budget, the GAO said. For instance, it said plaster is being used for marble in some monumental stairway areas, and gypsum board is replacing plaster in other areas.

Among amenities being deleted are wood paneling in senators' offices, dining and cafeteria facilities, security stations, gymnasium faciilities and a central control system for computers.