The proposed CIA operation planned by former clandestine operations chief Max Hugel that provoked a rare written protest from the House Intelligence Committee to President Reagan dealth with Mauritania, not Libya, administration sources said yesterday.

The sources declined to disclose the nature of the operation except to give the location as Mauritania, a poor, Moslem and military-ruled west African nation that earlier this year was the target of a pro Moroccan coup.

White House deputy press, secretary Larry Speakes yesterday denied the accuracy of a Newsweek report that said the Central Intelligence Agency had planned "a large-scale, multiphase and costly scheme to overthrow the Libyan regime" of Col. Muammar Qaddafi.

". . . The briefing described by Max Hugel in the current issue of Newsweek never took place," Speakes said at the daily White House briefing.

He declined to give any more information, saying: "We don't go into the business of discussing our intelligence."

The White House departed from its usual no-comment policy in an apparent effort to assist beleaguered CIA Director William J, Casey, whose judgment was questioned by the Newsweek article. Casey's judgment was even more strongly questioned by the House committee. According to sources, members of Congress of both parties thought the plan, whatever it ws, ill-prepared and "bizarre."

Hugel has resigned, following allegations of improper business activities. But the members of the committee have little confidence in Casey and wanted to be on record with a direct protest to President Reagan.

Under federal statutes dealing with congressional oversight of covert activities, the president must make a finding that such activities are necessary for national security.

Then the CIA director is obligated to inform the House and Senate Intelligence committees. While the committees cannot approve or disapprove such actions, their comments could lead to a change in plans.