Prince George's County Executive Lawrence Hogan said yesterday that his county might curtail its participation in the regional mass transit system and contract with a private bus company unless Metro costs go down and bus service improves.

In a letter to the Metro board and in an interview at a Metro conference in Warrenton, Va., Hogan claimed that a projected 50 percent rise in the county's Metro subsidy, jurisdictional disputes, budget constraints and deteriorating bus service were all forcing him to reconsider Prince George's role in the regional system.

"We're not happy with the bus service and we're paying too damn much for it," said Hogan, whose county is one of eight political jurisdictions the comprises Metro. He added, however, that the bus service had improved considerably in recent weeks after he complained to Metro officials.

When asked if he intended to abandon Metro, which provides bus and rail service for the entire Washington area, Hogan, replied: "No, I've always been committed to the Metro system. But if we can't solve the financial problems, we'll have to choice."

Hogan complained in a recent letter to Metro board budget committee chairman Cleatus Barnett that "irreconsiderable diversity of interests among" the eight local governments of significant budget reductions. In our circumstances, however, we can no longer afford to have our (Metro) subsidies reflect the higher levels that jurisdictional compromises produce."

The Prince George's circumstances include trim, a voter imposed curb on county government spending. Metro's proposed fiscal 1981 budget would mean an estimated $7.6 million increase in the Prince George's County subsidy, which is budgeted at $16.6 million in the current fiscal year. Hogan in his letter said the increase could be as much as $12 million.

Metro general manager Richard S. Page called Hogan's letter "disturbing" and said that "it indicates some lack of knowledge about the factors that are driving up our costs." He said his proposed 1981 budget, which is presently under review by the board's budget committee, contains a number of options for reducing projected subsidies, including fare increases and cuts in service.

Page said that inflation is the biggest factor in Metro's rising costs. Metro's bus drivers and train operators, by contact, receive quarterly cost of living salary increases based on the consumer price index. Because inflation has been running in excess of 10 percent during some months this year, so have the annualized salary increases of the bus drivers.

Page acknowledged that "we do pay a price for taking many months to reach decisions." For example, it traditionally takes the Metro board three to four months to decide on a fare increase package, then another three months to run it through a cumber-some hearing process. "That's not the most efficient way," Page said, "but it happens to be politicially expedient and politically desirable.

The tax increase proposal was made by the transportation subcommittee of the commission appointed by Gov. Harry Hughes to study all aspects of state and local revenues and taxes. The full commission is scheduled to meet Dec. 12 to consider the subcommittee's proposal, which has offered as a way of making up for a revenue shortfall predicted by the Department of Transportation.

Among subcommittee proposals was a penny-a-gallon increase next July in the state gasoline tax, which has been at 9 cents a gallon since 1972, and a $10 increase in the annual vehicle regristration fee. Fees for trucks also would be raised.