House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) and Senate Majority Leader Robert C. Byrd (D-W.Va.) sent unmistakable signals to President Carter this week that his legislative program will fare much better if there is improved consultation between the White House and congressional leaders.

The White House, in turn, has sent back a signal that it agrees with the leaders and will take needed steps to improve.

Byrd, in statements to reporters Tuesday and Wednesday, made clear he believed the White House should have consulted the leaders in advance on its energy proposals.

O'Neill, in a statement yesterday said he agreed the Senate majority leader should be consulted and "it's also obvious they'd better consult the Speaker of the House."

O'Neill also said he was "upset to hear that the former mayor of Pittsfield, Mass., a Republican, has been chosen chief of protocol."

(Evan Dobelle, former Pittsfield mayor and onetime aide in Massachusetts to Sen. Edward W. Brooke (R-Mass.), is expected to be named chief of protocol. Dobelle switched party allegiance and worked for Carter during his campaign.)

O'Neill indicated he was at least equally upset by the fact that Elliot L. Richardson, a former high Nixon administration official and anticipated GOP candidate for governor of Massachusetts, has been named ambassador to the Law of the Sea Conference. In neither case was O'Neill consulted in advance, he said, "I've has a little something to say with the persons involved," O'Neill said.

Both Byrd and O'Neill indicated yesterday they weren't quarelling with the President but merely want to make it easier for them to help him with the legislative program.

Byrd in an interview yesterday said that Frank Moore, Carter's congressional liaison chief, "Came to me [after Byrd's statements] and said, 'We're new at this, we apologize, you taught us a lesson.'"

Sources said O'Neill has been contacted by both Moore and Carter with reassurances.

O'Neill said the problem was "the newness of the Carter administration. They're just not aware of who should be informed."

Precisely who is to be blame for the lack of informational exchange between the White House and leaders isn't clear, but there has been some criticism of Moore in the past - even before the Carter took office - for failing to move smoothly in the delicate network of power relationships on Capitol Hill.

Moore, a member of the Carter Georgia group, isn't as familiar with Congress as some members believe he should be. They say he hasn't yet acquired the intuitive grasp of how to move in a situation to head off trouble and smooth the way for presidential proposals.

The problem on the energy proposal was brought out by the Democratic leaders at a Tuesday breakfast with the President, according to Byrd. The leaders said they were surprised to learn that the proposal was being put together between the White House and some members of Congress but without advance consultation with the leaders. Byrd said he hadn't known about it until he read it in the newspaper.

Byrd said that had he been consulted, he could have suggested the names of several congressional leaders in energy matters who had not been included in discussions and should have been.

Later that day Carter energy adviser James R. Schlesinger, at Byrd's suggestion, moved a scheduled meeting on the energy matter from the White House to the Senate and met with persons suggested by Byrd.

"I feel that for the good of the President he ought to let us know, so we can help him. After all, they're new at this. It's not a matter of being vindictive, but of smoothing things for him. The President impressed me as willing to go the extra mile," said Byrd.

Byrd made his feelings about lack of consultation known to reporters at a press conference Tuesday, and then at a breakfast meeting with reporters Wednesday. O'Neill made his remarks yesterday when asked about Byrd's statements.