The Senate yesterday rejected by a wide margin an effort to stop a liberal filibuster against an attempt to limit busing of students for racial balance. The 48-to-38 vote was 22 short of the 60 needed to limit debate.

But the motion was framed in a way that caused some hard-line busing opponents to vote with liberals against it. A second cloture vote that should more accurately reflect Senate sentiment on the busing question will be held Monday. But however that comes out, the Senate plans to set aside the busing issue Wednesday to take up the tax bill, and may not get back to busing until after the August recess.

The anti-busing language is a two-part amendment to the annual authorization bill setting spending ceilings for the Justice Department.

The original language offered by Sen. Jesse Helms (R-N.C.) would prevent the department from taking part in court suits that could result in busing orders. A tougher provision added to the Helms amendment by Sen. J. Bennett Johnston (D-La.) would limit court-ordered busing to trips of not more than five miles or 15 minutes from a student's home.

The Johnston language is tailor-made to stop court-ordered busing in his home state, and was made retroactive to undo a busing order that Baton Rouge is scheduled to implement this fall. It would also undo other busing orders in effect if it could withstand a challenge to its constitutionality. It is one of several proposals pending in Congress to keep the courts from acting on sensitive social subjects.

The cloture motion by Majority Leader Howard H. Baker Jr. (R.-Tenn.) voted down yestrday would have struck down Johnston's amendment as non-germane to the bill, and was opposed by Johnston and several other busing opponents. The vote Monday will come on a Johnston cloture motion drafted in a way to protect his amendment from a point of order.

The busing fight, which consumed nearly a week of the Senate's time before the Forth of July recess and broke out again when the Senate reconvened Wednesday, began as a lonely filibuster by Sen. Lowell P. Weicker Jr. (R-Conn.). It has gathered support from liberal senators and organizations who consider it a major civil rights battle, especially the Johnston language.

The Regan Justice Department may be in no rush to file busing suits, even if Congress says nothing on the matter. Civil rights groups, however, could continue to do so. But if Congress could tell the courts not to order any busing beyond specified distances and could strike down orders in effect, that would put a serious crimp in efforts to bring about school integration by busing students beyond their neighborhood schools.

Some opponents of the Helms-Johnston amendment say they believe both parts, if enacted, ultimately would be overruled by the courts on grounds that Congress does not have the power to forbid the Justice Department and the courts to take actions to protect constitutional rights.