In a political venture that was bizarre even by his own quirky standards, California Gov. Edmund G. Brown Jr. traveled 3,000 miles tonight not to make a speech here. The government, who sought to strike a blow for his twin political missions - balancing the federal budget and promoting his presidential prospects - had to cancel his talk minutes before it was to begin. Brown had been invited to testify before a state legislative hearing on the proposal for a constitutional convention to mandate a balanced federal budget. But that invitation came from the Republican legislative leadship. Brown's acceptance surprised and angered Democratic Gov. Hugh Gallen and House Minority Leader Chris Spirou, who oppose the balanced-budget proposal. Both are strong supporters of President Carter, whom Brown seems likely to challenge in the 1980 Democratic primary here. Spirou wrote Brown last week to disinvite him, but after extensive telephone negotiations the governor left Sacramento this morning thinking all problems resolved. When Brown came splashing up the state house steps in tonight's cold rain, he was met by Spirou, who insisted that an appearance by the governor would be "impossible under the circumstances." The circumstances were that the Democrats had concocted a ludicrous squabble at the start of tonight's hearing over the order in which witnesses would be called. At the height of it, the Democrats walked out in a fit of contrived fury. If Brown went ahead with his testimony, he would be aiding a GOP effort to embarrass Democrats. Afte a few minutes of hemming and hawing, he sheepishly announced that he would not testify after all. "I need Democratic friends in New Hampshire," he said meekly.