Former vice president Walter F. Mondale and his staff are engaged in a delicate and unusual political dialogue with the Rev. Jesse Jackson, in the course of which they have sought his support for Mondale's presidential candidacy while attempting to impress on Jackson the difficulties of mounting a presidential campaign of his own.

The stated purpose of the talks, according to participants on both sides, is to trade ideas on boosting black voter registration and to provide Jackson, whose potential presidential candidacy could draw considerable black support from Mondale next year, with technical information on the Democrats' complex delegate-selection process.

Both sides describe the talks as conversations among friends and say the Mondale camp has been careful to avoid appearing heavy-handed in what can only be described as one of the more unusual relationships in the annals of political consulting. "We're not trying to scare him away," said Maxine Isaacs, Mondale's press secretary. "He's asked for our advice--as well as the advice of others--on the process, which we've been glad to provide him."

But Mondale's aides have used the conversations to attempt to persuade Jackson that a Mondale presidency would be good for black Americans. They also have assured Jackson that Mondale would meet the party's affirmative-action guidelines for minority delegates to the Democratic National Convention and would fight for a platform that supports the interests of blacks.

"We've talked to him about what the Mondale presidency could do for the black community and our desire for his support among black Americans," one aide said. They also have explained the obstacles to starting up a national campaign this late and to trying to translate popular support in polls into convention delegates, according to campaign aides.

Mondale officials say Jackson, who will make his decision about his candidacy in September, also has been told that Mondale will compete vigorously for black votes in next year's Democratic primaries, whether or not Jackson is in the race. Mondale's advisers see the dialogue as an opportunity to maintain a relationship with Jackson while he decides whether to run, and to benefit from his help in registering black voters next year if he doesn't.

Jackson said he doesn't believe that Mondale's campaign aides are trying to dissuade him from entering the race, adding that he has "great, great respect" for Mondale. He also said he has met with most of the other announced Democratic candidates. But he acknowledged that there was some potential tension in the talks with Mondale's campaign.

"Obviously some of his people are concerned about our running or not running," he said.

"We feel strongly that it's his decision, and we've not been trying to lobby him on it," said Jim Johnson, Mondale's acting campaign chairman. Several recent public opinion polls show Jackson running third behind Mondale and Sen. John Glenn (D-Ohio) among Democratic voters, with the support of about 7 to 10 percent of those surveyed. Those polls show that he draws significantly more votes from Mondale than from any other Democrat in the race.

Jackson said that he talked to Mondale recently about a five-point voter enforcement program--enforcement of federal voting rights legislation to eliminate obstacles to black registration in the South--for the Democratic Party. He said the program is the key to registering more black voters. Jackson said action by the Democratic Party to embrace his plan "would have an impact" on his decision to run.

But in an interview, Jackson emphatically expressed "no interest" in taking over the Democratic Party's voter registration drive.

At last week's DNC meeting, Mondale adviser John Reilly offered Jackson's name, among others, to DNC Chairman Charles T. Manatt as a possible leader of that effort, prompting speculation that the Mondale forces were looking for a plum to give Jackson to keep him out of the race. Reilly said it was after Manatt had asked to meet with Mondale to discuss the former vice president's proposal for the party to commit $4 million to voter registration. Mondale, who had pledged to raise $1 million, was not available, and Reilly attended instead.

"I responded to a question as to who I would suggest" to head the registration effort, Reilly said, "and I mentioned Jackson's name among others." But Reilly said there was no deal between the Mondale forces and Jackson, adding, "I've never discussed this with Jackson or his people."

The origin of the Mondale-Jackson talks are vague, but in a sense they have been going on since Mondale endorsed Richard M. Daley over Harold Washington in the Chicago Democratic mayoral primary, which ruptured the relationship between Jackson and Mondale.

But the most recent talks have taken place since Jackson declared his interest in entering the race.

Mondale adviser Tom Donilon, who got to know Jackson while working on President Carter's reelection campaign in 1980, has had the most contact with Jackson and his staff. Donilon, an expert on the delegate-selection process, said he called Jackson to inquire about a possible presidential candidacy because "the worst way to communicate is through the newspapers." He said he asked Jackson if he had problems with the Mondale campaign and how the campaign could get involved in voter registration.

Neither side will predict how the talks will end. As one friend of Jackson put it, "Both men are looking at each other's piece of the apple and it behooves them to keep talking."