James R. Boyle, who was widely credited with solving a series of severe payroll problems in the D.C. school system and one of the few remaining high-level white administrators in the system, resigned yesterday under pressure from the new acting school superintendent, according to a letter obtained by The Washington Post.

In the letter, Boyle -- a payroll and computer expert hired by former superintendent Vincent E. Reed five years ago -- said Acting Superintendent James T. Guines told him Tuesday morning that "unnamed [school] board members" were "displeased" with his performance, and requested that Boyle tender his "written resignation or some other response" by 4:30 p.m. Tuesday.

According to sources, Boyle, who is director of the school system's finance department, was disturbed by a remark that Guines allegedly made at the Tuesday meeting that black people in general do not trust white people to handle their finances.

Boyle declined to comment on the purported statement. But he said in his letter of resignation, "I do not feel that any rebuttal or discussion of the statements made by you in my office yesterday would enhance either my position or that of the Board of Education. Needless to say, the contents of that meeting have made my decision [to leave] less complex."

Rung Pham, one of Boyle's top assistants who attended the meeting, said he recalls Guines "said something to that effect," but could not recall the acting superintendent's exact words.

Guines declined to comment on any statements that were not made directly in Boyle's resignation letter. "The white/black thing I'm not going to get into," he said.

The only other person present at the meeting was Rhody A. McCoy, the former New York school activist who was at the center of a bitter teachers strike in that city in 1968. He sought particularly to win dismissal of 19 white teachers and administrators who were considered by some to be unsympathetic to the black community. None of the 19 left the New York system.

Asked about the alleged statement by Guines, McCoy said: "I didn't hear it that way, although that [the assertion that blacks do not trust whites to handle their money] has been historically true . . . I'm not sure I heard that kind of statement [from Guines]," McCoy said. He added, "I think we're bordering on the black-white issue and I don't want to deal with that; that's suicidal."

McCoy joined Guines' staff last Wednesday as his executive assistant with a mandate to expand community involvement in local schools.

McCoy, a militant and highly controversial figure when he headed the Ocean Hill-Brownsville school district in New York, was present at Guines' meeting with Boyle simply to serve as a witness, Guines said yesterday.

Guines, clearly angered that the contents of the Boyle letter had become available to the press, said he did not demand Boyle's resignation. He said he merely told Boyle that by the end of the day, he wanted Boyle to decide whether he was going to stay with the school system or accept a job offer from the city's Office of Financial Management. Boyle decided after meeting Tuesday with Guines to accept the offer of the new post.

But in his letter of resignation, Boyle notes that on Monday, Guines "indicated that [he] would do anything possible" to keep Boyle with the school system. But on Tuesday morning, according to Boyle's letter, Guines said "unnamed board members" were displeased with Boyle's performance, and therefore, Guines had reconsidered his earlier request that Boyle stay with the school system.

Guines, who has already been criticized by board member Barbara Lett Simmons for being too heavily influenced by board member R. Calvin Lockridge, denied that any members had influenced him about Boyle.

Boyle has been credited by several school officials, including Washington Teachers Union President William H. Simons, for improving the system's overall accounting and management procedures. Matthew S. Watson, the city auditor, said that Boyle's finance department had the best accounting procedures of any city agency.

Arthur Andersen & Co., the nationally known accounting firm, audited the public schools accounts recently and determined that the school system had actually underspent its budget in 1980. Boyle at the time pointed to the "tight spending controls" he said he and his staff kept over the budget.

But some board members, including Lockridge and John E. Warren, have expressed dissatisfaction with the Andersen report, although they never offered specific evidence that there was anything wrong with the school system accounts or the audit itself.