Harry N. Walters, an assistant secretary of the Army, former All-American football player and New York businessman, is in line to replace Robert P. Nimmo as administrator of the Veterans Administration, sources said yesterday.

Walters was called to the White House yesterday to talk with key administration officials, one source said, adding it is "95 percent sure" that President Reagan will nominate him to take control of the VA, which has proven to be a political embarrassment to the president. The announcement could come today or early next week, the sources said.

Representatives of the nation's three largest veterans' groups were called to the White House yesterday morning and told that Reagan had narrowed his choice to four people, including Everett Alvarez Jr., who was sworn in three months ago as deputy VA administrator, and Rep. Robin L. Beard (R-Tenn.), who was defeated Tuesday in a bid for the Senate.

White House officials told the veterans' representatives that Reagan wanted to move quickly to replace Nimmo, who resigned last month following months of criticism and calls by most major veterans' organizations for his ouster.

By late afternoon, Walters' name had become known to key members of both the Senate and the House Veterans' Affairs committees, and to a select number of leaders of veterans' groups.

None of those contacted by The Washington Post said they were enthusiastic about the pending nomination, but they said they expected that Walters would be quickly confirmed by the Senate.

A number of congressional leaders had urged Reagan to fill Nimmo's spot immediately after the elections because the agency currently is drawing up its 1984 budget proposal.

Several leaders of veterans' groups also had encouraged Reagan to nominate a Vietnam veteran to replace Nimmo, who served during World War II, because they believed he had been unsympathetic to the needs of Vietnam veterans, who in a few years will constitute the majority of veterans.

But it appears that Reagan will turn instead to a political friend with more experience in business than in the military, and without any experience in running an organization the size of the VA, with its $25 billion budget. Walters, 46, spent four years in the Army after having been graduated from the U.S. Military Academy in 1959. He did not serve in Vietnam.

Walters grew up in Cincinnati. At West Point, he was named an All-American fullback on the "lonely end" football team coached by Earl (Red) Blaik.

After serving in Hawaii, Laos and Thailand, Walters left the Army in 1963 to join the Standard Paper Manufacturing Co. in Richmond. He worked at other paper companies, including Kimberly Clark, until 1977, when he purchased and became president of the Potsdam Paper Co. in Potsdam, N.Y. Walters turned the small plant, which was losing money, into a profitable business, according to his military biography. In June, 1981, Reagan appointed Walters to his Penta- gon post as director of manpower and reserve affairs.

At Walters' confirmation hearings for the Pentagon job, Sen. Strom Thurmond (R-S.C.) criticized him for spending only four years in the military after leaving West Point. Walters responded that he had a family to support and had "made a good deal more money" from his paper business than he did in the Army.

Walters was unavailable for comment yesterday.

The Veterans Administration has been troubled since Reagan took office. Veterans' groups fear that the White House will try again to cut the agency's budget, something Congress prevented last year, and are expected to be critical of any Reagan nominee.