The Senate voted unexpectedly yesterday to attach stiffening language to the nuclear trade agreement with China that goes into effect this week, casting doubt on whether the Chinese will stand by the pact.

The amendment, offered by Sen. John Glenn (D-Ohio) to the catchall government spending bill that is still under debate, would require the president to certify that materials sold to China and bought from it are subject to International Atomic Energy Agency safeguards against the spread of nuclear weapons. A motion to kill the amendment failed 59 to 28.

The safeguards issue has been central to the controversy that has dogged the pact since it was initialed by President Reagan during his 1981 visit to Peking. The agreement sets up the legal machinery with which the U.S. nuclear industry may bid for a share of the $6 billion China plans to spend on nuclear power-plant construction, but it relies mainly on verbal assurances from the Chinese that nuclear proliferation will not occur.

The Senate Foreign Relations Committee approved the agreement last month after adding qualifying language putting the Chinese on notice that their use of U.S.-supplied materials would be closely monitored. Glenn said that was not enough.

"If we don't put safeguards into this arrangement with China, the United States will be sending a message that the international safeguards system is tougher than it needs to be," Glenn told the Senate. "That message will be greeted with joy by countries like Iraq, Libya, South Africa and Pakistan," which are thought to be developing nuclear weapons.

Sen. Charles McC. Mathias Jr. (R-Md.) spoke at length against the amendment, which was brought up on short notice in apparent retaliation for the Senate's action in approving the pact on equally short notice last month.

"I suspect this will be fatal to the proposal," Mathias said after the vote on the measure. "I think the Department of State had pushed the Chinese just about as far as they were willing to go."

Mathias told the Senate that it would have been better to leave the pact as is, "and have some friendly influence with the Chinese, rather than take the risk of aborting the agreement and losing all influence whatsoever."

Republican sources indicated that the amendment will be a major target for them when the spending bill comes before a House-Senate conference committee.