The sharp swings in the stock market were matched yesterday in both the bond and the foreign exchange markets as investors and traders alike tried to figure out where the world is headed financially.

The dollar's value touched record lows against the Japanese yen and the West German mark before rebounding to close well above Monday's levels.

Both short- and long-term interest rates fell, with long-term government bond yields slipping below 9 percent, down more than one-tenth of a percentage point for the day. But the long-term rates, which were briefly above 10.45 percent a little more than two weeks ago, rose sharply before turning around.

Meanwhile, the Treasury successfully auctioned $9.76 billion worth of three-year notes at an average yield of 8.03 percent, about in line with market expectations. It will auction $4.75 billion worth of 30-year bonds today and $9.25 billion worth of 10-year notes Thursday.

The key federal funds rate -- the interest rate banks charge when they lend reserves, or cash, to each other -- fell below 6 percent, the first real break in that rate since the Federal Reserve began pumping cash into the jittery financial system nearly three weeks ago. The funds rate was down roughly a percentage point from Monday.

Some analysts took the decline in the funds rate as a sign that the market's rush to cash and to safe, short-term securities, such as three-month Treasury bills, has eased. The drop in the funds rate came even though the Fed did not add cash to the system yesterday, the analysts said.

Another indication of calmer conditions and a lessening of the desire for cash and highly liquid securities came Monday when yields on three-month Treasury bills, which had been unusually low compared to other short-term rates, climbed to more normal levels. Yesterday those rates eased in line with other rate declines.

The dollar, like bonds, dropped sharply and then rebounded. It touched a post-World War II low of about 1.7000 marks to the dollar, down from 1.7173 Monday, before rising back to 1.7173 in trading in New York.

The U.S. currency also hit a record low of 136.15 to the yen, down from 136.98 Monday, then rose to 137.50.

As usual, rumors helped drive the foreign exchange market. In New York, unconfirmed word that Reagan administration and congressional negotiators had agreed on a $60 billion federal budget deficit reduction package gave the dollar a boost. In Europe, a cut in the Dutch central bank's discount rate from 4.5 percent to 4.25 percent immediately sparked rumors that the West German Bundesbank was about to take a similar step. That weakened the mark relative to the dollar.

"There are these three big areas of uncertainty, the stock market, the bond market and the foreign exchange market," said Scott E. Pardee, vice chairman of Yamaichi International (America), an arm of a major Japanese securities house. "Any one of three could go any way."

Pardee, a former senior Federal Reserve official, usually has a firm view of where the markets are headed, but not this time. "For one of the first times in my life, I have not had a strong conviction of where things are headed," he said.

Most traders in the markets have tried to leave themselves as little exposed to risk as possible, Pardee said. "Many traders are pretty flat {in their view of the market}. They are trying to see what their customers will do."

One problem in trying to chart the future course of any of the three markets is that there is little if any information available about the impact on real economic activity from the simultaneous plunge of prices on most of the world's stock exchanges two weeks ago.

Some analysts believe any impact will be short-lived, with the U.S. economy in particular continuing to expand at a healthy clip. If that happens, the Fed could decide to withdraw some of the money it has been pumping into the economy and interest rates could rise again.

But other forecasters expect economic growth in the United States will run at between 1 percent and 2 percent in 1988, down a percentage point or more from forecasts made before stock prices plunged. In that case, interest rates likely would fall.

Meanwhile, the weakness of the dollar and the market gyrations have discouraged foreign investors who have become a major factor in U.S. financial markets, Pardee said.