FOR MANY vacationing Americans who travel outside this country, the biggest problem with coming home is not jammed jets, empty wallets or the return to reality and work -- it's facing a U.S. Customs officer at the port of entry.

Arriving tourists innocent of any smuggling activities have been known to become tongue-tied and dry of throat when questioned about the value or origin of a piece of jewelry fished from the bottom of a suitcase by an unsmiling inspector. Some passengers are automatically thrown into a swivet when they watch Customs ferreting through their once-neatly-packed underwear, or when they have to wait in line, or when they can't find the keys to open their luggage.

One young Latin woman will never forget an incident at the New Orleans airport. She was en route with her children to Washington from a trip to Mexico and wondering nervously if the plane would take off before the snail-paced baggage search ended and she could reboard. An alert Customs officer, with drug contraband obviously on his mind, was leisurely examining every parcel and suitcase on the long table.

"What have we here?" he asked her idly, holding up a large clay cooking utensil from a Mexican market.

"Oh," the woman replied quickly in English without thinking, "just a little pot."

The unamused inspector then proceeded with renewed vigor to tear up everything she had packed, and she got back on the plane just as they were ready to close the cabin door.

Unfortunately, passing through Customs can't be avoided. The U.S. Customs Service, a Treasury department, serves the vital purpose of policing our borders by protecting this country against illegal entry of dangerous drugs and other contraband, and by collecting duties on legally imported goods. Inspectors processed 314 million people in fiscal 1981.

But a little knowledge can make your experience relatively painless.

If you are entering the United States on a plane or ship, you will be handed a customs declaration on board. Assuming that the value of the goods you are bringing back doesn't exceed your exemption, merely fill in the identification portion at the top and hand it to the Customs officer. When the officer asks what goods you bought you can declare them orally. However, if your purchases and gifts exceed your exemption, you must itemize everything on the back of the form; family exemptions can be grouped on a single declaration. (If you are driving back over the Mexican or Canadian border, the procedure is usually a little less formal, but you risk losing your exemptions if you happen to swing back and forth across the border.)

U.S. residents are permitted to return with $300 worth of articles duty free if they were traveling outside this country, including Canada and Mexico. If returning from "any U.S. insular possession (the U.S. Virgin Islands, American Samoa, or Guam), the exemption is $600," based on the "fair retail value of the articles where acquired." You may include one liter of wine, beer or liquor if 21 years of age (four liters, if at least three were purchased in an insular possession)--check individual state restrictions on importation of liquor and tobacco.

But you must have been out of the country at least 48 hours--except for Mexico and the U.S. Virgin Islands--and cannot have claimed these exemptions within the preceding 30 days. You can mail duty-free gifts from abroad to relatives and friends, provided each gift did not cost more than $25 and each recipient does not get more than $25 in gift shipments on any day (gifts from insular possessions can be worth up to $40).

If you exceed your limit, don't despair. Charges are often more reasonable that you might suppose. Customs will figure the duty owed Uncle Sam and accept your personal check in the exact amount if drawn on a U.S. bank or trust company.

Also be aware that the Federal Currency & Foreign Transactions Reporting Act, aimed at impeding organized crime and drug smugglers, requires travelers to file Form 4790 with Customs whenever they take out or bring into this country more than $5,000 in U.S. or foreign currency, or the equivalent in bearer instruments. You are legally permitted to carry more than $5,000--but you may be in serious trouble if you fail to report it.

Customs officials are fully aware that many Americans view the department as a bottleneck that often causes travelers to wind up an otherwise happy vacation or business trip in bad humor because of unnecessary red tape and delays in clearance (though Customs maintains that its surveys at two major airports gave a favorable comparison between average processing time for international flights versus domestic flights, figuring in the time it takes airlines to deliver luggage). They have been working on a number of programs aimed at providing faster processing without sacrificing efficacy.

A six-month experiment was just completed at Miami and Los Angeles international airports to determine if it would be a practical, time-saving move to perform some clearance functions for passengers while waiting for the baggage to arrive. An official said results were promising and some other airports--including Dulles--"might benefit" if they wanted to try it.

"One-Stop," now in use at several large airports (but not at Dulles or Baltimore Washington International), permits all international travelers to clear both Immigration and Customs checks at one time, regardless of their nationality or whether they have passports. Customs says it's a big timesaver.

"Citizen-Bypass," in use since 1978 at almost all international airports (including Dulles and BWI), allows U.S. citizens with passports "to bypass full inspection stations and go directly to Customs for processing." This is "an interim procedure" until "One-Stop" can be installed.

Finally, at Chicago's O'Hare Airport, they've taken Bypass one step further with a self-selection idea. Each arriving international passenger is given a brochure with instructions. If passengers determine there's nothing to declare, they go directly to the "green light" lane and are quickly cleared (unless an inspector senses a warning, based on "profiles" or some other sign, and orders a search). All other passengers go to the "red light" lane for normal processing.

A few tips from Customs to help speed your way:

* Put all purchases in one bag, if possible, along with the sales slips.

* Don't try to use a false purchase price--inspectors are experts on current valuation of goods.

* Don't offer to bring in packages for strangers, and don't try to hide an item to avoid paying duty (wearing clothes or jewelry purchased abroad in an attempt to cheat Uncle is amateurish and foolish). The punishment for false statements may include confiscation of the item, along with payment of a penalty equal to the U.S. value of the item, and criminal prosecution. Customs reports that in fiscal 1981 it made "22,271 drug seizures worth more than $5 billion." It also collected $16 million in duties from passenger baggage in that period (actually a small amount compared to the total collection figure of $9.1 billion, received primarily from commercial shipments).

* Don't pack any fruit--it probably will be confiscated because Customs is authorized to enforce Agriculture Department regulations to protect U.S. crops against pests.

* Don't bring in other restricted or prohibited items, such as absinthe, obscene publications or toxic substances.

You may include articles bought in "duty-free" shops as part of your personal exemption total, but they will still be subject to U.S. duty and restrictions if applicable.

* Before leaving this country, register with Customs your cameras, watches and any items that are of foreign origin--or likely to be viewed as such--if they have serial numbers (forms are available at any Customs office including ports of entry). This will give you proof of prior possession and prevent duty being charged upon re-entering the country. If an item has no serial number, you should carry an insurance report or appraisal to prove prior ownership.

* Request free publications prepared by the Treasury Department. Though the detailed booklet, "Know Before You Go," is temporarily unavailable because it is being revised, Customs offers a small leaflet on regulations titled "Pocket Hints" (publication No. 506), and another one listing "State Laws on Importing Alcoholic Beverages" (No.523). Write to U.S. Customs Service, P.O. Box 7118, Washington, D.C. 20044.