LABOR DAY is rather a test of the garden's design and most of us, needless to say, fail sadly.

Nothing much is in bloom - a few clematis, Japanese anemones, some perversely early chrysanthemums, a few beatup roses - and yet everything has a through-the-wars look to it.

If we had thought ahead, and had been willing to sacrifice a few enthusiasms, we could have had fine effects this time of year from ivy, photinia, arundo, miscanthus, well-proportioned paving and pools or basins of water.

Architects sometimes deplore, and rightly deplore, the fanatical plant-lover's garden as a mish-mash shapeless greenery. I have seen gardens designed to look dandy all the year, with white marble and ivy and juniper - and these can be exciting the end of August.

But no gardener is satisfied with them. The things the gardener likes, the larkspur, poppies, roses, lilies, peonies, irises, daylilies, all have their price. They have their great seasons and their off-seasons.

I never plant an iris, say, without thinking how dull it will look for much of the year. And there are certain peonies I like that look bad, no matter what rose-tinted authors may say, from July on.

Let me point out there does not have to be any compromise at all in a garden. Indeed, one of the headiest discoveries of this world is the sudden awareness that if you like, you can have a solid quarter-acre of lambs ears, and nothing else. I knew a fellow who had nothing, literally nothing, but hybrid tea roses, and the paths were only wide enough to squeeze through. At some times of the year (much of the year, to be plain) the effect was less than lovely, and even when they were in full bloom the place resembled a particularly vigorous splatter-dash job.

But roses were his passion. He knew as well as anybody that he had gone overboard on roses, and he knew as well as anybody else what fine effects you get from simple masses contrasted with smooth grass or stone or water. He knew arbors are fine places to sit - he roof all dappled with grapevines - and the sharp delights of those clear and fiery things that bloom between February and mid-April (crocuses, wild anemones, little daffodils like "Tete a Tete," and wild tulips, squills, snowdrops and all the rest) appealed to him as they do to anybody else with eyes.

Yet he knew he would never enjoy those things in his garden, because every time he saw a drift of Iris danfordiae, say, he would think only that if they were not there, there would be space for a rose bush.

There are many nuts of his sort. The main trouble comes when the gardener is not really a fanatic for roses or daffodils or irises (willing to forego everything else for them) but overplants them anyway.

A most beautiful small garden, the size of a living room, was shown in London a few years back at the Chelsea Show. It had surrounding walls of white, with a triple arch in front of the white wall at the back. In the arches were three large clay tubs holding an orange tree or pomegranate - something glossy and subtropical - and angel trumpets (Datura suaveolens). There was a narrow canal of glazed tiles running down the middle with low myrtle hedges. There was a row of narrow conifers against one long wall. The floor was unglazed tiles decorated at intervals with small azulejos. There were vases of lustreware filled with cut roses against the cypresses and junipers.

Few gardens were ever more successful. But needless to say it would be a problem to keep buying roses for the vases, and any good grimy gardener would soon lose his mind thinking of all the treasures he could grow where that paving was. In the canal itself (neatly filled with good loam) considerable irises might flouish.

So I would be cautious using "discipline" on any gardener, and I would not hammer too hard at him on the value of paving, sheets of water, arbors, pavilions and the imperative of broad simple effects and noble scale

Many gardeners, after all, are not broad and simple and noble by nature.

The point I would like to-make is that in a small garden in town you cannot have it both ways.

I know form quite long experience that most pleasure from a garden that is presentable throughout the year with a nod here to spring, a salute yonder to fall and a tribute elsewhere to winter. Such gardeners will have to peonies, an early and a late one. They will have three irises. A clump a dozen tulip bulbs. A tuft or two of primroses and a handful of daffodils.

They will have a six-foot pool for a dozen goldfish and two modest-sized water lilies. They will have three clumps of phlox, four clematis vines, one grape. A few tuberoses, three small dahlias. Ten cushion chrysanthemums and five Japanese anemones. Three hostas.

An upright juniper like 'Skyrocket'. One climbing rose, perhaps 'Golden Showers.' A bergenia, a small varieties of ivy, one good-sized box bush.

An arbor beneath which four people can sit. Elsewhere, one good solid wood bench. Four clumps of daylily starting with the tiny 'Bitsy.'

Such a garden can give enormou s pleasure. The great danger - and it is a true hazard - arises when the gardener is echanted with the three primroses and says, "Why not explore primroses?" and suddenly has 225 sorts. Or suddenly notices he is short on lilies and gets a representative collection of 25 bulbs.

Or else he reads (in this very space, alas) that irises look ever so much better when you see the flowers rank upon rank, and says, "Earthman is right, they do look better that way."

He promptly acquires 50 varieties and plants them so that (sure enough) they look dazzling in one great block.

But what happens when the garden is not big enough, and no garden is, to have dandy displays of early spring bulbs, plenty of azaleas, lots of tulips, then a real show of irises. And don't forget peonies, you really need 100 or so to get the full range of peony variation and beauty. And poppies. I do not see how anybody can resist poppies.

But then much is to be said for cornflowers and larkspur. How often we ignore those wonderful animals. And of course pansies never look so good as in clotted masses. And why do we ignore the skunk cabbage? Big drifts of them in a bog, with Siberian irises beyond.

Japanese irises are even more gorgeous, and anyone can raise them from seed. Surely we can manage a small rice-paddy type site for them. Not too many, just 50 or 60?

But then there is the long summer. We need a great many phlox. Yet the lily pool should be as large as possible, in order to get the full effect of unruffled water and still have plenty of water lilies. And water cannas, and papyrus, and pickerel weed and . . .

In winter, that gardener suddenly realizes, it looks sad, so he busies himself with yews and hollies and camellias.

And long before anything settles down, some new lack is detected and some new enthusiasm embraced.

Which brings us back to the end of August, and why the garden looks bad. The answer is not few more clumps of physostegia or ligularias or early dahlias.

The answer is thought. Would it be better to have the brimming fountain basin or would it be better to have six more peonies?

It does not bother me in the least if the garden is solid with zinnias or roses or any other obsession. And it certainly doesn't bother me if the garden depends almost entirely on stone, water, a few shrubs and scarcely a flower in it.

What does bother me, because I know how gardeners hurt, is the garden in which the gardener has never considered his possibilities or consulted his heart sufficiently to know what he is willing to live without, and what he is not willing to live without.

His gardening life, therefore, is a revolving confusion born of impossibility. And that is the life of most town gardeners.

Even that is not a total loss. Many delightful things, mostly accidental, are seen. And of course some gardeners can stand the confusion and anxiety of such a garden better than others. It all depends on the gardener. But in general, the effect will be better and the anxiety much less, if the gardener comes early to a decision in the brain just how far he will willingly go for peonies (as it were) and then says: "No. I know I have to have it, and it's marvelous and all that. And no."