We are going to need some maps. So let's start with a map of the state, indicating the location of the cities of New York and Rochester, the latter being where I grew up.
I point that out to note that, growing up in Rochester, we referred to where we lived as "upstate." It wasn't complicated; it was just "not New York City." Which is how most people use the term: "Upstate" is everything north of the northern border of the Bronx. (Long Island, jutting out to the east of the city, just becomes "Long Island.")
New York versus everything else is the Wikipedia definition. But the article's "Talk" page -- where editors of Wikipedia articles discuss the article itself -- shows that this definition is itself debated.
Here is a quote from that page, written by NinjaRobotPirate (presumably not his real name):
It's kind of a subjective term. When you need to define subjective terms, sometimes it's best to do a survey of what the sources say, summarize them, and then report it in the lead. This is what I did in cult film, which is about a highly disputed term, both in fan and scholarly discussions. For the record, "anything outside NYC" is really only applicable to residents of NYC. Orange, Sullivan, and Dutchess counties really don't consider themselves to be upstate, and they would say that it starts further up toward Syracuse or Albany. This is discussed in The New York Times. I'm sure you could pull dozens of articles from The New York Times, The New Yorker, New York, The New York Observer, and The Village Voice just to start. After that, hit up the Hudson Valley newspapers (The Poughkeepsie Journal and Times Herald-Record, for example), and then the Southern Tier, Capital District, and further north.
Mr. RobotPirate links to a 2014 Times article exploring the same question. "[State capital] Albany’s working definition of upstate New York," Michael Pollak writes, "is based on what lies outside the Metropolitan Transportation Authority’s commuter rail area. Besides New York City and Long Island, it excludes Westchester, Rockland, Orange, Putnam and Dutchess Counties."
So that map has only the light blue below being "upstate."
Albany's working definition, though, isn't itself set in stone. Last month, Politico reported on a bill in the works that would set different standards for upstate and downstate. One definition is slightly larger than the map above, including Ulster County.
At other times, the map is smaller. Sometimes, somehow, it includes Long Island, which is nuts. Once, Gov. Andrew Cuomo proposed something called "Start-Up NY," where the "Up" was short for "Upstate." That proposal included parts of New York City.
The overlap of "people from or living in New York City" and "people on Twitter" is a powerful locus of snark, so when the subject comes up on social media, one can expect a rash of semi-helpful/dumb/funny responses, usually related to the city itself.
For out of town press: The usual rule of thumb is that "upstate" New York is everything north of Zabar's.— Nick Confessore (@nickconfessore) April 19, 2016
This, of course, circles back to the general "not New York City" idea.
But there's another exclusionary factor. People in Buffalo consider themselves residents of "Western New York," a whole different designation.
Wikipedia is even less clear on what counts as Western New York, offering two conflicting maps (as noted by author Josh Fruhlinger):
The good people at Yahoo Answers disagree on whether or not Buffalo is both "upstate" and "Western New York." There's no reason it can't be both, of course. Just as the Hudson River Valley (the stretch of counties leading up toward Albany from the city) is both "the Hudson River Valley" and "upstate" -- at least parts of it. (Buffalo's insistence that it is its own region is part of its general insecurity, I would say, perhaps stemming from losing four straight Super Bowls.)
Speaking of cities, let's check in on how Urban Dictionary would define "upstate New York." We've taken the liberty of replacing swear words with the word BUFFALO. (The typos are in the original text, obviously.)
Some people on here are some major stereotypical morons. First of all, get a BUFFALO geography book, Long Island is NOT part of Upstate. If you think that, then you are dumb as BUFFALO. You are also dumb as BUFFALO if you think that Upstate is all hicks. Do you not think there are cities in Upstate ny? less than 10% of people in Upstate NY live on farms, get a BUFFALO clue. There are 4, count em, 4 cities in upstae ny with metro populations of over 1,000,000 people. Buffalo [Ed. - this one was not a swear word], Rochester, Syracuse, and Albany/Schenectady/Troy. The entire population of Upstate NY is about 6,000,000 people. Do some BUFFALO math. If 4 cities have metro populations of over 1 million people, that means that between 4 and 5 million of the 6 million people that live in Upstate NY, live in either a city or a suburb. The vast majority of the remainder of the people in Upstate NY live in small cities, but cities nonetheless, like Ithaca, Elmira, Binghamton, and Watertown. NYC people, we in Upstate pay for your BUFFALO subway, highrises, taxicabs, and all of the other stuff with our thru the roof taxes that the Donwstate controlled government puts on us, have some BUFFALO respect!!!!
So: Anything north of the city, basically -- but with the important caveat that it's not "all hicks."
That's as good an answer as we'll get.
Update: Public Policy Polling asked real-life New Yorkers this question in a poll last week, as noted by Nathaniel Rakich on Twitter. The results? No one agrees at all.