Sag Harbor

Lia and I drove into Sag Harbor from the back roads, past potato fields turning to vineyards turning to shingled cottages, and the town announced itself with a church steeple that looked more New England than Long Island. That’s the Old Whalers’ Church, its Egyptian Revival tower rising over Union Street like a leftover from the town’s nineteenth-century boom, when Sag Harbor was one of the busiest whaling ports in America — busier, for a time, than New York City itself. Unlike its glossier neighbors East Hampton and Southampton, Sag Harbor kept its working waterfront intact, and that history is still legible on nearly every block.

Main Street and the whaling museum

We spent our first morning wandering Main Street, past the American Hotel’s wraparound porch and a run of nineteenth-century storefronts that once served whaling captains and now house bookshops and galleries. The Sag Harbor Whaling and Historical Museum, set in an 1845 Greek Revival mansion with a whale-jawbone archway out front, traces the town’s rise on sperm oil money and its long, slow fade after petroleum killed the trade. Lia lingered longest over the scrimshaw cases — carved whale teeth and bone, the sailors’ idle art from months at sea — while I got stuck reading captains’ logs of voyages that lasted years at a stretch.

The working harbor

Unlike the manicured marinas elsewhere on the East End, Sag Harbor’s waterfront still feels functional — fishing boats and sailboats share the docks, and the long public pier lets anyone walk out over the water for free. We ate lobster rolls at a shack near the marina, feet dangling over the bulkhead, watching a wooden schooner from the local sailing school tack out toward Shelter Island. In late spring the harbor fills with classic wooden boats for the annual Old Gaffers regatta, their sails patched and their varnish worn, a fitting fleet for a town that never quite gave up its maritime identity.

Literary ghosts and quiet streets

Sag Harbor has long drawn writers escaping the city — John Steinbeck lived and sailed from here for years, and his modest waterfront property is still pointed out by locals with a certain proprietary pride. We wandered the residential lanes off Madison Street, past whaling-captain houses with widow’s walks and picket fences, and stopped at Canio’s Books, a beloved independent shop that has hosted readings for decades. It’s a town that rewards slow walking rather than a checklist of sights — better to let an afternoon dissolve between the harbor and the porches.

Getting There

Sag Harbor sits on the North Fork side of Long Island’s East End, about a three-hour drive from Manhattan via the Long Island Expressway and Route 27, longer on a summer Friday when Hamptons traffic backs up for miles. The Hampton Jitney and the Long Island Rail Road both run from the city to nearby stations, with local taxis or rideshares covering the last few miles. From spring through fall, the small passenger-and-car ferry across Shelter Island Sound also connects Sag Harbor to the North Fork, a scenic shortcut worth timing your trip around.