Abandoned by William Colgate Colby in 1911, the “mystery mansion of Madison Square” sat untouched for 27 years.
In the first decades of the 20th century, a formerly elegant brownstone in Manhattan sat abandoned. The pre-Civil War beauty, situated across from Madison Square Park at 22 East 23rd Street, was built for the Colgate family.
William Colgate — the father of the University’s earliest major benefactor, James Boorman Colgate — founded his soap, starch, and candle company two years after arriving in New York as an English immigrant in 1804 and apprenticing with a soap boiler. Headquartered at 4-6 Dutch Street, the company grew thanks to products like “cashmere bouquet” scented soap, according to the Encyclopedia of New York City.
By the mid-19th century, Colgate was rich and influential. He was an ardent Baptist who worshipped at the Baptist Tabernacle Church on Second Avenue and 11th Street, which he reportedly contributed the funds to help build. He and his wife, Maria, moved with their children to their new brownstone mansion at 22 East 23rd Street in 1851. After their daughter Mary married Robert Colby a few years later, Mary and her new husband moved into the brownstone and shared it with her parents.
It was a befitting new neighborhood for this prominent family. Officially made a city park in 1847, Madison Square was becoming the center of the most exclusive residential enclave in New York — close in proximity to theaters and shopping but with the peace and privacy of a well-manicured park as a buffer.
“The unique location where Madison Avenue begins afforded the family an uninterrupted view up the avenue, as well as a prestigious place on the park,” wrote Miriam Berman in Madison Square: The Park and Its Celebrated Landmarks.
William Colgate didn’t live long in his East 23rd Street house. He died in 1857, after his wife passed away in 1855. Through the next several decades, various descendants of the family occupied the brownstone — seemingly still lovely but no longer in a fashionable neighborhood. Madison Square Garden, the Fifth Avenue Hotel, and other businesses made the enclave more commercial.
The last occupant, according to one newspaper account, was William Colgate’s grandson William Colgate Colby, who was born inside the house in 1858. Colby, a Yale graduate, seems to be the family member who, in 1911, walked away from what was described as the “sole remaining private residence on Madison Square.”
“The legend tells that when his wife died 25 years ago, Colby walked out of the brownstone structure, turned the key in the lock, and thereafter left the house unoccupied — with furniture, books, and clothes untouched from that day on,” reported one newspaper in 1934.
“Colby used to come down twice a year from his New Hampshire retirement, and on those occasions his short, square-set figure would be seen entering or leaving the silent house,” the newspaper continued. “But no one ever lived in the place, and the tightly sealed shutters were never removed from the windows or doors.”
William Colgate Colby died in 1936, but the full story of the mystery mansion didn’t appear until 1944. That’s when William Colgate Colby’s sister, Jessie Colgate Colby, decided to sell the house to a developer who intended to put up a one-story taxpayer.
Berman’s account has the brownstone abandoned in 1918. “For 27 years the house and its contents stood untouched, visited only on occasion by a guard to check the burglar alarm,” she wrote. “Upon its sale, boards were removed from its entrance and its French windows were opened wide.”
In 1945 four floors of furnishings and personal items frozen in time from the Gilded Age were up for sale. “Thousands of people on the street who passed the house stopped to gape at the ancient brownstone that had once been one of the finest homes on Madison Square,” wrote Berman. “The crowds peered through the dusty parlor windows at the elegant objects of a Victorian past, covered with at least 6 inches of dust. There were gold chandeliers, marble columns, elaborately carved gilt-framed mirrors, paintings, and mahogany furniture.”
Artifacts from the 19th century included a rosewood piano, two gold harps, volunteer fire department helmets, ostrich plumes, waistcoats, crimping pins, and leghorn hats, according to a 1945 Philadelphia Inquirer report.
“Just as the sleeping palace might have remained unchanged year after long year, so the Colgate mansion was exactly as it had been in the summer of 1918 when the last of the family stepped across the threshold and rode away,” wrote the Inquirer.
“Beds were neatly made beneath blankets of dust; towels hung over the old-fashioned wash stands. Writing paper and pens were ready on the desks. Great cedar closets were hung with the hobble-skirted, pointy-shoe styles popular at the beginning of the First World War.”
Antique dealers were admitted inside to pick through the objects and take what was valuable. Onlookers were able to glimpse pieces as they were carted away.
One onlooker stated that he had been a friend of William Colgate Colby, and he recalled how Colby told him about “the happy evenings he spent in this house,” wrote Berman. The now-empty brownstone sold for less than its assessed value of $77,000, and the taxpayer building that went up in 1944 still stands today. It now houses a McDonald’s.