James Madison left the office of president of the United States on March 4, 1817; James Monroe was inaugurated as the fifth president and Elizabeth Kortright Monroe became the nation's fourth first lady. The Madisons remained in the capital for another month, packing, going to parties, saying goodbye to their many friends, but on April 6 they returned to their estate in Orange County, Virginia, to their beloved Montpelier.

Montpelier was a beautiful and gracious home, originally built by Mr. Madison's father, James Madison, Sr., and steadily improved by James Madison, Jr. over the years. It was fronted by a four-columned portico, and the brick was covered in a limestone plaster to create a more elegant impression. Inside it was light and allowed the summer breezes to cool it. The walls were covered with paintings and busts in the fashion of the day, while the whole was decorated with silk draperies and French furniture. The grounds were planted with specimen trees including silver poplar, weeping willows, boxwood, walnut, and cedar of lebanons, reaching out to farm buildings and the slave quarters, past pear orchards and grape arbors to fields of grain and tobacco.

For Mr. Madison it was a permanent retirement. He lived nearly twenty years more, but he never again visited Washington. Only once did he make any sort of extended trip and that was to Richmond in 1829 for the Virginia Constitutional Convention. After Thomas Jefferson died in 1826 Mr. Madison became Rector of the University of Virginia, which necessitated numerous trips to Charlottesville. But if the Madisons stayed in place, the world came to them. They had streams of visitors, including his family and hers, leaders of American politics and European dignitaries. The Madisons were never short of company or out of touch with national politics and Washington gossip.

The couple remained financially stable until the 1830s, when a combination of circumstances began seriously to take their toll. They continued to entertain and to live graciously. Meanwhile increasing soil erosion and shifting conditions in the post-war international agricultural market created a farm depression throughout Virginia. But most of all there was the problem of John Payne Todd, Dolley Madison's son by her first marriage. Payne Todd never settled into a planter's life; he never made Montpelier his home; and he never found a career. Instead he was restless and he drifted: to New York, to Philadelphia, and to Washington; and then back again. He drank and he gambled. And he piled up debt after debt after debt.

The Madisons sent him money, but they could neither stop him nor keep up with his mounting obligations. In 1830 he went to debtors prison in Philadelphia. Mr. Madison found himself forced to sell off his land in Kentucky and to mortgage half of the Montpelier estate just to keep up with his step-son's debts. In 1834 he was forced to sell a parcel of slaves. Yet Dolley Madison loved her son, missed him, and wrote letters asking him to come home and visit them.

Partly to create an inheritance fund for his much younger wife, and partly to preserve the historical record, James Madison began arranging his papers. It was time-consuming work, and Dolley Madison spent hours every day devoted to the project. Before he died, Mr. Madison had edited his papers through 1787. By the mid-1830s, however, James Madison had become seriously ill, and Mrs. Madison was compelled to devote increasing hours to his physical care, to "comforting my sick patient" as she wrote her niece Dolly Cutts on May 11, 1836.

Among the problems that concerned the Madisons in their later years was what to do with his slaves, who numbered about one hundred. He considered whether or not he should free them in his will. A leader of the American Colonization Society, he offered to send them to Liberia, but his slaves declined in horror. He finally died at Montpelier on June 28, 1836. In his will he simply said: "I give and bequeath my ownership in the negroes and people of colour held by me to my dear wife," although he then added "but it is my desire that none of them should be sold without his or her consent, or in case of their misbehaviour; except that infant children may be sold with their parent who consents for them to be sold with him or her, and who consents to be sold."

After James Madison died Mrs. Madison remained at Montpelier for a year. One of her nieces, Anna Payne, came to live with her. Payne Todd also came for a stay, and Mrs. Madison mobilized her household to finish copying her husband's papers. In 1837 Congress authorized $30,000 as payment for the first installment of the Madison papers.

In the fall of 1837, however, Dolley Payne Madison decided to leave Montpelier for Washington, D.C., charging Payne Todd with the care of the plantation. It proved a disastrous decision. Nevertheless, she moved with Anna Payne into a house her sister Anna and her husband Richard Cutts had bought, located on Lafayette Square, the "Cutts" house. Thus Mrs. Madison ended her years of retirement and entered her next, and final, stage of life: her years as a widow living once again in the nation's capital.