Fairfield Public Library, one of the busiest libraries in the state, began in 1876 in the Fairfield Academy building on Town Green, and the library back then permitted people to borrow one book at a time when the library was open on Wednesday and Saturday evenings. The library, then called Fairfield Memorial Library, had 677 volumes, and it charged people $1 a year to borrow books.

In 1903, a building was erected at the corner of Post Road and Old Post Road in downtown Fairfield and became the library's new home. The town took over operation of the library in 1950, and additions were built in 1930, 1959, 1981 and 2004-05. The library's name was changed to its present name after the town assumed operation of the library.

Today, the library has a state-of-the-art children's department, a teen room, computer lab, gallery space for paintings, several meeting and reading rooms, and an entrance on the side of the building, next to the parking lot. Most of those features were part of the last renovation.