Building work was begun by Ascanio Caffarelli in 1538 on family property situated close to the Palazzo dei Conservatori; it was completed after 1680.
The building looked out onto the courtyard of the Palazzo dei Conservatori while its interior included two gardens - (the Caffarelli Garden and that known as the Roman Garden); the large portal on the Via delle Tre Pile was the main entrance to the property.
Over the centuries the palazzo has undergone a number of alterations which have substantially modified its original appearance. Some remaining parts of the frescoed vaulted ceiling are now housed in the Museum of Rome.
From the beginning of the 19th century till the end of the First World War the building was occupied by the Prussian Embassy.
In 1918 it was taken back by the Rome City Council and partially demolished. A large terrace, (the Caffarelli Terrace) took the place of the upper floors in the Eastern wing, while the ground floor, which had been partially obliterated by excavations into the Temple of Capitoline Jupiter, was taken over by the new museum section (Mussolini Museum, subsequently the Museo Nuovo).