Despite the fact that the palace is named after Razumovsky, its first owner was R. G. Levenvolde, close associate of Anna Ioanovna. But after the new empress, Elizaveta Petrovna, who did not very much favor the court of the former ruler, came to the throne, the house was left without a master.

In 1749, K. G. Razumovsky bought a house and ordered to build a stone palace (the former owner built it out of wood). Two architects took over the construction at once: A. F. Kokorinov, and then Zh. B. Vallin Delamothe. The building, built in the classical style, turned out to be captivating symmetrical: in the center of the facade there is a colonnade that connects the two floors and has six columns.

Externally, the palace has not changed much, although after Razumovsky it had more than one owner, but the interior decoration has undergone more and more changes every year. First, an Orphanage was located here, then the palace became known as the Orphan Institute, and since 1903, the building has housed a pedagogical institute.