The Secret Police
The main perpetrators of Stalin's terror in the 30s were the agents of the Secret Police. Now usually known by its later name, the KGB (Committee for State Security), the secret police went through several incarnations:
1917-22 Cheka (All-Russian Extraordinary Commission for Combating Counter-Revolution and Sabotage)
1922-23 GPU (State Political Administration)
1923-34 OGPU (United State Political Administration)
1934-43 NKVD (People's Commissariat for Internal Affairs)
As the GPU and the OGPU were subordinate to a larger organization, the NKVD, the latter was the term usually used to refer to the secret police. Sometimes it was referred to by the acronym "Narkomvnudel" made up of the first syllables of its component words. The NKVD also had control over the prison labor camps and censorship.
In 1926 the square was renamed Dzerzhinsky Square, after the first director of the Cheka, Feliks Dzerzhinsky (he was also known as "Iron Felix" and the "friend of children" for his work with war orphans!). Here's a close up of the building in the 20s (and note the trams!):
A statue of Dzerzhinsky stood in the middle of the square until it was removed by an angry mob after the 1991 coup attempt. In 1990 a monument to the victims of Stalin's GUlag was set up by Memorial nearby: a stone from the Solovki Prison Camp. The building housed administrative offices, interrogation rooms, and a prison. The NKVD building was expanded. There were even jokes that it was the "tallest building in Moscow" because "you can see Siberia from the top floor."
The KGB/FSB building in 1998 with the stone from Solovki
At the peak of the arrests in the mid-thirties, paranoia reigned in the cities. Intellectuals slept with suitcases of warm clothes and supplies ready under their beds. Arrests usually came at night, when there would be few witnesses. People lived in terror of the sound of a knock on the door or an elevator that opened at their floor. Private conversations were scrutinized as much as published work for any possible incriminating comment. In the later years jokes that satirized the Soviet state were rated according to how many years one could get for repeating them. Opportunists took advantage of this frenzy to rid themselves of opponents or of those standing in their way. One might be promoted by denouncing one's boss anonymously. One might get a better apartment by denouncing a neighbor. Of course a denouncer knew too much and often disappeared himself shortly after his victim.
Aside from being familiar with the atmosphere in Moscow in these days, Bulgakov had some direct experience with the NKVD. In 1925 his apartment was raided, and the police seized his diaries and the manuscript of his satirical novel, Heart of a Dog.
Many people were not so lucky and ended up in the Prison Camp System.
The Library of Congress Soviet Archive Exhibit has more on the Secret Police.