Left the established Church of Scotland at the Disruption of 1843. He then became minister in the Free Church of Scotland at Newington, Edinburgh. The church was one of the first built in Edinburgh after the Disruption. Begg was a key figure in the foundation of the Scottish Reformation Society in 1850 and the Protestant Alliance and was known not just for anti-Roman Catholicism but also his concern for working and living conditions. He was editor for The Bulwark or The Reformation Journal for 21 years from its beginning July, 1851. He also wrote frequently to The Witness. Together with Thomas Chalmers, Begg was a major influence behind the colony houses of Edinburgh, which were built between 1850 and 1910 as homes for artisans and skilled working-class families by philanthropic model dwellings companies. In the late 1850s, alarm at the high illegitimacy rates in the northeast counties led Begg to launch a moral crusade against the accommodation of unmarried male farm servants in bothies.