To completely remove a package use:
sudo apt-get purge <package name>
This will remove the configuration as well. Though you may still have additional dependencies that were installed. To remove these do:
sudo apt-get autoremove
You could also start fresh again and make a backup after every package installation. Though this will involve a significant amount of time, disk space and patience.
You may also want to look at etckeeper. Etckeeper is a package which basically allows you to keep your /etc directory in version control. It supports several VCS, but I would suggest using git.
If not already installed you can install git by doing:
sudo apt-get install git.
Then setup the global email and username for the root user by doing:
sudo git config --global user.name "Your First and last name"
sudo git config --global user.email firstname.lastname@example.org
then install etckeeper:
sudo apt-get install etckeeper
IIRC etckeeper will notice that git is installed and automatically use it, and make an initail commit of your /etc directory.
If not you can manually edit the /etc/etckeeper/etckeeper.conf file and change it from this:
# The VCS to use.
# The VCS to use.
Then in your /etc/ directory run:
sudo etckeeper init
to initialize the repository, and then:
sudo etckeeper commit "Initial commit."
to make your initial commit.
A couple of additional config file changes you may want to consider are:
see the additional resources below for more details on these settings.
Additional information can be found here and here
With etckeeper rolling back a change to the configuration is as simple as a git revert. This also makes testing changes easier because you can create a branch to test your changes. If they fail you delete the branch and checkout the master branch, if you decide to keep them you can merge the changes into the master branch.
This will not solve all of the problems when rolling back (as some config is installed elsewhere on the filesystem, and won't fix your current issue), but does keep a record of what was done and when (for all the config data kept in your /etc/ directory.