You use /bin/rpm -i (or -e) to install/remove packages in .rpm format. I
will guess that the "extra stuff" you installed was not installed using
/bin/rpm, so the rpm database has no way of knowing that the "stuff" exists.
If you ONLY install packages using /bin/rpm you can alway use it to remove
what has been installed. If you use any other method of installation, rpm
will not know about it.
Some makefiles include a make uninstall option, but if you've "cleaned up"
after the installation, it is unlikely to work.