I pondered this problem for a while and did not come up with a solution, only thoughts. Here is one. What if I detect a rootkit by placing rootkit associated files on a system. If the rootkit is installed, it should see its files and hide them from the user. If my files magically disappear, I know that I have that specific rootkit on the system. The next step would be to place rootkit files that the rootkit would accept and take ownership of on the system but these files would be modified to undo what the rootkit does. How would I undo the rootkit actions? Well, once I figure out how the rootkit operates, I can counter it. For research purposes, I would need to log the actions of the rootkit from this file I place on the system. Think of it as if I were placing a trojan horse on my own machine that gives us a back door into the realm of the rootkit.
There is also the possibility of accessing the system remotely or accessing the drives of a system through a removable hard drive kit or forensics kit. This would allow you to scan the data on the drive with a system that has not been compromised. If this approach was successful, I could set up agents that act as a buddy system scanning the nodes around them to ensure that they have not been compromised and rejecting communication from those that have been compromised. The agent could be included in antivirus software or as yet another package.
More thoughts for you? I welcome comments on them.