Instead of only activating in buffers that a real (which is more selective), simply activate in buffers that are file-visiting (this means that real buffers that aren't highlighted can exist, particularly term/eshell buffers).