`consult-xref` is autoloaded anyway. There are some packages that require `xref` (such as lispy through requiring etags) so without deferring, loading those packages may cause `consult.el` to be loaded.