A quick and effective way to see the contacts you're not connecting with but should: the "No Active Campaign" filter. On your Contacts section, select the Filters button, the first item is the "No Active Campaign" slider.
You should almost always have your contacts on some sort of campaign. You can pair this by also filtering for contacts which you've labeled their Status as Prospecting, Active, Future, or New. These ones you should certainly be staying in contact with.