Understanding the role of van der Waals forces in solids from first principles