I mean the social contract is not always a physical thing and not always by explicit consent. Just by carrying out our part of the system and accepting the benefits of it (infrastructure/protection/stability) we are implicitly consenting to it.

That being said, I absolutely agree with you that it is always slanted heavily toward the wealthy and not to benefit the working class but only to keep them in line.