The EU Citizens' Rights Directive 2004/38/EC. Clause 9 states EU citizens have an unrestricted right to visit for 3 months only.
Further clauses show some of the restrictions on stays over 3 months, for example clause 10 states the EU citizen must not be a burden on the host countries social assistance system. clause 12 says they must all be registered in the host country and, if the host countries government wishes, requires them to have ID cards.
Clause 22 allows the host country to suspend immigration on the basis of policy, security or health.
So the reality is, if the UK chose to enforce it, you could only come to the UK if you had work here and if you leave that job and are no longer employed you lose the right to stay here. You have to maintain employed status for 5 years to be granted permanent residence.
What are you expecting to be put in place post brexit? Presumably something stricter than the above so by definition you are looking to refuse entry to those who have jobs here which will be an issue for those industries that require that labour.
The EU Citizens' Rights Directive 2004/38/EC is more commonly referred to as the EU Freedom of Movement Directive, it is the rules around which Freedom of Movement is based so can't possibly go against it.