The racial element exists because being able to identify "the other" is a key part of being able to protect your own "tribe" and there is a innate desire amongst most humans to want to cohabit with and defend "their tribe". This is the view across most of Planet earth and for 99.9999999999% of human history. You can dress it up and create as many social constructs as you want to define "what is British, what is English" etc but deep down everyone knows the difference between the "in group" and "out group". Not a moral judgement - it just is.
If I were to live in Nigeria and have children, they may have Nigerian passports and have Nigerian nationality, but they are not Nigerian and never will be.
Nigerians would never consider them as Nigerian and if anyone can rock up and receive a document that turns them into a Nigerian, and is considered as Nigerian as anyone whose ancestors inhabited the region for thousands of years, does that not deny or nullify the existence of the Nigerian ethnicity if anyone can have it at the stroke of a pen? Would that actually, not be quite racist?
Many people don't have any issues identifying who the English/British are when it comes to assigning them guilt for things like slavery or colonialism yet I never see them asking David Lammy (who identifies as a proud Englishman) to apologise for the slave trade.
If you disagree and believe that David Lammy shouldnt apologise for the slave trade on account of his heritage, as his ancestors come from a region where they were victims of it, then by diferentiating him from other English on the basis of his skin colour you are actually in line with those who you deem to be racist.