Laws designed to outlaw discrimination in Britain "simply aren't working in practice", the peers said.
The report highlighted taxi drivers refusing to take guide dogs and wheelchair-using sports fans being forced to sit with opposing fans.
The government said it monitored those with a duty to make reasonable adjustments on disability issues.
Baroness Deech, who chaired the Lords committee said: "We have been struck by how disabled people are let down across the whole spectrum of life.
"Access to public buildings remains an unnecessary challenge. Public authorities can easily side-step their legal obligations to disabled people, and recent changes in the courts have led to disabled people finding it harder to fight discrimination."
One of the committee members, Baroness Brinton, told BBC Radio 4's Today programme the problems faced by disabled people on public transport were among the "major failings" highlighted.
"That stops disabled people being independent," she said. "Witness after witness demonstrated how either not bringing laws into effect or not enforcing them has had a major effect on their lives."
The committee examined the impact of the Equality Act 2010, which brought together reforms over the last 40 years and applies in England, Wales and Scotland. The Disability Discrimination Act 1995 still applies in Northern Ireland but has been updated.
Its report, which consulted more than 200 witnesses including disability charities, disabled people, and businesses, said increased fees for discrimination tribunals had led to an "abrupt fall" in claims since their introduction in 2003 and should be reconsidered.
The peers heard from many witnesses who had suffered "transport nightmares" because of poorly adapted stations and vehicles.
They expressed astonishment that seven stations on the new Crossrail network were due to open without step-free access - something which has since been changed and the committee says should be required in all new stations.
It highlighted that for 20 years successive governments had failed to ensure that taxi drivers were obliged to take passengers with wheelchairs and concluded that venues and vehicles that breach their legal obligations should face being stripped of their licences.
Dave Thompson was paralysed after being injured playing American football nearly 30 years ago. He says while some things have improved, disabled people face many barriers on a daily basis.
"Just being able to be spontaneous," he said, "that's something we can never do. My daughter calls it dad's military exercise. We have to plan everything in great detail - the taxi, the venue, the access.
"And then all that can be a waste of time because it all depends on the attitude of staff when we get there. It's really sad. Why is my pound worth less than someone else's pound?"
Tom Coleridge was shot and paralysed in Afghanistan six years ago. Now, even going to his local restaurant can prove difficult.
"I only want to take my girlfriend out for dinner," he said. "But it can often be a problem. Sometimes I am told to go round the back and fight my way through bins of rotting food and through the kitchen which feels really embarrassing.
"Some places just say no, sorry mate, you'll have to get out of your wheelchair if you want to come in. How am I supposed to do that?"
Transport minister Andrew Jones announced hours before the report was published he hoped this measure, which also bans charging a higher fare for carrying a wheelchair, would be in operation by the end of the year.
His evidence to the committee that he had to consider the burden on drivers and firms had been dismissed by the peers as "entirely unconvincing".
Baroness Deech added: "The government bears the ultimate responsibility for enabling disabled people to participate in society on equal terms, and we believe it is simply not discharging that responsibility.
"It's time to reverse the attitude that disabled people are an afterthought. Many of the changes we suggest are simple and do not require legislation."
Shadow work and pensions secretary Owen Smith said: "The Tories should hang their heads in shame at this report. It shows just how much more difficult they have made the lives of disabled people across Britain."
But a spokesman for the Department of Work and Pensions said: "We're committed to ensuring disabled people live their lives free from discrimination.
"That's why we've strengthened the Equality Act to create a level playing field and ensure that the law properly protects them.
"All organisations are under a duty to make reasonable adjustments and we closely monitor to make sure this is happening."