As explained here -
http://www.bobatkins.com/photography/eosfaq/manual_focus_EOS.html - a purely mechanical adapter (with no optics) for mounting a Minolta MD lens on a Canon EOS body would result in a system that would be unable to focus at infinity. The lens would need to be closer to the sensor than such an adapter would allow. That's why they have optics and I think you're correct that the optics (which are generally simple and inexpensive) are degrading the image quality.
There's really no way around the problem (which is even worse for Canon FD lenses). If you don't need to focus to infinity you can try removing the optics from the adapter. Some allow the optics to be unscrewed while others require more serious efforts (like a hammer) to get the glass out! You may also be able to find glassless "macro" adapters. Depending on the lens and the adapter the focus distance could be long enough for non macro work, especially with telephoto lenses.