You know it actually wasn't a brake issue. I had assumed it was the left hand side but in reality it was due to loose steering. At low speed there was no way to tell but at higher speeds the braking force really put more stress on the steering. Turns out I had about 1-1.5" of loose play in my steering wheel. So what I did was adjust the worm gear on the steering box to eliminate this play. Braking was no longer an issue.