Gerad,
I ran a setup last year that had me at zero caster on the left front. It seemed to pull similarly to yours. Mine was actually more noticeable at lower speeds when I was going straight in the pits. On our track, you kind of arc into turn 1, so you have to turn and be on the brakes slightly at the same time, so it was hard to notice.
Just pull your driver side fender off real quick and give your spindle a look from directly above it. Caster is very easily seen. I mean as far as it being positive, negative, or dang close to zero. Let us know which ball joint is farther forward, or if they're pretty close to even. Try to eliminate it as a possibility,and even if it doesn't appear to be an issue, at least its getting one more step toward what it could be.
I've never noticed an issue with tracking one way or the other being caster related. That's more of a toe thing.