Brown: Not falling for 'fall guy'