Yeah, it totally depends on the car. My car is pushing a little less than 400HP, but its borderline unsafe because it is a 25 year old car, with 25 year old suspension, and marginal brakes. Then again a 4th gen F-Body w/ more power is perfectly safe because its a newer car w/ superior parts.
I would "too much" would be something of over 400HP that could not stop about as well as a newer midsize car. A car that could lose traction in 1-4th gear at any speed would be pretty wild as well.