If your website is slow to load, doesn’t provide adequate security, or gives rise to too many error codes, Google will likely reduce your ranking by means of a penalty. In this case, your best option is to have a comprehensive SEO audit undertaken in order to identify any and all issues which might have given rise you receiving a penalty in the first place.
As an alternative to poor performance issues, Google will often also place a penalty on your website if you can be identified as having used black hat SEO techniques to improve your ranking. Such techniques can include (but are by no means limited to) keyword density spamming, article spinning, rapid link building and websites using spammed backlinks and paid traffic.