Optimization models are widely used in power system engineering to improve efficiency and cost effectiveness. The main challenge in power system optimization is nonconvexities that arise from various sources, such as binary decisions for investment or commitment for power generators and nonlinear physical constraints for electric current. The primary focus of this thesis is on important nonconvex planning and operational optimization problems in power systems and energy markets. We improve decision making for such problems by proposing suitable models and developing algorithms to efficiently solve them. We illustrate their benefits by extensive case studies. First, we study a generation capacity expansion problem that combines investment pl...