BPS state counting using wall-crossing, holomorphic anomalies and modularity

In this thesis we examine the counting of BPS states using wall-crossing, holomorphic anomalies and modularity. We count BPS states that arise in two setups: multiple M5-branes wrapping P x T2, where P denotes a divisor inside a Calabi-Yau threefold and topological string theory on elliptic Calabi-Yau threefolds. The first setup has a dual description as type IIA string theory via a D4-D2-D0 brane system. Furthermore it leads to two descriptions depending on the size of P and T2 relative to each other. For the case of a small divisor P this setup is described by the (0,4) Maldacena-Strominger-Witten conformal field theory of a black hole in M-theory and for the case of small T2 the setup can by described by N=4 topological Yang-Mills theory on P. The BPS states are counted by the modified elliptic genus, which can be decomposed into a vector-valued modular form that provides the generating function for the BPS invariants and a Siegel-Narain theta function. In the first part we discuss the holomorphic anomaly of the modified elliptic genus for the case of two M5-branes and divisors with b2+(P)=1. Due to the wall-crossing effect the change in the generating function is captured by an indefinite theta function, which is a mock modular form. We use the Kontsevich-Soibelman wall-crossing formula to determine the jumps in the modified elliptic genus. Using the regularisation procedure for mock modular forms of Zwegers, modularity can be restored at the cost of holomorphicity. We show that the non-holomorphic completion is due to bound states of single M5-branes. At the attractor point in the moduli space we prove the holomorphic anomaly equation, which is compatible with the holomorphic anomaly equations observed in the context of N=4 Yang-Mills theory on P2 and E-strings on a del Pezzo surface. We calculate the generating functions of BPS invariants for the divisors P2, F0, F1 and the del Pezzo surface dP8 and dP9 (1/2 K3). In the second part we study the quantum geometry of elliptic Calabi-Yau threefolds and examine topological string theory on these spaces. We find a holomorphic anomaly equation for the topological amplitudes with respect to the base that is recursive in the genus and in the base class. The topological amplitudes with respect to the base can be expressed in terms of quasi-modular forms, which resembles the holomorphic anomaly. In particular this generalises a holomorphic anomaly discovered for the 1/2 K3. For genus zero and base F1 we prove the holomorphic anomaly by using mirror symmetry and we motivate our holomorphic anomaly equation by establishing the connection to the holomorphic anomaly equations of Bershadsky-Cecotti-Ooguri-Vafa. Using T-duality allows us to relate this anomaly to the anomaly for the case of D4-D2-D0 BPS state counting. We calculate the generating function of BPS invariants of 1/2 K3 for higher rank branes in topological string theory and by using algebraic-geometric techniques that were developed in the context of stability of sheaves