A crucial technology in fully autonomous aerial swarms is collaborative SLAM (CSLAM), which enables the estimation of relative pose and global consistent trajectories of aerial robots. However, existing CSLAM systems do not prioritize relative localization accuracy, critical for close collaboration among UAVs. This paper presents $D^2$ SLAM, a novel decentralized and distributed ($D^2$) CSLAM system that covers two scenarios: near-field estimation for high accuracy state estimation in close range and far-field estimation for consistent global trajectory estimation. $D^2$ SLAM has a versatile and powerful front-end that can use stereo cameras or omnidirectional cameras as input, the former being easy to obtain and the latter being an excellent solution to the Field of View problem in relative localization. Our experiments verify $D^2$ SLAM achieves high accuracy in ego-motion estimation, relative localization, and global consistency. Moreover, distributed optimization algorithms are adopted to achieve the $D^2$ objective to allow the scale-up of the swarm and ensure robustness against network delays. We argue $D^2$ SLAM can be applied in a wide range of real-world applications.