Quantum dots (QDs) are semiconductor nanocrystals with sizes on the order of several nanometers, whose bandgaps can be tuned by controlling the particle size. Owing to this bandgap tunability, QDs can absorb near-infrared (NIR) and short-wave infrared (SWIR) light, spectral regions that are difficult to access with conventional silicon-based devices. However, colloidal QDbased infrared photodetectors still suffer from intrinsically high dark current, trap-induced noise, and limited response speed. As a result, they exhibit fundamental performance gaps in terms of detectivity and speed–bandwidth product compared to epitaxial infrared detectors, highlighting the need for structural and architectural design strategies to overcome these limitations. In this review, we discuss recent advances in enhancing the spectral selectivity and sensitivity of infrared photodetectors through three-dimensional optical architectures, including metasurfaces and metamaterials. We focus in particular on design strategies and the underlying mechanisms responsible for performance enhancement, and we outline how structural approaches can be leveraged to effectively control the sensitivity and wavelength selectivity of QD-based infrared detectors.