Detecting and characterizing objects in images in the low signal-to-noise ratio regime is a critical issue in many areas such as astronomy or microscopy. In astronomy, the detection of exoplanets and their characterization by direct imaging from the Earth is a hot topic. A target star and its close environment (hosting potential exoplanets) are observed on short exposures. In microscopy, in-line holography is a cost-effective method for characterizing microscopic objects. Based on the recording of a hologram, it allows a digital focusing in any plane of the imaged 3-D volume. In these two fields, the object detection problem is made difficult by the low contrast between the objects and the nonstationary background of the recorded images.
In this thesis, we propose an unsupervised exoplanet detection and characterization algorithm based on the statistical modeling of background fluctuations. The method, based on a modeling of the statistical distribution of patches, captures their spatial covariances. It reaches a performance superior to state-of-the-art techniques on several datasets of the European high-contrast imager SPHERE operating at the Very Large Telescope. It produces statistically grounded and spatially-stationary detection maps in which detections can be performed at a constant probability of false alarm. It also produces photometrically unbiased spectral energy distributions of the detected sources. The use of a statistical model of the data leads to reliable photometric and astrometric accuracies. This methodological framework can be adapted to the detection of spatially-extended patterns in strong structured background, such as the diffraction patterns in holographic microscopy. We also propose robust approaches based on weighting strategies to reduce the influence of the numerous outliers present in real data. We show on holographic videos that the proposed weighting approach achieves a bias/variance tradeoff. In astronomy, the robustness improves the performance of our detection method in particular at close separations where the stellar residuals dominate. Our algorithms are adapted to benefit from the possible spectral diversity of the data, which improves the detection and characterization performance. All the algorithms developed are unsupervised: weighting and/or regularization parameters are estimated in a data-driven fashion. Beyond the applications in astronomy and microscopy, the signal processing methodologies introduced are general and could be applied to other detection and estimation problems.