Multimodal Representation Learning and Set Attention for LWIR In-Scene Atmospheric Compensation

Document Type


Publication Date



A multimodal generative modeling approach combined with permutation-invariant set attention is investigated in this paper to support long-wave infrared (LWIR) in-scene atmospheric compensation. The generative model can produce realistic atmospheric state vectors (T;H2O;O3) and their corresponding transmittance, upwelling radiance, and downwelling radiance (TUD) vectors by sampling a low-dimensional space. Variational loss, LWIR radiative transfer loss and atmospheric state loss constrain the low-dimensional space, resulting in lower reconstruction error compared to standard mean-squared error approaches. A permutation-invariant network predicts the generative model low-dimensional components from in-scene data, allowing for simultaneous estimates of the atmospheric state and TUD vector. Forward modeling the predicted atmospheric state vector results in a second atmospheric compensation estimate. Results are reported for collected LWIR data and compared to Fast Line-of-Sight Atmospheric Analysis of Hypercubes - Infrared (FLAASH-IR), demonstrating commensurate performance when applied to a target detection scenario. Additionally, an approximate 8 times reduction in detection time is realized using this neural network-based algorithm compared to FLAASH-IR. Accelerating the target detection pipeline while providing multiple atmospheric estimates is necessary for many real-world, time sensitive tasks.


© The Authors

This is an open access article published by IEEE and distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. CC BY 4.0

The "Link to Full Text" on this page opens or downloads the open access article, hosted at the IEEE website. Shared in accordance with guidelines at Sherpa.

Author(s) marked [*] were enrolled in an AFIT graduate program at the time of publication.



Source Publication

IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing