Optimization of Remote Sensing Networks for
Time-sensitive Detection of Fine Scale Damage to
Critical Infrastructure

Infrastructure Management and Extreme Events (IMEE) program
NSF Award #: 1361222

Team

San Diego State University – Department of Geography (lead)
Douglas Stow (Co-PI), Lloyd (Pete) Coulter (Project Manager), Emanual Storey (PhD student), Sory Toure (PhD student), Andrew Kerr (MS student), Garrick MacDonald (MS student), Eugene Schweizer (MS student), Christopher Chen (Intern)

San Diego State University – Department of Electrical and Computer Engineering
Sunil Kumar (Professor)

University of New Mexico
Christopher Lippitt (Co-PI), Andy Loerch (MS student), Tammira Taylor (BS student)

Consultants
Richard McCreight (NEOS Ltd., airborne data consulting) and Nicholas Zamora (software development support)

Background

Within the first hours following a hazard event (e.g., earthquake, flood, tsunami, wildfire passage, nuclear accident, etc.) the key priority is to initiate life-saving activities. First responders and emergency managers need validated situational awareness of the status of critical infrastructure (e.g., utilities, bridges, hospitals, dams, etc.). The most reliable, detailed, and comprehensive means for early and documentable reconnaissance of post-event damage assessment is through low cost airborne imaging systems supported by semi-automated image processing and analysis, and coordinated image/map dissemination capabilities. As the only synoptic sensing technology available, remote sensing represents a critical source of information on the status of infrastructure following hazard events.

The focus of this study is on assessing damage to infrastructure following a major hazard event using airborne remote sensing. The premise is that some infrastructure, particular in cities, is so critical to saving human lives and supporting emergency response actions that near real-time information on the damage status of such infrastructure is essential and yet may be difficult to ascertain with conventional, ground observations and sensor networks. We hypothesize that the solution to this post-hazard information access challenge is to design flexible, ready-to-deploy, time-sensitive remote sensing systems (TSRSS) based on a network of airborne platforms and digital cameras. Our team is collaborating on research pertaining to important elements of end-to-end TSRSS that supports post-disaster assessment of damage to critical infrastructure and allocation of emergency response resources.

The seven critical elements of an end-to-end system for rapid infrastructure assessment from aerial image based damage detection are: (1) pre-event planning and baseline data preparation, (2) platform/sensor type and deployment (i.e., tasking), (3) capture and transmission of image data (or derived products) from airborne platforms to ground-based command and control centers, (4) precise registration (alignment) of time sequential images so that they may be compared (either on-board the aircraft or on the ground following transmission), (5) detection of changes evident within airborne images, (6) dissemination of primary data and derived information to analysts and first responders, and (7) the use of that information in a decision process of some value (e.g., recue prioritization, improved routing, evacuation, etc.).

An approach referred to as repeat station imaging (RSI) (formerly frame center matching) is used during airborne image capture and processing, in order to yield highly precise co-registration of airborne images captured over time. Precise co-registration of multitemporal airborne images is required for automated change detection, and also facilitates rapid visual change detection. RSI is based upon matching imaging stations in terms of horizontal position and altitude between multitemporal image acquisitions. When image frames are captured from exactly the same imaging station in the sky between multitemporal acquisitions, there is no parallax between images and they exhibit the same terrain related geometric distortions (they are essentially carbon copies of each other). SDSU has developed specialized procedures for precisely collecting airborne images from pre-determined image stations and co-registering multitemporal images collected from matched stations. The procedures enables automated and precise co-registration of multitemporal imagery with ultra-high spatial resolution. The simplified image registration approach enables automated and accurate image co-registration for near real-time change detection. Examples “b”-“g” in Figure 1 illustrate the differences between non-station matched image sets and station matched image sets.


Figure 1. Example of repeat station imaging approach for precise registration. (a) Time-2 image chip is displayed atop a lighter toned (i.e., washed-out) Time-1 image. The quality of the spatial co-registration between the 8 cm spatial resolution image sets is evident. A vehicle turning the corner is only present in the smaller image chip, and not in the larger chip. View angle differences between non-station matched images (b & c) are apparent, while view angle replication between station matched images (e & f) is demonstrated. Station matched images align precisely when co-registered (g), compared to non-station matched images (d) which do not align well and are not appropriate for detailed change detection.

Achieving reliable results from automated detection, delineation and/or identification of post-disaster damage based on registered, high spatial resolution multitemporal image sets is challenging. However, some form of automated, rapid detection and mapping is critical to meeting information timeliness requirements. Automated damage detection may be based on multitemporal differencing of image brightness, texture and/or spectral transforms derived from normalized repeat-pass imagery. Once changes are detected, maps of potential damage accompanied by before and after disaster images may be provided to an image analyst who may accept, edit or over-ride the detection results. As part of a Department of Homeland Security (DHS) funded project for post-earthquake disaster response, our team developed and demonstrated a system for rapid and automated change detection. We participated in the Research and Experimentation for Local and International Emergency and First-Responders (RELIEF) event at Camp Roberts, CA in August 2012, and demonstrated rapid multitemporal image collection with automated image co-registration and change detection (Figure 2).


Figure 2. Multitemporal color digital camera images (a & b) and a darkened Time-2 with automated change detection results displayed in yellow (c). The image set was captured using the repeat station imaging approach.

Return to Top

Objectives

  • Investigate the information requirements of infrastructure managers following hazard events.
  • Examine factors affecting the timeliness of delivery from airborne remote sensing systems.
  • Analyze factors affecting the reliability of infrastructure damage detection.
  • Evaluate the capacity of current remote sensing technology and practice to address information requirements of various emergency response and facilities managers.
  • Assess the viability of design and operation paradigms borrowed from systems engineering, operations research, and software engineering to enable the design of TSRSS and networks to meet user requirements.
  • Research and develop tools and techniques for automated change detection.

Return to Top

Research Questions

  • What infrastructure is critical, how soon after a natural disaster is information about critical infrastructure condition needed, and what information can be supplied by remote sensing?
  • How can TSRSS be designed to satisfy the information needs of emergency and infrastructure managers following natural and anthropogenic hazard events?
  • How well can geographic and engineering theories of communication and complex systems design improve the design and effectiveness of TSRSS of infrastructure damage assessment?
  • What are the image collection requirements that enable detailed damage detection required by emergency response and infrastructure managers?
  • Does faster information delivery from remote sensing necessitate a reduction in the reliability of the information provided?
  • How much time would lapse from determination of an information need to delivery of sufficiently reliable information derived from an optimized airborne TSRSS?
  • What are the components of a TSRSS that hold the most promise for minimizing time expenditure and what technical refinements are needed to generate reliable information?
  • What types of damage features can be detected semi-automatically with a degree of reliability through implementation of a TSRSS based on the repeat station imaging approach?

Return to Top

Research Plan

The research plan includes theoretical exploration and empirical implementation of a remote sensing communication model (RSCM) developed by Dr. Lippitt, as well as testing pre-event flight planning, image registration, and automated damage detection tools, to determine the timeliness and accuracy characteristics of these TSRSS components. We are working with existing remotely sensed data sets already collected by the team and newly acquired imagery that more realistically represented damage features of primary interest. Surveys of participating emergency managers during the project (San Diego County Office of Emergency Services (OES), City of Albuquerque Office of Emergency Management, Los Angeles County Fire Department, and U.S. Civil Air Patrol) are guiding image data collection requirements. The research methodology is organized as follows:

Survey emergency managers

Optimal configuration of remote sensing systems requires that information delivered to the user be both relevant to and readily employed by that user. Therefore, it is critical to understand the types of information required by infrastructure managers following hazard events, the timeliness in which that information is required, and the levels of geometric and thematic accuracy necessary to facilitate effective decision making. To determine this information a series of interviews is being conducted with the hazard and infrastructure managers listed above. A two-stage interview process is being employed to: (1) determine the technical capacity, training, and decision processes of infrastructure manager in each jurisdiction and (2) determine the qualitative utility and desirability of products derived from TSRSS.

Quantify time for collection and delivery of primary image data

We are utilizing RSCM to simulate the timeliness of primary image data delivery for several possible TSRSS configurations under a range of hazard scenarios identified by hazard and infrastructure managers. We are also utilizing F-Planar flight planning software from TerraPan Labs LLC to enable rapid and interactive flight planning for repeat station imaging. This tool delivers flight plan information (e.g., number of flight lines and images) to simulate and estimate time required for the tasking, acquisition, and data transmission portions of the TSRSS for the available assets in each region.

Develop analytical model for image collection specifications

We are developing an analytical model based on photogrammetric principles to explain the interdependence between repeat-pass station matching accuracy, type of registration warping function, and the ultimate quality of multitemporal image co-alignment (i.e., co-registration error). We are validating the model using existing and newly acquired airborne image sets collected by our team.

Develop and test automated image co-registration software

We have engaged a consultant to assist us in developing and testing a software tool for automated co-registration of repeat station image sets. This tool is facilitating validation of the analytical model described above and testing of robust change detection algorithms with numerous multitemporal images by enabling automated co-registration of a range of image pairs collected from matched imaging stations. It also enables us to quantify timeliness and reliability of automated repeat station image alignment.

Evaluate optimal design and associated time for wireless air-to-ground image transfer

The data generated and captured by airborne imaging sensors must be transmitted to the ground-based command and control station(s) in a reliable and timely fashion while meeting the required quality of service (QoS) demands of the user and application. We are investigating: (1) required compression and target detection so that only the features of interest in an image frame are transmitted with higher fidelity; (2) considering the priority of source data and their packets based on their roles in the mission; and (3) cross-layer wireless protocols (e.g., admission control, bandwidth allocation and resource optimization) for reliable transmission of images and other sensor data for meeting latency constraints.

Develop and test automated damage detection algorithms

SDSU is building off of existing experience, reviewing current literature, and communicating with colleagues to identify effective techniques for achieving reliable detection of change associated with infrastructure damage and minimizing detection of false change. False change may be associated with such things as moving shadows, moving objects (e.g., vehicles), vegetation phenology, etc. Therefore, we are utilizing methods such as Maximally Stable Extremal Regions to detect and remove shadows. Methods for shadow restoration are being evaluated for restoring image data in shaded regions. We are also creating shadow and illumination invariant, reflectance-only panchromatic images from red, green, and blue (RGB) color composites from color digital cameras.

Change detection steps include radiometric/brightness normalization between images, differencing of brightness, local texture and indices, and/or other image transformations, and a series of post-classification filtering steps to refine results (e.g., majority filtering to remove isolated changes, specifying minimum object size, etc.).

Estimate total time to information product delivery for a matrix of TSRSS model scenarios

The total time to information delivery is being estimated for a range of TSRSS and for the likely hazard event scenarios identified through interviews of hazard response and infrastructure managers. Time to image delivery is being combined with empirically derived estimates of processing, distribution, and dissemination times to produce a matrix of potential TSRSS capable of assessing infrastructure damage. The matrix is being provided to hazard response and infrastructure managers along with derived damage detection products during the second phase of interviews to assess their prioritization of reliability or timeliness of information when responding to extreme hazard events.

Evaluation of theoretical approaches and design paradigms for optimizing TSRSS

System design paradigms from computer science and engineering is being critically evaluated based on their potential to (1) enable the design of TSRSS based on user information requirements and (2) to enable the reliable characterization of user requirements even in cases where users are unable to readily supply those requirements. Potential to enable reliable characterization of user requirements is being assessed by comparing the paradigm prescribed methodology for obtaining user requirements to our experience trying to obtain requirements from infrastructure managers in and near San Diego County, CA and Bernalillo County, NM.

Return to Top

Outcomes

Book Chapters

Coulter, L., D. Stow, C.D. Lippitt and G. Fraley, 2015. Repeat Station Imaging for Rapid Airborne Change Detection, book chapter in Time Sensitive Remote Sensing, Springer-Verlag, 29-43.

Lippitt, C.D. and D. Stow, 2015. Remote Sensing Theory and Time-Sensitive Information Requirements, book chapter in Time Sensitive Remote Sensing, Springer-Verlag, pp. 1-9.

Stow, D., C.D. Lippitt, L. Coulter and B. Davis, 2015. Time Sensitive Remote Sensing Systems for Post-Hazard Damage Assessment, book chapter in Time Sensitive Remote Sensing, Springer-Verlag, pp. 13-28.

Journals or Juried Conference Papers

Coulter, L., M. Plummer, N. Zamora, D. Stow, and R. McCreight (). Assessment of Automated Multitemporal Image Registration Using Repeat Station Imaging Techniques. Photogrammetric Engineering & Remote Sensing. Status = SUBMITTED; Acknowledgment of Federal Support = Yes ; Peer Reviewed = Yes

Kerr, A. and D. Stow (). Optimizing Radiometric Fidelity to Enhance Aerial Image Change Detection Utilizing Digital Single Lens Reflex (DSLR) Cameras. Photogrammetric Engineering & Remote Sensing. Status = ACCEPTED; Acknowledgment of Federal Support = Yes ; Peer Reviewed = Yes

Loerch, A.C. and C.D. Lippitt (). Modelling the Timeliness of Airborne Remote Sensing Data. International Journal of Remote Sensing. Status = UNDER_REVIEW; Acknowledgment of Federal Support = Yes ; Peer Reviewed = Yes

Schweizer, E., D. Stow, and L. Coulter (). Automating Near real-time, Post-hazard Detection of Crack Damage to Critical Infrastructure. International Journal of Remote Sensing. Status = ACCEPTED; Acknowledgment of Federal Support = Yes ; Peer Reviewed = Yes

Storey, E., and D. Stow, D. (). Normalizing Shadows in Multi-temporal Aerial Frame imagery Using Relative Radiometric Adjustments to Support Near-real-time Change Detection. GIScience & Remote Sensing. Status = SUBMITTED; Acknowledgment of Federal Support = Yes ; Peer Reviewed = Yes

Storey, E., D. Stow, C. Chen, L. Coulter and S. Kumar (2017). Automated Detection and Restoration of Shadows in Aerial Images to Support Visual and Automated Change Detection in Urban Environments. GIScience & Remote Sensing. 54 (4), 453. Status = PUBLISHED; Acknowledgment of Federal Support = Yes ; Peer Reviewed = Yes ; DOI: http://dx.doi.org/10.1080/15481603.2017.1279729

Stow, D. L. Coulter, C. Lippitt, G. MacDonald, R. McCreight, and N. Zamora (2016). Evaluation of Geometric Elements of Repeat Station Imaging and Registration. Photogrammetric Engineering & Remote Sensing. 82 (10), 775. Status = PUBLISHED; Acknowledgment of Federal Support = Yes ; Peer Reviewed = Yes ; ISSN: ISSN: 0099-1112

Stow, D.A., C.D. Lippitt, L.L. Coulter and A.D. Loerch (2017). Towards an End-to-end Airborne Remote-Sensing System for Post-hazard Assessment of Damage to Hyper-critical Infrastructure: Research Progress and Needs. International Journal of Remote Sensing. 39 (5), 1441. Status = AWAITING_PUBLICATION; Acknowledgment of Federal Support = Yes ; Peer Reviewed = Yes ; DOI: https://doi.org/10.1080/01431161.2017.1407052

Other Conference Presentations / Papers

Loerch, A. and C.D. Lippitt (2016). Analyzing the current capacity of aerial survey firms for time-sensitive remote sensing using the Remote Sensing Communication Model. Association of American Geographers Annual Meeting. San Francisco. Status = OTHER; Acknowledgement of Federal Support = Yes

Bogus, S., C.D. Lippitt, andS. Zhang (2015). Development of a Remote Sensing Network for Time-sensitive Detection of Fine Scale Damage to Transportation Infrastructure. ASCE New Mexico and APA New Mexico Joint Fall Conference. Las Cruces, NM. Status = OTHER; Acknowledgement of Federal Support = Yes

Coulter, L.L., C.D. Lippitt, D. Stow, S. Walker, H.Lan, and R. McCreight (2015). Development of a Remote Sensing System for Rapid Post Hazard Assessment of Transportation Infrastructure. Annual Meeting of the American Society for Photogrammetry and Remote Sensing. Tampa, FL. Status = OTHER; Acknowledgement of Federal Support = Yes

Other Products

Software or Netware.

Aerial Data Acquisition Processing Transmission - Timeliness Estimator (ADAPTTE): A software tool for estimating the timeliness of acquisition, transmission, and processing of image data from various platforms. ADAPTEE is not distribution ready, but was developed for internal use by the project team.

Software or Netware.

SIFT and RANSAC Alignment (SARA): A software tool for automated co-registration of multitemporal airborne image frames collected with matched view geometry. Executable code available

RSI Image Registration Data Sets

SARA Software Input Data for Download and Testing

SARA Software Output Data for Download and Review

Thesis/Dissertations

Taylor, Tammira. An Index of Criticality for Transportation Infrastructure: A Vulnerability and Resilience Approach. (2017). University of New Mexico. Acknowledgement of Federal Support = Yes

Loerch, Andrew. Estimating the Timeliness of Remote Sensing Information Delivery. (2016). University of New Mexico. Acknowledgement of Federal Support = Yes

Kerr, Andrew. Optimizing Radiometric Fidelity to Enhance Aerial Image Change Detection Utilizing Digital Single Lens Reflex (DSLR) Cameras. (2017). San Diego State University. Acknowledgement of Federal Support = Yes

Schweizer, Eugene. Automating Near Real-Time, Post-Hazard Detection of Crack Damage to Critical Infrastructure. (2017). San Diego State University. Acknowledgement of Federal Support = Yes

Zhang, Su. Pavement Surface Distress Detection, Assessment, and Modeling Using Geospatial Techniques. (2016). University of New Mexico. Acknowledgement of Federal Support = Yes

Websites

Remote Sensing Network Optimization
University of New Mexico web page for this project

News Media

Homeland Security News Wire
San Diego Union Tribune
NBC San Diego News
San Diego 6 News
San Diego State University press release
Daily Aztec (SDSU campus newspaper)
UNM Project Web Page

Return to Top

Impacts

What is the impact on the development of the principal discipline(s) of the project?

The principle disciplines of the project are emergency management and remote sensing. The primary immediate impact on emergency management is the knowledge gained from surveys and interviews pertaining to which infrastructure features are truly critical to emergency responses and insights into how some types of damage are manifested. The future impact to the discipline of emergency management will be the implementation of repeat station imaging for post-hazard damage detection; the most likely implementation will be on small unmanned aerial systems, which still requires some research and development activities. The primary impact on remote sensing is moving very close to implementing a near-real-time image registration and change (damage) detection capability through the development of automatic image registration software that exploits repeat station imaging for pre- and post-event image comparisons.

What is the impact on other disciplines?

By moving closer to realizing near real-time, spatially comprehensive change detection, other disciplines such as geography, environmental management, criminal justice/law enforcement, wildlife biology, etc. benefit by being able to immediately gather information on land surface changes, and moving people, animals, and transportation vehicles.

What is the impact on the development of human resources?

At least 11 geography or engineering students participated in this project and developed skills pertaining to remote sensing, image processing, geographic information systems, and spatial analysis, specifically in the context of post-hazard damage assessment for critical infrastructure. Several of these recently graduated students are seeking careers that build on their interests in emergency management and geographic information science and technology.

What is the impact on physical resources that form infrastructure?

Research results from this project enable rapid inspection of the integrity and damage to physical infrastructure features following extreme hazard events. Upon implementation, this would enable emergency managers and structural engineers to make decision on whether to utilize, abandon or repair such physical infrastructure during the 24-hour period immediately following an extreme event.

What is the impact on institutional resources that form infrastructure?

Findings from our surveys and interviews provide information that will allow emergency managers to define what is truly critical infrastructure in the first 24 hours following a major hazard event. Our results should also encourage them to carefully and systematically determine what forms of damage are best indicators of compromised infrastructure.

What is the impact on information resources that form infrastructure?

The primary impact is the advancement of knowledge regarding how remote sensing derived information can be integrated into the response phase of hazard management cycle. Specifically, we found that doing so requires automated generation of information products, either through outsourcing or by an autonomous system, and that those products be distilled to a product that is readily integrated into existing common operating pictures.

What is the impact on technology transfer?

We provided guidance and hardware specifications for repeat station imaging technology to Civil Air Patrol. The Civil Air Patrol is currently the principal source of aerial imaging support for the Federal Emergency Management Agency during post- hazard disaster response situation. This included transferring information on: (1) how pilots can repetitively fly the same flight lines over time, (2) sources of low cost camera mounts for the Civil Air Patrol’s fleet of Cessna (high-wing) aircraft, and (3) sources of location-based triggering devices for digital frame cameras used by the Civil Air Patrol.

What is the impact on society beyond science and technology?

The most significant impact to society is the saving of lives and reduction of injuries resulting from the situational awareness from rapid image-based damage detection and avoidance of secondary hazards in the first 24 hours following an extreme hazard event. This is made possible from the TSRSS that we have designed, tested and documented as part of this project. In addition, we envision a future where RSI enables UAS to autonomously feed real-time image data and derived information products to the Internet of Things (IoT), which can have a wide range of impacts on society such as other real-time data sources already have (e.g., weather, traffic, etc.).

Return to Top