Novel Matrix Completion Methods for Missing Data Recovery in Urban Systems

Urban systems are composed of multiple interdependent subsystems, for example, the traffic monitoring system, the air quality quality monitoring system or the electricity distribution system. Each subsystem generates a different dataset that is used for analyzing, monitoring, and controlling that particular component and the overall system. The quantity and quality of the data is essential for the successful applicability of the monitoring and control strategies. In practical settings, the sensing and communication infrastructures are prone to introduce telemetry errors that generate incomplete datasets. In this context, recovering missing data is
paramount for ensuring the resilience of the urban system.
First, the missing data from each subsystem is recovered using only the available observations from that subsystem. The fundamental limits of the missing data recovery problem are characterized by defining the optimal performance theoretically attainable by any estimator. The performance of a standard matrix-completion based algorithm and the linear minimum mean squared error estimator are benchmarked using real data from a low voltage distribution system. The comparison with the fundamental limit highlights the drawbacks of both methods in a practical setting. A new recovery method that combines the optimality of the Bayesian estimation with the matrix completion-based recovery is proposed. Numerical simulations show that the novel approach provides robust performance in realistic scenarios.
Secondly, the correlation that results from the interdependence between the subsystems of an urban system is exploited in a joint recovery setting. To that end, the available observations from two interconnected components of an urban system are aggregated into a data matrix that is used for the recovery process. The fundamental limits of the joint recovery of two datasets are theoretically derived. In addition, when the locations of the missing entries from each dataset are uniformly distributed, theoretical bounds for the probability of recovery are established and used to minimize the acquisition cost. The numerical analysis shows that the proposed algorithm outperforms the standard matrix completion-based recovery in various
noise and sampling regimes within the joint recovery setting.