## Abstract

In this study, an intelligent video surveillance (IVS) system is designed based on the particle filter. The designed IVS system can gather the information of the number of persons in the area and hot spots of the area. At first, the Gaussian mixture background model is utilized to detect moving objects by background subtraction. The moving object appearing in the margin of the video frame is considered as a new person. Then, a new particle filter is assigned to track the new person when it is detected. A particle filter is canceled when the corresponding tracked person leaves the video frame. Moreover, the Kalman filter is utilized to estimate the position of the person when the person is occluded. Information of the number of persons in the area and hot spots is gathered by tracking persons in the video frame. Finally, a user interface is designed to feedback the gathered information to users of the IVS system. By applying the proposed IVS system, the load of security guards can be reduced. Moreover, by hot spot analysis, the business operator can understand customer habits to plan the traffic flow and adjust the product placement for improving customer experience.

### Keywords

- intelligent video surveillance (IVS)
- Gaussian mixture model
- particle filter
- Kalman filter

## 1. Introduction

Video surveillance systems are often utilized at some specific places such as exits, entrances, parking lots, convenient stores, etc. for management. Traditionally, security guards watch screens of monitored places for surveillance. However, according to statistics, one security guard can only watch four monitors at the same time, and the concentration can last only for 10 minutes such that more than 50% of key information is lost. Moreover, it is not economically worthwhile to hire multiple security guards to monitor one video surveillance system. Therefore, intelligent video surveillance (IVS) systems have become more and more important in commercial sector and have attracted a lot of attention in research area as well [1, 2, 3].

IVS system can be defined as the real-time monitoring of both persistent and transient objects within a specific environment. IVS is also referred to as video analytics (VA) which involves the use of software to automatically detect the objects of specific interest and analyze their behaviors. For finding the objects of interest, it is usually done by detecting the movements or changes in the image that can be achieved by background subtraction technique. To do background subtraction, an effective way to build up the background is the Gaussian mixture model [4, 5]. After the object of interest is detected, the goal is to analyze their behavior that sometimes can be done by tracking them. Usually, particle filters and Kalman filters are employed for the purpose of tracking objects in IVS systems [6, 7, 8, 9, 10].

IVS systems have been applied for different kinds of purposes. Ref. [11] presents an architecture for a perimeter security system dedicated to critical transport infrastructure protection. Ref. [12] addresses a framework for event decision of vision-based intelligent surveillance system based on the fuzzy model. A shape-perceived algorithm using the building block-based matching method is presented in [13] for object tracking of intelligent surveillance applications. A way of unification of flame and smoke detection algorithms by merging the common steps into a single processing flow is proposed in [14] for IVS systems.

The aim of this study is to design a IVS system based on the particle filter. The designed IVS system can gather the information of the number of persons being in the area, the number of persons having been in the area, and hot spots (places of more than usual interest, activity, or popularity) of the area. The Gaussian mixture background model is utilized to detect moving objects by background subtraction in the designed IVS system. The moving object appearing in the margin of the video frame is considered as a new unit (person). When a new person is detected, a new particle filter is established and assigned to track the new person. For saving the computational load, the particle filter is terminated when the corresponding tracked person leaves the video frame. Moreover, the Kalman filters is utilized to estimate the position of the person when the person is occluded. Information of the number of persons in the area (having been in the area) and hot spots is gathered by tracking persons in the video frame. Finally, a user interface is designed to feedback the gathered information to users of the IVS system. By applying the proposed IVS system, the load of security guards can be reduced. Moreover, by hot spot analysis, the business operator can understand customer habits to plan the traffic flow and adjust the product placement for improving customer experience.

## 2. Preliminary

In this section, three well-known techniques, i.e. adaptive Gaussian mixture model, particle filter, and Kalman filter, are presented for constructing the IVS system.

### 2.1. Adaptive Gaussian mixture model

The recent history of each pixel, * k*Gaussian distributions. The probability of observing the current pixel value is given as Eq. [4]:

where * k*is the number of distributions,

*th Gaussian distribution in the mixture at time*i

*,*t

*th Gaussian distribution in the mixture at time*i

*,*t

*th Gaussian distribution in the mixture at time*i

*, and*t

The updating rules for the parameters of the adaptive Gaussian mixture model can be found in [5]. After the Gaussian mixture model is established, the foreground pixels (representing the moving objects) can be obtained by applying the Mahalanobis distance:

The adaptive Gaussian mixture model has been applied for different kinds of applications, such as automatic speech emotion recognition [15], tracking targets on long-range radar systems [16], fast sampling-based motion planning [17], etc.

### 2.2. Particle filter

The key idea of particle filtering is to approximate the probability distribution by a weighted sample set [18]:

Each sample consists of an element * N*samples are drawn with replacement. The mean state of the object is estimated at each time step by

Particle filter provides a robust tracking framework.

The particle filter has been successfully applied to many applications. An algorithm to track the vehicle with the adaptively changed scale based on particle filter is propose in [19]. The vehicle guidance with control action computed by a Rao-Blackwellized particle filter is proposed in [20]. The localization of indoor robot based on particle filter with EKF proposal distribution is proposed in [21].

### 2.3. Kalman filter

The Kalman filter [22] addresses the general problem of trying to estimate the state of a discrete-time controlled process that is governed by the linear stochastic difference equation:

with measurement equation

where

where * t*, and

and the a posteriori estimate error covariance is

The equations for the Kalman filter fall into two groups: time update (predictor) equations and measurement update (corrector) equations. The time update equations are given as

The measurement update equations are given as

Figure 1 shows the operation architecture of Kalman filter.

The Kalman filter has been widely applied to time series analysis and statistical modeling problems. This study [23] improves the navigation performance, when refraction starlight is used to compute the position and velocity of a satellite in unscented Kalman filter. An anti-spoofing algorithm based on adaptive Kalman filter for high dynamic positioning in global positioning system is proposed in [24]. In this work [25], the robust Kalman filter is applied to the people occupancy estimation problem, and an iterative algorithm is developed to handle the state-dependent model uncertainties.

## 3. IVS system design

The IVS system design is presented in this section. Figure 2 shows the operation process of the IVS system. Firstly, the adaptive Gaussian mixture model is applied for constructing the background model to detect moving objects in the video image frame. The moving object appearing in the margin of the video image frame is considered as a new unit (person), and then a new particle filter is established and designated to track the new person. Moreover, the Kalman filter is utilized to correct the position obtaining by the particle filter and to estimate the position during occlusion. After that, the information of the number of persons in the area, the number of persons having been in the area, and hot spots are obtained by analyzing the tracking paths. Finally, the information is fed back to the user by the user interface. Each block of the IVS system design shown in Figure 2 will be illustrated in the following subsections.

### 3.1. Gaussian mixture model for detecting new units

By applying the adaptive Gaussian mixture model described in Subsection 2.1, the moving objects can be detected by using the Mahalanobis distance of Eq. (3). Figure 3 illustrates the foreground pixels representing the moving objects obtained by the adaptive Gaussian mixture model. Here, we assume that a new unit (person) will appear only from the border of the monitored place. Therefore, for a moving object detected in the margin of the monitoring video frame, we need to determine that it is a new person or not.

Figure 4 shows the checking process for determining the object detected in the margin of the monitoring video frame as a new person or not. At first we need to check the size and ratio of the detected object to identify that the detected object is a person or not. If the size and ratio of the detected object are identified as a person, then we have to check that the detected person is new or not. In the case that there is no tracked person in the video frame, the detected person in the margin of the video frame is determined to be a new person. In the case that there has (have) been tracked person(s) in the video frame, we need to calculate the distance(s) between the detected person and tracked person(s) to check that the detected person is new or not. If the distance(s) is (are all) longer than a predefined threshold * T*, the detected person is considered as a new person. If some distances are shorter than

_{d}

*, we need to apply Eq. (18), which will be described in the following subsection, to calculate the similarities of color distribution between the detected object and the tracked units with distance shorter than*T

_{d}

*. If all similarities between the detected object and the tracked units with distance shorter than*T

_{d}

*are lower than a predefined threshold*T

_{d}

*, the detected object is determined as a new person.*T

_{s}

It is noted that there are several people detection methods [26, 27]. However, in the designed IVS system, we assume that the only moving objects are persons. Hence, we choose a simple method, which is adaptive Gaussian mixture model, to detect that the persons appear from the border of the monitored place for reducing the computational load.

### 3.2. Particle filter for tracking units

For a new detected person, a new particle filter is established and designated to track the new person. In the design of the particle filter, the target model of target region (the detected person) is the color distribution which is represented by histograms calculated in the HS (Hue, Saturation) space using 8 × 8 bins. A popular measure between two color distributions is the Bhattacharyya coefficient. Considering discrete densities such as two color histograms

the Bhattacharyya coefficient is defined as

The larger

where * x*and

*represent the center location of the rectangle and*y

*and*H

_{x}

*are the width and length of the rectangle, respectively, as shown in Figure 5. The sample set is propagated through the application of a dynamic model:*H

_{y}

where is an 4×4 identity matrix and

The tracking result can be calculated by Eq. (5). During filtering, samples with a high weight may be chosen several times, leading to identical copies, while others with relatively low weights may not be chosen at all. Figure 6 illustrates persons tracked by particle filter.

Although the particle filter is a robust method for tracking objects, it cannot deal with some special cases. Since we use color distribution for the target model of the particle filter, it may lose tracking when the color of the background is similar to the color of the tracking object. Moreover, if the tracking object is occluded, still the particle filter will lose tracking.

### 3.3. Kalman filter for correcting and estimating positions

In the IVS system design, the Kalman filter is utilized to correct the position obtained by the particle filter and to estimate the position during occlusion. Here, the uniform linear movement is considered. Hence, the linear stochastic difference equation is given as

and the measurement equation is given as

where

For improving the tracking results of particle filter, after propagation by Eq. (20), the estimating speeds

When the tracked object is occluded, the Kalman filter is applied to estimate the position of the occluded object. Therefore, for the case that all the similarities between samples of particle filter and the target are lower than lower than a predefined threshold * T*, the Kalman filter doesn’t use the measurement correction and only takes the filter prediction as object position. Moreover, all samples of the particle filter are uniformly distributed around the estimating position such that the particle filter can retrieve tracking after the object recovering from occluded. However, if the occluded object is stayed in the back of obstacle without moving, the Kalman filter will still lose the tracking.

_{p}

### 3.4. User interface for feeding back information

Finally, by analyzing the tracking paths, the information of the tracking result, the number of persons in the area, the number of persons having been in the area, and hot spots are obtained and then fed back to the user through the user interface as shown in Figure 7. Furthermore, several parameters can be adjusted through the user interface for adapting different environments.

## 4. Experiment results

The experiment is done in the San Shia Campus of the National Taipei University, Taipei, Taiwan. Figure 8 illustrates the operation process of the IVS system. In Figure 8, the up-left frame is the original image; the up-right frame is the background subtraction binary image applying Gaussian mixture model; the left-down frame shows the tracking result by applying particle filter (green rectangle) and the result corrected by Kalman filter (white rectangle), and the right-down frame is the final tracking result. Figure 9 makes a comparison to show the utility of adding the estimating speeds of the Kalman filter to the position of each sample of the particle filter, i.e., to show the utility of Eq. (24). It can be seen that a fast-moving bicycle can be tracked by the particle filter with Eq. (24) as shown in Figure 9(a). However, the fast-moving bicycle cannot be tracked by the particle filter without Eq. (24) as shown in Figure 9(b). Figure 10 makes a comparison to show the utility of applying Kalman filter to estimate the position of the occluded object. It can be seen that the particle filter can retrieve tracking after occlusion with applying Kalman filter to estimate the position of the occluded object as shown in Figure 10(a). However, it can be seen that the particle filter cannot retrieve tracking after occlusion without applying Kalman filter to estimate the position of the occluded object as shown in Figure 10(b). Moreover, it can also be seen in Figure 10(a) that all samples of the particle filter are uniformly distributed around the estimating position such that the particle filter can retrieve tracking after the object recovering from occluded.

## 5. Conclusion

Based on the particle filter, an IVS system has been designed in this study. Utilizing the Gaussian mixture background model, the moving objects appearing in the margin of the video frame can be detected and considered as a new person. Then, a new particle filter is established and designated to track the new considered person. Moreover, the Kalman filter is applied to correct the tracking result and estimate the position when the tracked person is occluded. By analyzing the tracking paths, the information of the number of persons in the area, the number of persons having been in the area, and hot spots can be obtained. Finally, the information is fed back to the user through the user interface.