DW Spectrum® IPVMS Large Systems Deployment
-----------------------------------
Affected Roles: Administrator, Owner
Related Digital Watchdog VMS Apps: DW Spectrum® IPVMS
Complexity: Medium
Software Version: DW Spectrum® IPVMS v4.1
Last Edit: January 18, 2021
-----------------------------------
How do I design and implement a large system?
Large systems are comprised of:
- > = 20 servers and/or
- > = 50 TB footage storage per server and/or
- > = 500 Mbit/s Rx+Tx network interface streaming load
Important aspects we must consider are performance, reliability, and the configuration optimization of these VMS-based systems.
The picture above represents a logical diagram of a system. The surveillance system contains cameras, servers, operator terminals (using the IPVMS client), and a network that mediates the interaction between all of them.
The most important function of the System is to acquire video streams, store them, and provide access to the recorded footage by request.
“Discovery” is the process of automatically establishing communication with cameras and other devices that are new to the system. It reduces the necessity to manually add a new device to the existing system and undergo the configuration process using the DW Spectrum® Client application. Discovery is a continuous process, where devices already discovered by the Server have their online status refreshed then filtered out from the result.
Camera drivers are developed in order to maximize the amount of utilized features from a wide range of camera models. Each driver's purpose is to use the maximum of camera features, set them up, allow PTZ usage, etc.
HTTP/RTSP front-end allows the VMS server to interact with clients and other VMS servers and also broadcast video (either live or recorded footage).
The DW Spectrum® Server application is a cross-platform application that works inside the operating system’s environment. Taking this into account, we must also consider the impact of OS components on performance and reliability.
What are some potential performance bottlenecks?
Server-Server Network Connection
Each server in the system can be used to re-configure the entire system (such as adding users, changing recording schedules, and so on), so servers must be able to communicate with each other within the system.
Servers also communicate in order to monitor each other. If one server fails or was shut down intentionally, the other servers must take on the load of its cameras and continue recording.
Having a connection between servers is necessary to provide availability in complex network environments. If one server cannot establish a direct peer-to-peer connection, it may use other servers as a proxy.
Client-Server Network Connection
The DW Spectrum® Client application connects to the DW Spectrum® Server application in order to manage the System and allow users to watch playback or live video. The client subscribes to the stream as it is stored on the server’s drive whereas both the Web Client and Mobile Client require transcoding on the Server’s end.
Server-Cameras Network Connection
Generally speaking, there are two data streams between the server and a camera - management and video stream.
The management stream is used for configuring the stream and camera, while the video stream contains the picture itself. Both data streams are used to determine the availability of the camera.
The video stream is very susceptible to network losses and jitter. A subpar network leads to worse footage quality and fullness and excessive error messages.
Storage Throughput
Block devices are used to store the footage and footage index along with persistent data storage (internal database).
The load on the storage increases when
- A user requests to re-index the archive
- The daily index rebuild occurs
- The VMS client requests high-resolution footage recording for layout with many cameras.
CPU
It is unusual for the VMS to require extensive CPU usage, but some clients require the VMS server to prepare a stream for them. Transcoding is enabled for a stream that is either requested by the VMS Web Client, Mobile Client, or through the VMS Server API. A good rule of thumb is that two 1080p streams at 30 fps will load one CPU kernel.
Failover
“Failover” is a feature that allows healthy servers to take over cameras that used to belong to a server that has failed.
Network Considerations
It is necessary to have direct IP network connectivity between cameras of a failed server and at least one healthy server, otherwise failover is not viable. In complex network environments, it is necessary to set up a process of verifying the network reliability to confirm that in the case of a server failure, the network links towards other servers can withstand an excessive streaming load. The excessive load is created by the redistribution of streams that had been previously transmitted to the now failed server.
Server Capacity Considerations
Not only in regards to a network, but healthy servers can also become stressed if one of the servers in a System fails. This is because it creates a higher load on the storage and CPU due to the redistribution of client connections. Depending on the planned fault tolerance level, it is recommended to leave a capacity reserve on hardware resources of every server in a system.
Important Things to Remember
Large systems usually require more scrupulous capacity planning and system robustness verification. Taking this into account, the following points should be considered:
- Use robust network connections, especially between servers. The VMS still can work in case of connectivity failure, but that failure never can be considered as a normally functioning environment.
- Throughput of connections between all servers must slightly exceed the total bitrate of all the cameras of the most loaded server. This measure guarantees system can successfully bear failure of every single server.
- Pay special attention to completely avoid losses and jitter in the network between cameras and servers. Such losses influence the quality of footage drastically as in most cases lost streaming data cannot be recovered.
- If the customer uses mobile client or Video API calls, consider using better performing CPUs.
- Use monitoring for continuous evaluation of the network environment, hardware status, current load (CPU, storage and RAM) and application status.
- Have a failure recovery plans that contain a model of threats and appropriate actions to mitigate each.
