Key Technology Application of Security Cloud Storage in AI Big Data Era

For the video cloud storage technology, the mainstream security manufacturers in the industry can be said to have done a good job. Security manufacturers combine the underlying technology of cloud storage with security-specific streaming media to form a security-independent cloud direct storage product. No need to pull the external device, the storage can directly receive the data transmitted from the front end. But nowadays, the security industry has undergone earth-shaking changes. The video stream is completely unable to represent the data characteristics of the security industry. The AI ​​era is coming soon.

At present, the AI ​​data content in the industry mainly includes face data and structured data, including motor vehicles, non-motor vehicles, and pedestrians. The data type includes a series of unstructured data such as pictures, snapshot records, alarm records, and image attribute information. This type of data is characterized by fragmentation and is different from the video stream data type. The video stream guarantees continuous writing and the file packing size is relatively uniform. However, fragmented files are unknown due to their size and number. Scattered writes consume a lot of CPU and hard disk resources. For the CPU, you need to handle many threads at the same time. For hard drives, the head requires constant lane change addressing, which greatly reduces the life of the hard drive.

For this particular type of data, traditional streaming services cannot be processed. At present, mainstream security vendors have specially developed software for pulling such data streams, and the storage functions can be realized by installing them in general storage hardware. Because it is an emerging market, the storage of a single device can be satisfied in most scenarios. However, with the popularity of AI, the amount of data will continue to increase. For a city, in order to grasp the traffic conditions in the city, it needs to be collected. The number of vehicles, the congestion information, and the traffic flow direction of each road and every intersection. Through the data after the algorithm, the operation status of the urban traffic can be simulated, so as to predict the trend of the next second, and make an early warning plan to realize the real era of big data. When the data scale is expanded to a certain extent, the underlying cloud storage mechanism will be the technical support that people have to consider. However, the problem arises. Traditional security cloud storage only has access to video and cannot actively acquire structured data. Therefore, in the short term, this AI data cloud storage is bound to become the mainstream of the storage application layer.

Although the integrated AI data cloud storage can be realized through the docking of the application layer and the underlying layer, how will cloud storage respond when data types are further evolved and new data structures emerge? It is not a long-term solution to do compatible development blindly, and it will waste manpower and resources. To make matters worse, if there are multiple data types in one site, it is necessary to deploy multiple sets of cloud storage to store different data, which is a great waste of storage space and costs a lot of money. The feasibility is extremely low.

For the business characteristics of the security industry, the following two major technical directions for cloud storage require key breakthroughs:

The first is efficient metadata organization and framework construction to solve the problem of large-scale cluster management and massive files.

The number of nodes that need to be managed in the entire distributed system is hundreds or thousands. A real file of the user will be distributed on multiple nodes, and multiple nodes are responsible for carrying the write of real data. When reading, it is necessary to request the data location information through the metadata management server to initiate reading. The performance of the metadata request is a step-by-step recursion or a single access to complete the operation, is a key factor to measure the performance of the entire system.

For a single large file, whether it can fully exert the read and write performance involves the problem of split granularity. As the core of the metadata service, it is required to support high-speed concurrent processing in thousands of nodes and tens of thousands of clients. This needs to be considered in the basic protocol framework and signaling interaction model, and serialized by ultra-high protocols. De-serialization performance, scalable protocol design, network framework model, and task processing model are all layered up and down, and are processed efficiently in each link. A reasonable number of organizational structures can use the bucket-based method of type object storage, so that the data hash is distributed, and the file is simple and efficient to manage. For the data in the bucket, it is not necessary to adopt a traditional directory tree to perform step-by-step traversal, only once. Positioning can complete the operation.

For the data block organization management of files, on the one hand, it is necessary to control the better granularity to achieve IO. The advantages of multi-node and multi-disk can be fully utilized. On the other hand, it is necessary to reduce the management pressure of metadata and improve the number of clusters and the number of files managed. The user's data block exists on the storage node and is divided into segments in each disk. The system runs for a long time or restarts, powers down, byte jumps, etc., and needs to be able to perform data blocks in the node management and data blocks in the metadata. Compare and find out the difference item to complete the correction, and trigger the recovery of the damaged data early. This requires the metadata to be reasonably organized, and can quickly find the metadata information of the corresponding node, and does not affect other metadata in real time during the comparison process. Access and add.

The second is a clear read-write model that provides business usage semantics and addresses different write and read requirements for video and images.

Common read and write does not provide an interface, and requires explicit read and write semantics. For example, the file system provides file operation semantics, according to the open/write/read/close mode, and supports the semantics of seek and modification and appendment; the S3 interface provides the putObject/getObject interface, which can be seen after uploading once. Semantics; HDFS provides operational semantics similar to file systems, but does not support modification.

For video, it should follow the semantics of the file but does not need to support append and modify. It only needs to support streaming write, and supports reading while reading, to avoid the business layer needs to open a large cache or cache the video file locally to upload. The same is true for the image writing method, and file stream mode writing should also be supported. Although it seems that the picture can be written at one time, but the current picture can be 1MB or larger, and only by setting the cache size to complete a picture write of the application, there will be memory usage in the cloud storage client. Too large or not smooth enough to write will have a meal effect and cause the cache to be full. On the read, there is no need to read the data in one picture, but the whole write is completed immediately.

From the perspective of the file name, since each picture corresponds to a front-end capture record, the picture address can be stored along with the structured record, and the user does not need to be related to the picture address generation mode, which means that the picture address can be returned by the system. Generate. For the video file formed after the video stream is stored, the user can record the file name by using the specified file name capability provided by the cloud storage, and generate the file name according to the customized business logic, and then query according to the rules. Complete the recording list or specify the playback of the recording file.

In addition, as AI is deployed in the security field, heterogeneous cloud storage will independently deploy and deploy the storage application layer and file management layer and resource allocation layer. In this way, vendors that do cloud storage infrastructure and hardware can concentrate on ensuring the stability of the storage mechanism. Sex, application vendors can concentrate on the compatibility of different data types. As long as the underlying standardization is done, major security and storage vendors can form a stable ecological cooperation. One party provides physical resources, and one party provides upper-layer services, which is no longer limited to the product model of software and hardware. On this basis, some manufacturers that are limited by capital investment can even develop their own cloud services. The upper application software can even be stored in the cloud as a common resource for end users to develop their own professional storage services.