The Central Financial Work Conference held in 2023 proposed five major articles on the high-quality development of digital finance, including technology finance, green finance, inclusive finance, pension finance, and digital finance; The 14th Five Year Plan for National Economic and Social Development of the People's Republic of China and the Long Range Objectives Through the Year 2035 clearly state the need to promote the security and controllability of core technologies in financial informationization and maintain the security of financial infrastructure; In the Financial Science and Technology Development Plan (2022-2025), the People's Bank of China requires financial institutions to strengthen the application of core technologies. The adoption of new thinking, models, and technologies to continuously evolve the core business system of banks is the key to building digital finance and promoting high-quality financial development. It is also the foundation for implementing the requirements of financial technology core technology research and maintaining national financial security.
Agricultural Bank of China is deeply promoting digital transformation, solidly doing a good job in digital finance, and strengthening core technology research and development in accordance with the twelve character policy of "stability, efficiency, security, precision, agility, and simplicity". We have successfully explored a solution of cloud native distributed architecture+core system innovation technology stack migration and its supporting engineering implementation methodology, which helps Agricultural Bank of China's digital construction and secure and stable continuation of financial business. The Agricultural Bank of China's distributed core system serves over 800 million individual customers and 1.8 billion individual accounts, with a peak daily transaction volume of nearly 2 billion transactions. The availability rate during major business hours reaches 99.99%, providing a demonstrative reference for the construction of distributed systems in commercial banks in China.
Current situation and challenges
With the rapid development of digital finance, the user and data volume of information systems are growing rapidly, and the diversity of scenarios is increasing, which puts higher demands on system stability, scalability, and flexibility. Traditional centralized architectures are gradually unable to meet the needs. Distributed architecture and cloud native technology provide a new foundation for the transformation, development, and agile innovation of banks, with significant advantages in scalability, flexibility, and scalability.
As a systemically important bank, the biggest challenge faced by Agricultural Bank of China in the process of transforming its core system to a distributed architecture is how to maintain stability. This challenge includes both the application of new architectures and technology stacks, as well as the industry wide issues that continue to support business innovation and development, and the unique challenges brought by Agricultural Bank of China due to its large scale and full range of business types.
'New and Stable': How to select, verify, and promote the maturity and improvement of technological foundations? How to solve common industry challenges such as transaction consistency and system high availability in a distributed architecture?
'Sensitive and Stable': How can the core system ensure continuous business innovation and development during the architecture transformation process? How to better support business agile innovation needs after construction?
'Large and Stable': How to ensure the smooth transformation of the core system architecture with large volume, high throughput, and continuous operation 24/7?
Practical ideas and goals
Agricultural Bank of China is fully committed to promoting the implementation of the distributed core construction project, completing the reconstruction of large-scale bank core systems, and successfully exploring a new generation of distributed core system architecture that is innovative in technology, agile in business, and stable and efficient. Through application, it aims to promote technological maturity and achieve the following three goals.
One is to deepen the application of technological innovation and build a new generation of core architecture. Based on a new technology stack, the system's horizontal scaling and disaster recovery capabilities are comprehensively improved to achieve high-performance, high reliability, and multi active disaster recovery, effectively supporting the core system's daily peak transaction volume of nearly 2 billion transactions with high throughput.
The second is to strengthen technological empowerment capabilities and assist in agile business innovation. Comprehensively enhance the scalability of the system, support rapid business innovation, empower business transformation and upgrading with technological upgrades, and provide customers with more convenient and efficient financial services.
The third is to efficiently prevent implementation risks and ensure smooth migration and operation of the system. On the one hand, a scientific implementation path was proposed to smoothly complete the system architecture migration; On the other hand, an efficient and precise operational support system has been established to ensure the continuity of system business.
Highlights and Achievements of Technological Innovation
1. Stable base
One is the implementation of product standards in the construction of distributed core systems, and the establishment of a unified accounting and clearing system. Based on enterprise level architecture standards and methods, abstract standardized business components covering products, contracts, accounting, customers, operations, etc. The product application facilitates the construction of new products through assembly in the product factory, achieving the effect of rapid and agile product innovation.
The second is to establish a service-oriented technology base that organically collaborates with infrastructure and technology platforms, achieving efficient supply of resources on demand and unified support for service-oriented technology capabilities in all aspects, shielding the complexity of distributed architecture development.
In terms of infrastructure, a cloud platform with "one cloud and multiple cores" has been built to achieve application centric resource self-service supply, on-demand delivery of computing power, dynamic expansion, and continuous improvement of resource utilization. Adopting a distributed database, deploying multiple replicas for high availability, horizontal high scalability, and meeting the full chain analysis characteristics of the database, supporting the stable operation of the core system. And overcome the high threshold and difficult stability verification of distributed database technology, build a one-stop developer center and specialized chaos tools, fill the gap in distributed database application platforms, and give back to the industrial ecosystem.
At the same time, in terms of technical capabilities, we have developed a full stack technology platform that provides key technical components such as distributed transaction coordinators and distributed interconnectivity, covering various technical capabilities in various fields of distributed technology. Build and operate a system that supports distributed core high throughput and large-scale batch scheduling.
The third is the precipitation of standardized research and development processes, which comprehensively improves the quality and efficiency of research and development under distributed architecture. Taking enterprise level digital production lines as the core, we aim to improve quality control by solidifying standards into automated quality access control; Build APM tools to continuously optimize system performance through full link performance tuning; Introducing chaos engineering and improving system stability through practical fault injection.
2. Implement stability
One is to propose a controllable core system verification mode to ensure the correctness of key design and implementation paths, as well as the ability to roll back critical business at any time, smoothly achieving large-scale single application transformation. Adopting a real traffic system verification method, the infrastructure availability, key design feasibility, application version maturity, applicability of performance indicators, and operability of disaster recovery switching are verified through real traffic, data, and environment, forming a controllable migration path.
The second is to form a multi-dimensional seamless traffic switching path. By adopting a data migration model that supports different granularities such as accounts, customers, and institutions, as well as multiple batches, data migration risks are controlled, migration time is reduced, and business continuity during the switching process is maximally guaranteed. For personal businesses with the largest impact range, real-time online migration is achieved for each customer, achieving a "seamless switching" with "zero impact" on customer service.
3. Stable operation
The construction of a distributed core system is the first step of the Long March, and long-term stable operation is a greater test.
Firstly, based on the characteristics of distributed system architecture, a full chain monitoring system and indicators have been established, creating a closed-loop operation and maintenance process from monitoring alarms to analysis and review, realizing the five in one of "monitoring, management, control, allocation, and analysis", and forming a perceptible, manageable, controllable, and measurable operation guarantee solution. And summarized the most commonly used and effective emergency operation methods, combined with continuously improving automation scheduling capabilities and regular practical exercises, to ensure the continuous improvement of emergency level.
Secondly, through the innovative application of AIOps (Intelligent Operations and Maintenance) and SRE (Site Reliability Engineering), intelligent and engineered operational support capabilities have been achieved. On the one hand, fully utilizing big data and machine learning to enhance the level of intelligent operation and maintenance, achieving automatic positioning of fault locations and intelligent matching of emergency plans. On the other hand, we will carry out organizational innovation, establish cross disciplinary SRE teams, and strengthen cross disciplinary collaboration.
The third is to improve the solution to distributed consistency problems and form a one-stop distributed transaction error handling system. By innovatively constructing a distributed transaction framework, we can achieve deep matching between different distributed transaction solution modes and diverse scenarios of banking business; Establishing an automated error handling system based on the collaborative model of industry and technology, comprehensively ensuring the accuracy of accounting processing, and effectively enhancing risk control and automated disposal capabilities.
Summary
The successful completion of the construction of the distributed core system of Agricultural Bank of China, as well as the smooth migration of the original core system's full functions, has injected new momentum into its digital transformation, providing a solid foundation for accelerating the formation of a new smart banking model led by technology, empowered by data, and operated digitally. It comprehensively assists Agricultural Bank of China in becoming a leading bank serving rural revitalization and a main bank serving the real economy.
At the same time, the secure and stable operation of the Agricultural Bank of China's distributed core system provides important reference for the transformation of China's commercial bank core system architecture, demonstrating the driving role and demonstration effect of large financial institutions, providing impetus for enhancing information technology innovation applications, improving industrial ecosystems, and contributing to accelerating the construction of "digital finance".
(This article was published in the first half of January 2025 issue of Financial Electronics)