At the second CCF Nightingale Observability Innovation Forum, Qingyu Song, a technical expert from a futures company in Shanghai, shared practical experience implementing OnCall in the futures industry.
This futures company was among the first in China to obtain full financial futures clearing business qualification and has received an AA rating in the classification by the China Futures Association. It provides comprehensive financial services, including commodity futures brokerage, financial futures brokerage, and asset management. The company has more than 20 branches and over 300 securities IB offices nationwide, and its business network covers the country. Its operating revenue, net profit, customer count, total customer equity, market share, and other business indicators have ranked in the first tier of the futures industry for many years.
In his talk, Qingyu Song noted that operations assurance in the futures industry has several special characteristics and challenges:
- Futures businesses follow the schedules of different exchanges and involve multiple trading sessions, including day and night sessions. Operations teams participate throughout the process, so engineers need to provide around-the-clock duty coverage.
- The continuity, uniqueness, real-time nature, high risk, and peak-load pressure of futures trading impose extremely high requirements on the security and stability of futures information systems.
- Operations engineers handle tens of thousands of alert notifications on average each week. At that scale, alert sensitivity can decline, important alerts can be missed, and historically, missed critical alerts have caused production incidents.
Therefore, the team needed one place to carry the management mechanisms, technical requirements, and operating mechanisms behind on-call work. The goal was concrete: faster alert response, less pressure on operations engineers, and more reliable systems.
Implementation Approach
- Build the platform
- Choose a mature platform that follows OnCall best practices and supports unified alert ingestion, schedule management, alert escalation, and alert noise reduction for the company's operations scenarios.
- Build the process
- At the company level, establish relevant systems and roles, and staff a dedicated team, the EEC monitoring role, to build and improve the OnCall process, retain capabilities, and follow up on remaining OnCall issues.
- Unify metadata
- Integrate with internal metadata systems such as CMDB and reuse existing metadata, lowering platform construction cost and improving automation and intelligence.
- Operate continuously
- Quantify each team's operations OnCall work on a regular basis using metrics such as MTTA and MTTR, continuously govern alerts, and improve OnCall efficiency.
Build the Platform
When selecting a platform, the team focused on five factors: flexible scheduling and alert escalation, a trading calendar for financial-industry operations, integration with the company's alert data sources, alert noise reduction and suppression, and metadata integration with CMDB and other metadata centers. After research and testing, the team selected Flashduty as its OnCall platform.
When integrating existing monitoring events, Flashduty's combination of email integration and label enrichment was especially helpful. Network analysis platforms such as Tiandan and Colasoft, as well as OceanBase, TdSQL, Qfusion, and SmartX, only provided email-based alerting. They were hard to connect to third parties through webhooks, making IM-based alerts difficult. Flashduty can parse incoming email content, extract key information from email titles and bodies, such as alert details, thresholds, and system status updates, convert that information into structured data, and then use Flashduty's notification capabilities to reach operations engineers through different channels.
Build the Mechanism
Establishing mechanisms that fit the business scenario can effectively remove uncertainty from the OnCall process, improve engineers' sense of safety, and increase efficiency.
Unify Metadata
Take CMDB as an example. CMDB already stores relationships between resources, as well as relationships between resources and people. If an OnCall platform can directly use this information, alert dispatch becomes more automated, duplicate configuration work is reduced, and the risks caused by inconsistent metadata are lowered.
Flashduty supports CMDB integration. It can obtain asset dependency mappings from CMDB and use that metadata for alert label enrichment. The enriched labels make automated dispatch easier, and engineers see enough context in the alert to judge impact scope and severity faster.
Continuous Operations
"Without measurement, there is no improvement." With alerts centralized in one platform, every step in the alert handling lifecycle is recorded. Flashduty provides workload statistics, TopK alert statistics, MTTA, and MTTR, giving managers a concrete basis for OnCall optimization.
Read https://www.flashduty.com/en/products/on-call for more on Flashduty On-call.
Mr. Song's WeChat: giggs06. If you would like an in-depth discussion with him, you can add him on WeChat directly.
