Futures trading runs in several sessions, including day and night sessions, and ops engineers handle tens of thousands of alerts each week. A futures company in Shanghai brought alert response onto one platform. Their approach has four parts:
- Choose an on-call platform that supports a trading calendar and parses email alerts.
- Staff a dedicated monitoring role and set up an on-call process.
- Connect the CMDB and use its metadata to add context to alerts.
- Measure each team's on-call work regularly with MTTA and MTTR.
This post is based on a talk by Qingyu Song, a technical expert at the company, at the second CCF Nightingale Observability Innovation Forum.
The company was among the first in China to qualify for full financial futures clearing and holds an AA rating from the China Futures Association. It offers commodity futures brokerage, financial futures brokerage, and asset management. It has more than 20 branches and over 300 securities IB offices nationwide, and several of its business metrics have ranked in the industry's first tier for many years.
Three challenges in futures operations
Song described three traits of operations in the futures industry:
- The business spans several exchanges and trading sessions, including day and night sessions. Ops covers every session, so engineers provide around-the-clock on-call coverage.
- Futures trading is continuous, real-time, and high-risk, with heavy peak loads. Information systems must be highly secure and stable.
- Ops engineers handle tens of thousands of alerts a week on average. At that volume, people grow less sensitive to alerts. The company has had production incidents caused by missed critical alerts.
So the team needed one alert response platform to carry its management process, technical requirements, and operating practices. The goals were faster alert response and less pressure on ops engineers.
Choose a platform: Scheduling, trading calendar, sources, noise reduction, and metadata
The team evaluated five capabilities:
- Flexible scheduling and alert escalation.
- A trading calendar that fits the rhythm of the financial industry.
- Integration with the company's alert sources.
- Unified noise reduction and suppression.
- Integration with CMDB and other metadata centers.
After research and testing, the team chose Flashduty On-call as its on-call platform.
When the team connected existing monitoring, Flashduty's email integration and label enrichment solved a hard problem. Network analysis platforms (Tiandan and Colasoft), OceanBase, TdSQL, Qfusion, and SmartX only send alerts by email. They're hard to connect to other tools through webhooks, and their alerts can't go straight to IM.
Flashduty parses the subject and body of alert emails. It extracts alert details, thresholds, and status updates into structured data, then notifies ops engineers through different channels.
Build the process: A dedicated monitoring role and clear on-call rules
At the company level, a dedicated team fills the EEC monitoring role. It builds and improves the on-call process, retains know-how, and follows up on open issues from on-call shifts.
On-call rules that fit the business reduce uncertainty during response. Engineers know what to look at, whom to contact, and how to escalate.
Unify metadata: Enrich alert labels from the CMDB
The CMDB already stores relationships between resources, and between resources and people. When the on-call platform reuses this data, alert dispatch becomes more automated. There's less duplicate configuration and less risk of inconsistent metadata.
Flashduty connects to the CMDB, pulls asset dependency mappings, and uses that metadata to enrich alert labels. The enriched labels drive automatic dispatch. When engineers get an alert, they also see fuller context and can judge impact and severity faster.
Operate continuously: Measure on-call work with data
Once all alerts are on one platform, every step of the handling lifecycle is recorded. Flashduty provides workload statistics, top-K alert statistics, mean time to acknowledge (MTTA), and mean time to resolve (MTTR). Managers use them to decide which rule, team, or process to improve.
To talk with Qingyu Song in depth, add him on WeChat: giggs06. For more about Flashduty On-call, see the product page.
