Ctrip.The meaning of multilingual power function.Here: I will explain to you.Ctrip system shark

Original: Li Sheng

1. Introduction

Shark

This is a set of multilingual content management and multilingual translation created during internationalization of IBU Ctrip

Multilingual platform with features such as multilingual content distribution

We currently provide stable multilingual content management and distribution services for dozens of overseas Ctrip sites and thousands of business applications

here

Let me explain our company Ctrip IBU

Some recommendations and summaries of multilingual team in process of improving system stability to give you some information and help

2. Background

As company's international business deepens, business becomes more and more diversified

More and more international sites are supported, and number of business parties using Shark is also increasing

The internal shark itself supports more and more business scenarios and business goals

Shark's application architecture is getting more complex

However, providing stable service capabilities has always been Shark's main goal, since its birth

The server went through about two stages of optimization and updates, namely: building a cache and solving file redundancy

Two updates based on shark's own business characteristics to develop a targeted solution

3. Evolution

Before presenting evolution of server system

Let me first explain current overall architecture of Shark app

Shark application architecture is divided into 5 levels

Application layer respectively

SDK layer, interaction layer, service layer, storage layer

The server described in this article is mainly distributed at service layer and includes several microservices

In following way:

Early version 3.1 (1.0-2.0)

At initial stage of project due to lack of access to business

The Shark system itself is also in a period of rapid development, and amount of multilingual content data managed by platform is small

The server architecture is relatively simple

It mostly depends on service capabilities of database. There is no excessive redundancy, except for separating reads and writes according to business characteristics

3.2. Creating a cache (3.0)

With gradual advancement of internationalization

The volume of data and business partners accessed by shark has increased significantly, and service response time and database load have gradually increased

Pressure testing showed that maximum number of requests per second that system can support is about 4k, while average qps value of system at that time was close to 2k.

The database server NIC export traffic peaks at around 300MB. If a large-scale full data extraction occurs, resources of network card may be exhausted

In addition, performance fluctuations sometimes occur in database due to impact of offline jobs and batch data updates on portal side

Therefore, reducing load on database and improving response of system is main goal of optimization

3.2.1 Preliminary studies

1. Almost in real time

shark is different from configuration system

Mainly to provide a multilingual near real-time data delivery service

No strict real-time performance requirements, 1-3 minute data latency is acceptable

2 Read more and write less

Shark, as a multilingual content management and distribution platform, is essentially a service that reads more and writes less

When business side business is stable, multilingual content will make fewer changes

In addition, SDK provided to business side basically obtains latest multilingual information through polling

The number of read requests will be very large

Repeat requests 3

For business application clusters

Requests to server initiated in same time period are often repeated. For example, data initialization at publish time

The operation performed by each machine is consistent, so server-side cache can efficiently solve this repeated request

4 caching methods

shark

Most query scenarios are range queries for multidimensional data matching

It's hard for a server to fully predict specific conditions of a client's request

So it's hard to create a cache in advance with an async method and finally choose a passive buffer

5 real-time data optimization

A passive cache will typically expire automatically when an expiration time is set

Thus to ensure real-time data and cache hit rates

The cache time should approximate SDK polling cycle. The server can also actively purge cache while monitoring data changes to provide real-time data

3.2.2 Caching scheme

3.2.3 Optimization effect

Cache hit rate is 90%+

Peak database traffic decreased from 300M to around 40M

Requests per second for baseline database access requests dropped from almost 2000 to about 500

Average system response time reduced from 20ms to approximately 3ms

3.3 File Redundancy (4.0)

3.3.1 Issues with 3.0

1. Server problem

Caching reduces database load and improves system response time

But stability of server indirectly depends on stability of caching system

Under normal circumstances, a business system's cache dependency is a weak dependency

Although DR can be used to increase availability of caching system

However, during failover phase, services will be unavailable for a short period of time, and disaster recovery will also waste hardware resources

2. Client problem

The main process of current client looks like this:

During startup, load full amount of data via SOA

Cache downloaded data in memory

Periodic incremental and full updates using scheduled tasks

The current client has following issues

Heavy reliance on server stability

No downgrade plan (no warranty option

Frequent and pointless full sync process

3.3.2 Circuit design

1. Overdesign

Redundant construction is dumbest way to improve reliability

It's also most efficient method. Redundant design is also called redundant design technology.

This refers to adding more than one set of functional channels, work items, or components to perform same function, when system or equipment plays a key role in completing task

To ensure that if that part fails, system or equipment can operate normally

Reduce chance of system or equipment failure, improve system reliability

2. Resource file redundancy

Server

We hope to have a redundant data retrieval channel to distribute load on server

For client, we hope to have a backup local multilingual resource

To reduce heavy reliance on server and increase client availability

Under normal circumstances, once business side business iteration is complete, multilingual content will not change frequently

Therefore, redundant resource files do not need to be updated very often

Even if contents of file are out of date, client can still correct data through incremental interface

The final solution looks like this:

3.3.3 Server redundancy

The server basically generates multilingual resource files regularly through offline tasks

Then update resource file in file system and store metadata of multilingual file in database

To better manage multilingual files and notify customers when resources need to be updated

1. File system

As a result of research, we finally chose ceph as shark resource file storage system

ceph

I have worked for Ctrip for many years and have rich experience in operation and maintenance

High stability, multi-IDC deployment, and mature disaster recovery solution

2 compression schemes

Because generated multilingual content and configuration files are small text files

There is no need to worry about compression ratio and decompression performance. To reduce external SDK dependency

We prefer to use jdk's built-in compression method as Gzip does not support direct folder compression

In end, we chose more versatile zip compression method

3. The process of creating a server resource file

3.3.4 Client Redundancy

Client mainly reduces dependency on server through local redundant multilingual files

When server is unavailable, client can provide limited services through full local redundancy

Avoid completely unavailable business application

There are two main steps in reserving multilingual resources on client side: offline initialization at application compile time and packaging step

Asynchronous update at runtime

1. Initializing compilation and packaging step

Shark-maven plugin

Java project

Typically, project management and construction is done through maven

maven divides overall task into steps, similar to a flowchart

Then do them in sequence. Therefore, we only need to develop plugin and bind it to a certain stage of default maven lifecycle

When final packaging is complete, multilingual resources that business side depends on can be added to business release package

Specific process

Reduce unnecessary resource packs

package type filter

The purpose of a multilingual resource pack is to embed it in a business release package, and typically service will depend on several maven modules

And company's release system recommends default release of military packages for Java projects

So we can filter wrapping type in plugin and only support war type for resource injection

Binding package preparation step

Currently, plugins need business development to show dependencies, and there will be frequent compilation and testing during development phase

To reduce resource downloads during development

in combination with default maven

Main features of each stage of life cycle, and finally binding of plug-in during package preparation stage

Exclude macOS and WindowsOS environments

Same as above to reduce unnecessary downloads of multilingual resources during development

Currently, plug-in does not support development of commonly used Mac and Windows environments

Business plug-in

java 
   
   
    com.ctrip.xxxx
    
   
    shark-maven-plugin
    
   
    x.x.x
    
    
     
     
      pack-download
      
     
    
  

2. Asynchronous Update at Runtime

SDK

RemoteFileLoader encapsulated for different data sources

LocaleFileLoader and RemoteSOALLoader

SDK

This is mainly used to determine if a resource has been updated based on a timestamp. If there is an update, download latest resource file from remote file system to update local backup file

To provide SDK

In order to increase availability and reduce server load, complete sdk-loader data loading process is as follows

Shark multilingual system

This is a sophisticated system platform providing multilingual content solutions for multiple terminals

I have only outlined problems and direction of development faced by server side and java-sdk side of Shark platform

Specific implementation details and detailed business logic are not described in great detail

Shark is now an important member of Taichung multilingual network

The main direction of Shark development is to improve efficiency of multilingual translation and support more machine translation scripts