GDA Institutional-grade crypto data lake architecture design
Collecting High-resolution Cryptocurrency WebSocket L3 Equivalent Feeds with Libwebsockets, Workflows, ECS, S3, Athena and Lambda
Websockets tend to be unstable, especially on busy networks, be it an abnormal exit of a process, a server breakdown or broken network. As cryptocurrencies are traded around the clock, these occurrences may prevent successful trading. Trades, especially those with high leverage, may be liquidated, resulting in a loss. It would be unacceptable for a high-frequency trading algorithm to remain down for maintenance while major movements are occurring in the market, as this is exactly when a reliable feed is most needed.
Low Latency & High Availability Crypto Data Collection
The feed must be operational 24/7. An ideal solution involves a primary ECS microservice connected to one content delivery network, and a replica service pointing to another content delivery network. The trick however, is being able to respond to an outage in microsecond timescales. This is achieved by each microservice using the libwebsockets library, which is low level and includes policies dictating the handling of emergencies.
Libwebsockets is a simple-to-use, MIT-licensed, pure C library providing client and server for http/1, http/2, websockets, MQTT and other protocols in a security minded, lightweight, scalable and flexible way. It is easy to build and cross-build via cmake and is suitable for tasks from embedded RTOS to mass cloud serving.
Advantages of Flow Programming and AWS ECS Microservice Architecture
Microservices are Independently Deployable and Cloneable
Each microservice is independent, enabling continuous improvement and faster app updates.
Microservices Reduce Downtime Through Fault Isolation
If a specific microservice fails, you can isolate the failure to that single service and prevent cascading failures which would cause the app to crash. This fault isolation means that your critical application can stay up and running even when one of its modules fails.
Microservices typically have small codebases, making them easier to maintain and deploy. It’s also much easier to keep the code clean and for teams to be wholly responsible for specific services.
What is AWS Lambda?
AWS Lambda is a serverless computing service provided by Amazon Web Services (AWS). Users of AWS Lambda create functions, self-contained applications written in one of the supported languages and runtimes, and upload them to AWS Lambda, which executes those functions efficiently and flexibly. They can also be scheduled or used in an AWS workflow to instantiate a data feed microservice or to monitor the health of an existing feed.
Pricing: $0.20 per 1M requests
Comment: The free tier includes 1M requests. And latency is low with Python cold call 200ms, warm call 20ms
What is ECS Fargate?
Amazon Elastic Container Service (Amazon ECS) allows you to deploy containerized workloads on AWS quickly. The powerful simplicity of Amazon ECS enables you to grow from a single Docker container to managing your entire enterprise application portfolio. It also allows you to run and scale your container workloads across availability zones.
What Does Fargate Mean?
Serverless by default with AWS Fargate: AWS Fargate is built-in to Amazon ECS, which means you no longer have to worry about managing servers, handling capacity planning, or figuring out how to isolate container workloads for security. Just define your application’s requirements, select Fargate as your launch type in the console or CLI, and Fargate takes care of all the scaling and infrastructure management required to run your containers. Meaning there are no servers to manage where traditionally there would be a swarm of servers to maintain.
Using Lambda functions and Cloudwatch to monitor the health of the feeds
It is important to offer a final-level but thorough protection. This involves stereoscopic monitoring, ECS tasks, health status, performance indicators, access frequency and times, error abnormality, client interface (error abnormality, page performance, network status, hardware and software environment, etc.) and business indicators (Bybit LOB IDs incrementing, feed traffic, volumes of events per minute, volatility, rates).
We will monitor all valuable indicators through pre-warning and quick post response, and provide corresponding responsible persons with different warning methods such as mail, SMS, and telephone for varying levels of problems.
ECS tasks are co-hosted with Bybit servers in Singapore.
- Bybit Inverse Perpetual BTCUSD L3
- Bybit USDT Perpetual BTCUSDT L3
- Bybit Inverse Perpetual Others L3
- Bybit USDT Perpetual Others L3
- Bybit funding rate
- Bybit maker and taker fees (No API)
- Bybit system and maintenance alerts (No API)
- Binance Spots ALL L2
- Coinbase Spots ALL L3
- Binance Derivatives ALL L3
- Binance Spots ALL L3
- Uniswap ALL L2
- 1inchechange All L2
- Yearn Finance
- All JSON documents will need a type and version number stored in minute buckets and stored in Parquet format.
- Contact Bybit via Telegram for Maker-Taker fees, funding rate.
- Multiple Content Delivery Networks (Bybit) & (Bytick) can be used as a fallback in extreme emergencies
- Continuous integration toolchain with GitHub and high unit test coverage.
Leverage and Margin Checks
In trade execution, we need to calculate the impact on margin and leverage level; this will be calculated pre-trade and compared with the WebSocket feed post-trade. Any discrepancies will be flagged and actioned according to the severity. Dev Ops will code all business rules into a C library, and checks must run in the microsecond range.
Warning level will be set in a configuration to send out messages before liquidations occur.
There are often new announcements or changes to the margin rules. Should they impact our systems, we require the correct toolchain: continuous integration, testing, etc. — additionally, where we can, rolling back changes while handling multiple changes if need be.