GDA Institutional-grade crypto data lake architecture design

Collecting High-resolution Cryptocurrency WebSocket L3 Equivalent Feeds with Libwebsockets, Workflows, ECS, S3, Athena and Lambda

Websockets tend to be unstable, especially on busy networks, be it an abnormal exit of a process, a server breakdown or broken network. As cryptocurrencies are traded around the clock, these occurrences may prevent successful trading. Trades, especially those with high leverage, may be liquidated, resulting in a loss. It would be unacceptable for a high-frequency trading algorithm to remain down for maintenance while major movements are occurring in the market, as this is exactly when a reliable feed is most needed.

Low Latency & High Availability Crypto Data Collection

The feed must be operational 24/7. An ideal solution involves a primary ECS microservice connected to one content delivery network, and a replica service pointing to another content delivery network. The trick however, is being able to respond to an outage in microsecond timescales. This is achieved by each microservice using the libwebsockets library, which is low level and includes policies dictating the handling of emergencies.

Libwebsockets

Libwebsockets is a simple-to-use, MIT-licensed, pure C library providing client and server for http/1, http/2, websockets, MQTT and other protocols in a security minded, lightweight, scalable and flexible way. It is easy to build and cross-build via cmake and is suitable for tasks from embedded RTOS to mass cloud serving.

Advantages of Flow Programming and AWS ECS Microservice Architecture

Microservices are Independently Deployable and Cloneable

Each microservice is independent, enabling continuous improvement and faster app updates.

Microservices Reduce Downtime Through Fault Isolation

If a specific microservice fails, you can isolate the failure to that single service and prevent cascading failures which would cause the app to crash. This fault isolation means that your critical application can stay up and running even when one of its modules fails.

Microservices typically have small codebases, making them easier to maintain and deploy. It’s also much easier to keep the code clean and for teams to be wholly responsible for specific services.

What is AWS Lambda?

AWS Lambda is a serverless computing service provided by Amazon Web Services (AWS). Users of AWS Lambda create functions, self-contained applications written in one of the supported languages and runtimes, and upload them to AWS Lambda, which executes those functions efficiently and flexibly. They can also be scheduled or used in an AWS workflow to instantiate a data feed microservice or to monitor the health of an existing feed.

Pricing: $0.20 per 1M requests

Comment: The free tier includes 1M requests. And latency is low with Python cold call 200ms, warm call 20ms

What is ECS Fargate?

Amazon Elastic Container Service (Amazon ECS) allows you to deploy containerized workloads on AWS quickly. The powerful simplicity of Amazon ECS enables you to grow from a single Docker container to managing your entire enterprise application portfolio. It also allows you to run and scale your container workloads across availability zones.

What Does Fargate Mean?

Serverless by default with AWS Fargate: AWS Fargate is built-in to Amazon ECS, which means you no longer have to worry about managing servers, handling capacity planning, or figuring out how to isolate container workloads for security. Just define your application’s requirements, select Fargate as your launch type in the console or CLI, and Fargate takes care of all the scaling and infrastructure management required to run your containers. Meaning there are no servers to manage where traditionally there would be a swarm of servers to maintain.

Using Lambda functions and Cloudwatch to monitor the health of the feeds

It is important to offer a final-level but thorough protection. This involves stereoscopic monitoring, ECS tasks, health status, performance indicators, access frequency and times, error abnormality, client interface (error abnormality, page performance, network status, hardware and software environment, etc.) and business indicators (Bybit LOB IDs incrementing, feed traffic, volumes of events per minute, volatility, rates).

We will monitor all valuable indicators through pre-warning and quick post response, and provide corresponding responsible persons with different warning methods such as mail, SMS, and telephone for varying levels of problems.

ECS tasks are co-hosted with Bybit servers in Singapore.

Requirements

Critical Features

- Bybit Inverse Perpetual BTCUSD L3

- Bybit USDT Perpetual BTCUSDT L3

- Bybit Inverse Perpetual Others L3

- Bybit USDT Perpetual Others L3

- Bybit funding rate

- Bybit maker and taker fees (No API)

- Bybit system and maintenance alerts (No API)

Important Features

- Binance Spots ALL L2

https://binance-docs.github.io/apidocs/spot/en/#websocket-market-streams

- Coinbase Spots ALL L3

- Binance Derivatives ALL L3

https://developers.coinbase.com/

Desirable Features

- Binance Spots ALL L3

- Uniswap ALL L2

- 1inchechange All L2

- Yearn Finance

- All JSON documents will need a type and version number stored in minute buckets and stored in Parquet format.

- Contact Bybit via Telegram for Maker-Taker fees, funding rate.

- Multiple Content Delivery Networks (Bybit) & (Bytick) can be used as a fallback in extreme emergencies

bybitglobal.com

- Continuous integration toolchain with GitHub and high unit test coverage.

Trade Execution

In trade execution, we need to calculate the impact on margin and leverage level; this will be calculated pre-trade and compared with the WebSocket feed post-trade. Any discrepancies will be flagged and actioned according to the severity. Dev Ops will code all business rules into a C library, and checks must run in the microsecond range.

Warning level will be set in a configuration to send out messages before liquidations occur.

There are often new announcements or changes to the margin rules. Should they impact our systems, we require the correct toolchain: continuous integration, testing, etc. — additionally, where we can, rolling back changes while handling multiple changes if need be.

--

--

--

GDA is developing the decentralized financial application development environment and rapid financial engineering protocol built on Ethereum.

Love podcasts or audiobooks? Learn on the go with our new app.

Analyzing and resolving dependencies to free microservices from a monolith

How To Learn Data Structures And Algorithms

What does it mean to be a Dev(Sec)Ops Engineer?

WLSDM Wishlist: New Feature Requests

What to do when all else fails!

AMP vs Responsive Web Design: Must Read Before Choosing Either

AMP vs Responsive Web Design : Must Read Before Choosing Either

Rari’s Multi-sig and Security Standards

Project planning, setting milestones for app development at Startup

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
GDA Fund

GDA Fund

GDA is developing the decentralized financial application development environment and rapid financial engineering protocol built on Ethereum.

More from Medium

Introducing Snowflake

New Pyth Data Provider: ML Tech

NFT Analytics With Evaluate Market

Deploy a ML Model using the Google Cloud Console