Connectivity Tools, Technologies and Approaches in Crypto and Web 3.0
CEX and DEX Adapters, Routers, Smart Order Routing & Order Matching Engines
· CEX and DEX Adapters
∘ Orion Protocol
∘ O3 Swap
· Smart Order Routing
∘ Uniswap Auto Router
∘ SFOX Smart Routing and Net-Price Routing
· Order Matching Engines
∘ Market Grid
∘ B2 Broker
As part of our upcoming Project Fusion, a series of adapters are required in order to connect our services to those provided by centralized exchanges, decentralized exchanges and beyond. In this article, we will explore some of the key technologies behind the task ahead, as well as a number of existing solutions throughout the spaces of crypto and traditional markets.
CEX and DEX Adapters
The purpose of the CEX and DEX adapter is to allow for a direct communication route between our IDE and the exchanges. The IDE will be able to operate multiple accounts and place orders directly to any of our supported exchanges simultaneously.. While solutions for connection to multiple exchanges do exist, it is often not very well documented publicly and the vague terminologies in crypto space make searching for solutions often frustrating. Additionally, some protocols claim to be solving similar problems in their whitepaper, currently available products are not still in development.
Sfox is a CEX aggregator with an adapter connecting to several centralized crypto exchanges. By aggregating multiple CEXs on one platform they give you the ability to trade on all major exchanges from a single account with low-latency, automatically slicing your orders and routing them to the most competitive venues for a better price. Instead of having to decide on which is the best exchange to be trading on at any given moment, Sfox draws data from several CEXs and creates a unified order book and will intelligently place your order on the best exchange when executing orders.
They provide a well documented API allowing their users to directly interact with their platform and place orders and instructions. With various variants on execution algorithms and routing types, Sfox has leveraged its platform to provide its users with a smarter way to trade. Unfortunately, while their solution seems well thought out and sophisticated, public information on their methodology is practically non-existent, with them failing to even list their supported exchanges, only listing their currently supported trading pairs on their market page.
Finxflo is another company that has aggregated multiple exchanges, this time in both the CEX and DEX ecosystems. They are a hybrid liquidity aggregator that allows its users to access the best prices from both CEXs and DEXs. Like Sfox, when users trade on the FinxFlo platform they are effectively trading across multiple exchanges and benefiting from the combined liquidity which mitigates any price discrepancies and minimizes slippage.
They are a centralized regulated exchange located in Singapore and describe themselves as a multi-jurisdictional, crypto prime brokerage and liquidity aggregator allowing for portfolio management for traders through a single account. However, they don’t seem to have any proper public documentation with even less information than SFOX. Additionally, they currently lack an API to allow users to connect to their services directly.
Swapzone is an instant non-custodial cryptocurrency exсhange aggregator. They “monitor the best exchange rates for thousands of available cryptocurrency pairs supported by exchange providers and gear you up with the information on the crypto exchange offers and services so that you could pick the best and convert cryptocurrency instantly, with low fees and no registration.”
Swapzone describes its operating structure as: “the platform sends users’ inquiries in; gets the deals directly from its partners, sorting them out by rate, speed and rating; and lets users make swaps via chosen providers without leaving the Swapzone website.” Users are provided with a list of all the exchangers and the rates (& trading fees) they charge from the users for respective transactions. The user’s customized list is generated by the exchange’s provided open APIs.
The platform sends in a request to the respective exchanges for the transaction entered by the individuals and fetches information for the same. They do have API support although unlike SFox, there doesn’t seem to be customisation in trading algorithms or trading routes.
Soft-fx provides a similar platform to the examples above, but deals in both FOREX and crypto space. They provide a cryptocurrency trading platform described as an all-in-one application for margin trading and exchange of Crypto and derivative trading instruments. With features of: Advanced Trade matching engine, FIX API, REST API, and Web Sockets API support, and Algorithmic studio with preset Market Making algorithms.
They also provide a TickTrader Liquidity Aggregator, designed to help you connect and collect the best liquidity from an unlimited number of providers as an FX or cryptocurrency exchange aggregator. Like the others, public documentation and information is not available.
Orion Protocol is a decentralized liquidity aggregator protocol that bridges the gap between the centralized and decentralized crypto platforms. Orion Protocol aggregates the liquidity of CEXs, DEXs, and swap pools into one decentralized platform. The protocol aims to connect traders to the entire crypto market via one interface.
Orion has developed a custom-designed matching engine, which connects to dozens of different exchanges in real-time enabling their users to trade their combined order books from one place without giving up their private keys.
O3 Swap is a liquidity aggregator for DEXs. They describe itself as a “cross-chain aggregation protocol that enables free trading of native assets between heterogeneous chains by deploying ‘aggregator + asset cross-chain pool’ on different public chains and Layer 2 granting users access to cross-chain transactions with one click”. Unlike most companies/protocols here, O3 swap does provide a list of where the liquidity for each of its asset pairs come from.
0x is a peer-to-peer exchange of ERC-20 tokens and can be thought of as its own DEX. 0x founders describe their system as an “off-chain order relay with on-chain settlement” and it’s often refers to its solution as the “Craigslist for cryptocurrencies,” in that any developer can build their own cryptocurrency exchange and post onto its platform.
The 0x protocol utilises common smart contracts over a shared infrastructure. Its technology combines two strategies: state channels and automated market markers (AMMs). State channels reduce the costs incurred of transactions on the Ethereum network by taking transactions offline. AMMs are introduced to facilitate trades between two parties if the price of a crypto asset reaches a certain threshold. 0x’s own Ethereum token (ZRX) is used to pay trading fees to relayers, which are used to connect makers with takers.
Hummingbot is an open-source software client that offers users the opportunity to create and monetize automated and algorithmic trading bots. It provides an easy-to-use command-line interface that makes it possible for you to configure, customize, and run automated bots and other strategies. With this, users can make markets on both decentralized and centralized digital asset exchanges. As the project is open source, documentation is thorough and public and would serve as a good base for any project with the goal of aggregating multiple crypto exchanges.
Smart Order Routing
Smart order routing (SOR) is an automated process for handling orders with the intent of attaining the most desirable path across venues. Generally, an SOR searches for the most efficient route to swap tokens for another token type.
Uniswap Auto Router
Uniswap’s automated liquidity protocol allows users to swap one asset for another. To facilitate this, they have implemented an auto router which makes a series of swaps across multiple pairs (the ‘route’) to attain the desired set in the event of a trader creating an order for an asset that is not included in the pair pool they are trading through.
The route from one asset to another may consist of segments (points at which tokens are exchanged for other tokens along the way) between pairs. Though each segment will contribute to the overall cost of the trade in the form of gas fees.
In the event of a large order being placed, or low liquidity in the pair pool, routes may also consist of multiple paths. By hedging routes, orders are protected against a singular point of loss to slippage. Introducing multiple paths will also increase the gas fees required to complete the trade as each path adds at least one additional transaction.
Routes can be split across up to seven routes, which enables a trader to take advantage of the liquidity depth of different pools. The following diagram depicts a scenario where a trader saves $134,689 on account of the Auto Router’s splitting function.
The Auto Router also factors gas fees into its calculations for cost efficiency — small trades will execute with minimal hops to reduce the computations which incur gas fees. This can be seen in the following example where the net cost is reduced by 10% by not making an additional trade along the route.
The Auto Router comes complete with a user interface through which a trader can view the path of their trade before execution.
In an analysis of 4,429 on-chain swaps from June 1, 2021, Uniswap claims the Auto Router improved pricing on 13.97% of all trades and 36.84% on trades between the top ten tokens by TVL.
As of mid December, 2021, Uniswap updated their Auto Router by introducing the following improvements:
Uniswap V2 and V3 Support
The Auto Router now further improves trade prices by routing trades across all Uniswap pools, including V2 and V3 pools, while optimizing gas fees in the process — meaning that traders need not consider which version of Uniswap they are using.
Layer 2 Network Support
All Uniswap Layer 2 networks are now supported by the Auto Router, this includes Optimistic Ethereum and Arbitrum. The intent is to combine the trade pricing efficiency of the Auto Router with the low gas fees associated with Layer 2 networks.
Gas Aware Route Splitting Across V2 and V3
Trades can now be split across both V2 and V3 liquidity pools to find the best price across both, improvements brought about by the introduction of a SwapRouter smart contract. To save gas fees, old router prices are compared against new router prices and selects the lower of the two.
Real-Time Gas Estimates
Gas fees may now be estimated via the application’s interface, allowing traders to further gauge the profitability of a swap.
Smarter Auto Slippage
Gas estimates are incorporated into slippage calculations to automatically set slippage based on the trader’s expected gas cost and trade size. For smaller trades, slippage tolerance is adjusted to improve the odds of a trade successfully executing, in order to avoid situations where the gas fees involved with failed trades requiring additional transactions — and thus, higher gas fees.
In this update, Uniswap also introduced a number of developer tools:
Auto Router npm Package
The Auto Router has been packaged onto the npm registry, allowing developers to quickly and easily import it into their projects.
The SwapRouter contract is fully open source and can be viewed on the project’s public GitHub repository.
Swap and Add
The router contract now offers atomic swap-and-add functionality, allowing liquidity provision of only one of the assets in a pool, or any ratio of the two assets (opposed to the traditional requirement of offering a set ratio of both assets in a pool as liquidity provision).
Auto Router allows a limit to be set on the relative price impact of trades as measured against V3 oracles, rather than only measuring against on-chain liquidity. For example: “a user swapping ETH for USDC could specify that their swap can only execute within one percentage point of the five minute ETH/USDC time-weighted average price”.
SFOX Smart Routing and Net-Price Routing
SFOX approaches SOR with their Smart Routing algorithm, which handles orders in such a way that aims to take the best opportunity throughout a range of different trading venues.
In 2020, SFOX improved upon their Smart Routing solution by introducing their new Net-Price Routing (NPR), which touts a supposed 28% average saving on trading fees. NPR aims to address issues relating to uneven fees in crypto, and the fact that many smart order routers calculate using the gross prices of assets. NPR finds the route to execute a trade which yields the best net price, factoring fees in.
Balancer’s SOR solution is an “off-chain linear optimization of routing orders across pools for best price execution”. It takes an amount of input tokens and desired tokens to trade them for, then returns a list of pools and amounts that should be traded to maximize the amount of returned tokens. The tool is available for free as an npm package for use by developers who aim to implement order routing across pools.
Balancer sees SOR as an optimization problem where the aim is to find the path through a set of Balancer Pools with the highest net yield after gas costs. The algorithm adds pools to a trading set until there are no pools left where the net gain from trading with the pool would exceed the gas cost.
The current Balancer SOR runs off-chain, but is EVM-tractable by design in preparation for a future on-chain release. In order to achieve EVM-tractability, the function used to calculate the spot price of a Balancer pool has been linearized — this is also useful in adjusting for post-trade spot price change and slippage.
We can expect the price to increase after trading due to slippage, the larger the trade, the larger the resultant change in spot price. This linearized approximation of the spot price is exactly that, an estimate, the following diagram shows the real (non-linear) spot price after a trade compared with the linearized approximation.
As the spot prices of all Balancer Pools are linearized, Balancer interpolates prices and amounts to simplify their optimization problem. Imagine three Balancer Pools containing the token which is being traded, the estimated prices (EP) interest are defined as sets of prices under the following conditions:
- There is a pool with exactly that initial spot price
- The spot prices of two pools ‘cross’ at that price
An Example SOR Optimization Problem
For the following walkthrough, Balancer consider only Pools 1 and 2 for their explanation.
“Let A be the amount of token in (Ai) to be traded on pool 1 such that its price increases from EP1 (initial spot price of pool 1) to EP2 (initial spot price of pool 2). The SOR’s solution for any amount of tokens lower than A is simply: “trade the entire amount with pool 1.”
When the amount traded is greater than A, SOR will start including pool 2 in the solution, as not doing so would mean the trader is trading some amount (Ai — A) for a higher price than they could with pool 2.
The solution for trading an amount B + C can be found by interpolating the trades that result in EP2 and EP3. By trading C with pool 1 and B with pool 2, both pools end at the same price (Final Price) which means that the best solution was found.”
Essentially, the algorithm predicts spot prices in pools as a function of the amount to be traded, identifies price points of interest, and aims to find swaps with pools that yield greater returns depending on the amount the trader actually trades.
Order Matching Engines
Order matching engines are used to ‘match’ limit orders to market orders, this system is used in traditional finance and centralised exchanges to a great extent. However, as our adapter system will be connecting to multiple venues an order matching engine is essential for DEXs and other smart contracts, an order matching engine is required to ensure optimal trade execution.
Market grid is a order matching system for traditional finance, their building process consists of three steps
- Build the Data Model. The first step in building a MarketGrid system, before a line of code is written, is to build the data model.
2. Generate the Base System. From the data model a single make command generates the entire system:
- Matching Engine Code
- Communications Gateway
- Static Database
- Market Database
- GUI — HTML based
3. Implement new or changed business logic. Once a base system has been generated using the MarketGrid Tools, all that remains is to implement any new or changed business logic for the particular implementation.
- Capable of 190,00 Orders Per Second.
MarketGrid has built-in timing metrics that can time-stamp each transaction as it transitions through the various components of the system. Using off-the-shelf PC hardware, the matching engine itself can process orders under two microseconds, whilst a client process can achieve round-trip latency of less than five microseconds.
This was achieved using a single server with configuration:
- Intel Core i7–2600K (SandyBridge) @ 3.4GHz
- 16GB DDR3 RAM @ 1333MHz
- CentOS 7.1 Linux
Exchange-grade customisable tool for matching supply and demand on a wide range of asset classes — from limited edition fashion items to prediction markets. Light to deploy, and easy to integrate, operate, and scale as necessary. The website provides very limited information on the performance and architecture of their system.
DXmatch is a modular system built for launching exchanges and dark pools that operate in OTC (FX and crypto), commodities, and regulated equities and derivatives markets.
- Sub 100-microsecond end-to-end latency via FIX API
- Throughput of 500,000+ orders per second through up to 128 horizontally scalable matching segments
- Hardware-accelerated Direct Memory Access (DMA) zero-copy networking for ultra-low latency
- Runs on bare metal or in the cloud
Event sourcing architecture and deterministic behavior and output, the system employed a Multi-node reliable clustered configuration with automatic failover. In addition to this the infrastructure and algorithms avoid a single point of failure by leveraging consensus RAFT protocol.
A Matching Engine is the core mechanic of a digital exchange which matches up bids and offers to execute trades. The ME algorithm keeps a record of all open orders in a market and generates new trades if the two orders can be fulfilled by each other. It provides solutions for traditional finance. They boast speeds of 15,00 OPS at 50us latency. B2 Broker also provides highly tailored data for brokerage related information such as Balances, User Activity, Commissions, Dealing and Transactions. This data is essential in a decentralised space where data can make or break trade decisions. The system employs the following system, architecture:
W. Warren and A. Bandeali, “0x: An open protocol for decentralized exchange on the Ethereum blockchain,” p. 16.
“Warren and Bandeali — 0x An open protocol for decentralized exchange on.pdf.” Accessed: Feb. 07, 2022. [Online]. Available: https://0x.org/pdfs/0x_white_paper.pdf
C. | O. Wallet, “Annual Report 2021 — O3 Labs,” O3 Labs, Jan. 21, 2022. https://medium.com/o3-labs-o3-wallet/annual-report-2021-o3-labs-f687bd3791d5 (accessed Feb. 07, 2022).
SFOX, “API Reference.” https://www.sfox.com/developers/ (accessed Feb. 07, 2022).
marketg, “Architecture | MarketGrid.” http://marketgridsystems.com/technology/architecture/ (accessed Feb. 07, 2022).
“Auto Router V2,” Uniswap Protocol, Dec. 16, 2021. https://uniswap.org/blog/auto-router-v2 (accessed Feb. 07, 2022).
“B2Trader — The Best Matching Engine Software,” B2Broker. https://b2broker.com/products/b2trader-cryptocurrency-exchange-software/ (accessed Feb. 07, 2022).
“Better Prices, Lower Fees: Announcing Net-Price Routing by SFOX,” SFOX, Feb. 05, 2020. https://www.sfox.com/blog/sfox-pricing-crypto-trading-net-price-routing/ (accessed Feb. 07, 2022).
H. Foundation, “Documentation — Hummingbot Foundation.” https://hummingbot.org/docs/ (accessed Feb. 07, 2022).
“DXmatch_web.pdf.” Accessed: Feb. 07, 2022. [Online]. Available: https://devexperts.com/app/uploads/pdf/DXmatch_web.pdf
“FINXFLO — The First Hybrid Liquidity Aggregator.,” Medium. https://finxflo.medium.com (accessed Feb. 07, 2022).
“Introducing the Auto Router,” Uniswap Protocol, Sep. 16, 2021. https://uniswap.org/blog/auto-router (accessed Feb. 07, 2022).
“Order Types & Execution Algorithms,” SFOX. https://www.sfox.com/order-types/ (accessed Feb. 07, 2022).
“Smart Order Router.” https://balancer.gitbook.io/balancer/smart-contracts/sor (accessed Feb. 07, 2022).
“Solutions,” Exberry. https://exberry.io/solutions/ (accessed Feb. 07, 2022).
“Swapzone Review 2022: Check Features, Fees & More,” Jan. 31, 2022. https://www.cryptonewsz.com/cryptocurrency-exchange/swapzone-review/ (accessed Feb. 07, 2022).
“Swapzone Review 2022: Check Features, Fees & More,” Jan. 31, 2022. https://www.cryptonewsz.com/cryptocurrency-exchange/swapzone-review/ (accessed Feb. 07, 2022).
“The Best Cryptocurrency Exchange is not an Exchange,” SFOX, Jul. 14, 2020. https://www.sfox.com/blog/the-best-cryptocurrency-exchange-is-not-an-exchange/ (accessed Feb. 07, 2022).
“Top Crypto Liquidity Providers for Exchange or Broker and How to Choose the Best of Them,” Soft-FX, Mar. 23, 2021. https://www.soft-fx.com/blog/top-liquidity-providers/ (accessed Feb. 07, 2022).