API scalability refers to the ability to support concurrent requests without impact on performance.
The ecommerce industry has been exponentially growing year over year since 2015 and saw an even higher spike with the advent of COVID-19 last year. However, many top shipping API providers are built on outdated infrastructure that wasn't designed for the scale we are now seeing in 2021 as well as the growth we expect to see moving forward.
A truly scalable API supports heavy volume across its entire customer base through the peak shipping season with no impact on performance. Shipping API companies shouldn't be able to hide behind the excuse: shipping API performance decline is unavoidable during peak season. Ecommerce companies invest significant resources to meet the scale of demand during the peak season including increased inventory, staffing, additional hours, and more. The same should be expected from the shipping API solution in which they invested.
Our APIs are designed to support magnitudes of volume more than we see today. The EasyPost team is constantly improving our infrastructure so we are ready to grow exponentially in the next year. We proudly publish the performance of our shipping API response time metrics for the last 24 hours here. In the next section, we describe in detail how we measure scalability and what customers can expect from our APIs.
EasyPost evaluates the scalability of our APIs by looking at our maximum supported volume in requests per second without performance degradation. This tells us how much new volume we can onboard without expecting latency in steady-state and peak season volumes.
The below metrics outline our load capacity and API performance through peak season when we saw peak spikes in volume. We saw no performance degradation on our core label API and shipment API services during the peak season. We define peak season as November and December because this is where we see the spike in volume happen.
In the next section, we outline some of the key innovative things we are doing to improve the scalability of our APIs.
The below list of items details some of the key innovative methods we employ to optimize API scalability. However, there are many common industry best practices we also employ not described below.
1. Stateless
EasyPost ensures APIs are stateless where possible to optimize the efficiency of requests and load to our servers. Stateless APIs do not access information from previous interactions because, by design, incoming requests include all information needed to fulfill the request. For this reason, multiple servers can support a given request.
One great example is EasyPost uses stateless authentication by independently verifying customers on each new request with an API key. A stateful API accesses information from previous client sessions to fulfill requests. To scale a stateful API across many servers, each client session would need to be stored across all of those servers. In cases where we are storing information, we regularly analyze the usefulness of that information to ensure we are making the optimal scalability trade-offs.
2. Asynchronous job processing
EasyPost uses asynchronous job processing to rapidly complete bulk operations which help to mitigate long queues of backlogs in our system. If two or more jobs do not depend on one another, we run them in parallel (asynchronously) to improve performance. We have seen asynchronous job processing improve performance by up to 10x speed for bulk operations such as purchasing a batch of 1,000 shipments. This is especially critical during the peak shipping season when bulk requests are more common and backlog queues are longer.
3. Hybrid NoSQL and SQL data management approach
EasyPost leverages both SQL and NoSQL databases for much more efficient and scalable data management. In 2020, EasyPost took a massive scalability leap by making a significant investment to pursue this unique hybrid approach. Simpler, faster read operations are usually better suited for a SQL database, whereas heavier write operations (which are often the bottlenecks for scaling infrastructure) are better suited for NoSQL databases. For any given operation, we dynamically leverage whichever infrastructure is better for performance and scalability. Our infrastructure is now set up to exponentially scale with the burgeoning ecommerce industry.
4. Regular monitoring and alerting
EasyPost reduces problematic bottlenecks that may cause latency for customers by regularly monitoring the load on our system and alerting internal teams if unhealthy thresholds are exceeded. When alerted, we immediately begin work to build and deploy solutions to return our systems to a healthy state. We have staff on call at all times of the day ready to respond to critical alerts for all supported systems. Regular monitoring and alerting also gives us an accurate pulse of where our current scalability limitations are so we can better prioritize builds and push scalability boundaries in the future.
5. Dynamic rate limiting
EasyPost uses dynamic rate-limiting thresholds to increase max allowable throughput for customers. Different requests cause different loads to our system depending on the amount of information requested. For this reason, we dynamically set thresholds for our rate limits based on the cumulative load on the API rather than a threshold based on the number of requests made to the API. This means that users won't be rate limited due to sheer volume when making numerous performant requests in a short span.