Building a High-Performance Time Series Database from Scratch

Application Performance Monitoring and Kubernetes monitoring in their current state are pretty expensive. The average VictoriaMetrics installation is processing 2-4 million samples/s on the ingestion path, and 20-40 million samples/s on the read path. The biggest installations account for 100 million samples/s on the ingestion path. This requires being very
clever with data pipelines to keep them efficient and scalable by adding more resources. In the talk, I’ll cover the following optimizations we find very helpful for keeping the database fast:

1. String interning for lowering GC pressure. We use string interning for storing time series metadata (aka labels). The metadata usually remains unchangeable for data streams, but still is required to be collected and ingested each time. String interning helps us to heavily utilize in-memory caches instead of allocating memory when parsing such data. However, string interning has the downside of increased memory usage. So the tricky part is maintaining a balance between CPU and memory resources.

2. Results caching for functions like ` func (string) string` and `func(string) bool`. This helps to save CPU when processing metrics lookups via regular expression or various metadata manipulations in-memory. The downside is, as in the previous, increased memory usage.

3. Limiting the number of concurrently running goroutines with CPU-bound load by the number of available CPU cores. This helps to control the memory usage on load spikes (which is a frequent event in monitoring). The limit also improves the processing speed of each goroutine, since it reduces the number of context switches. The downside of the approach is its complexity – it is easy to make a mistake and end up with a deadlock or inefficient resource utilization.

4. The better understanding of `sync.Pool`. For us, `sync.Pool` shows itself the best when used in CPU-bound code, while in IO-bound code it leads to excessive memory usage. The CPU-bound code has short ownership over the objects retrieved from the pool. In combination with p.3 (limited number of goroutines processing CPU-bound code) it gives the most efficient processing speed and memory usage since the chance to get a “hot” object from the pool is much higher.