Performance & tuning

Syncs

Rate limiting

Omnata's rate limiting support acknowledges that different workloads have different levels of importance, and sometimes you need to restrict particular syncs to stay under a certain request rate.

In other instances, applications set rate limits based on pricing plan or environment type. For out of the box connectors, Omnata is aware of these rate limits and sets sensible defaults for the connection.

Once a sync is configured, each Omnata Plugin is able to define "categories" of rate limits which match particular HTTP requests. The rate at which the plugin is allowed to make HTTP requests to a particular category, can be overridden at the sync level.

Tuning parameters

Additionally, Omnata exposes tuning parameters for each plugin.

These levers enable you to adjust other performance related behaviours specific to the application API or database. For databases, tuning parameters primarily adjust the query workload on the source server. For applications APIs, tuning parameters can adjust query workload and also whether to use an API feature or not.

Tuning parameters can be used to (amongst other things):

• Adjust parallelism and size of queries

• Optimize for initial loads versus incremental syncs

• Resolve timeout issues and errors

• Optimize sync latency

Snowflake

Warehouses

For virtually all sync workloads, we recommend using an XS warehouse.

If you are kicking off multiple syncs at the same time and seeing query queuing, add multi-clustering to the XS, as opposed to using a larger warehouse size.

Specifically for ingestion workloads from databases, upgrading to a S warehouse can improve processing times for large sync payloads.

Streamlit performance

The Omnata UI runs inside Streamlit as a custom component. The initial load of Streamlit can sometime take more than one minute. If the warehouse you are using for Streamlit is busy with other workloads, you may also see load errors or slow query response times.