Inbound

Overall process

Inbound syncing involves less configuration complexity than outbound, since you don't need to define exactly how the data is received by the app. Instead it's pretty simple - you get a Snowflake table per object from the app. For this reason, and to avoid you needing to configure lots of syncs, each sync can support any syncing number of inbound objects from the app.

In keeping with established industry naming conventions, each object within an inbound sync is called a "stream", and each one will have a table per branch of the sync (and the "main" branch).

Inbound sync strategies

By default, Omnata will automatically fetch data incrementally when possible, reverting to full-refresh if not supported by the source. You can choose the storage behaviour; whether to merge changes or keep history (append).

The configuration wizard presents three options:

• Auto / Merge changes

• Auto / Keep history

• Custom - choose sync behaviours for each object

Custom sync strategies

For more complex use cases, you can set the sync strategy for each object in a sync:

Full - All data is reloaded every sync

Incremental - Using a watermark, only new changes are pulled

Storage behaviours determine how the fetched data is stored in Snowflake, considering existing synced data.

The Full strategy has two behaviours:

• Replace - The latest copy of the full set of records replaces the existing records

• Append - The latest copy of the full set of records is added to the existing set of records. Note that this will create duplicates of existing records.

The Incremental strategy has two behaviours:

• Merge - The new records are matched up to existing records with the same primary key, or inserted if there is no match.

• Append - The new records are added to the existing set of records. Note that this will result in multiple records with the same primary key.

Inbound storage location

A local table is nominated and created automatically, with all of the available fields for the selected app object.

By default, Omnata lands the data in the Omnata Sync Engine app database. Raw data is landed in the INBOUND_RAW schema and a normalized view is automatically created for most sources in the INBOUND_NORMALIZED schema.

You can customize the inbound storage location for a sync, under the "Storage Location" section. This will modify the storage location for that sync, and will move the existing tables to the nominated location. You will need to grant the Omnata application privileges to operate on the custom location. Go to: Modifying the inbound storage location to a database and schema outside of the app

You can also customize the default location under "Data Management" in the Settings menu. This will not modify existing syncs, only set the default location for new syncs.

When customizing the storage locations, you are defining templates using the Jinja language. This ensures that as tables and views need creating for different branches, streams and even columns, that they are all written to a unique location.

When creating templates for database, schema and table/view names, the following Jinja variables are available:

• sync_slug: The slug of the sync, visible on the sync home page. As this is a unique identifier, it makes the name unique across different syncs.

• branch_name: The name of the branch, this makes the name unique across different branches within a sync. If branching mode is not enabled, this will resolve to 'main'

• stream_name: The name of the stream, as provided by the plugin.

When creating templates for column names, the following Jinja variables are available:

• column_name: The name of the column as provided by the plugin (usually the field name as represented in the source app). Note that these can change over time as the source data changes, and sometimes these may represent drill-down into nested objects.

Column name templates only ever apply to normalized views. Raw tables always have a single object per inbound record, stored in an object column named RECORD_DATA.

You can use any of the standard Jinja2 filters to modify values. E.g. {{stream_name|upper}} would convert the stream name to uppercase.

In addition to these, we provide the following custom filters:

• fivetran_word_division: Divides alphanumeric strings containing letters and numbers into multiple words and numbers joined with underscores, as described in the Fivetran docs. We include this for naming compatibility for customers migrating from Fivetran.
• multiple_underscores_to_single: Replaces two or more consecutive underscores with a single underscore

• prepend_starting_number_with_underscore: If the text starts with a number, prepend it with an underscore

These can be changed together, e.g. {{column_name|upper|multiple_underscores_to_single}}

Do not include double quotes in these templates - these are added automatically by the sync engine as needed.