📘Release History


Created: 2024-07-19

  • When changing the outbound sync strategy during initial configuration, the form definition is reloaded in case there are variations between strategies

  • Outbound syncs will now have access to the previous state of an updated record, in case there are situations where the plugin needs to know exactly what changed between syncs

  • Errors during stream listing are now handled gracefully and can be retried


Created: 2024-07-15

  • Syncs can no longer be deleted if other sync's schedules depend on them

  • Fixed a Streamlit error that can occur when rendering outbound records table

  • Changed the behaviour of inbound syncs, so that full refresh+append streams can have previously deleted records restored back to an undeleted state

  • Errors that occur while populating form data sources from plugins, are now caught and displayed within the form boundary

  • Fixed some Streamlit dependencies who's pinned versions had been removed from Anaconda


Created: 2024-07-10

  • Improved handling of transient transaction-related errors when syncing


Created: 2024-07-09

  • Changed inbound stream lister to ignore source defined primary keys if it's set to an empty list

  • Added a button to refresh the fields list for primary key/cursor selection

  • When saving inbound configuration, store any newer json schemas in case the sync is edited again before the next sync


Created: 2024-07-08

  • Modified PLUGIN_CONNECTION to only return connection metadata for completed connections. This ensures that consumer UDFS do not try to use partially completed connections


Created: 2024-07-08

  • Fixed an issue where quoted columns in outbound syncs caused an error during record staging


Created: 2024-07-05

  • Add retries to the plugin message receiver on the Sync Engine

  • Modified stream configuration UI to always require a primary key, even with the Append storage behaviour


Created: 2024-07-03

  • Fixed an issue where inbound syncs were able to be created without all streams having a ID Field chosen


Created: 2024-07-03

  • Modified the function which fetchings connection information for consumer functions, to make it more compatible with UDTFs

  • For inbound syncs, reduced the amount of concurrent post-processing work to reduce transaction errors

  • Fixed an issue where viewing inbound sync records in the UI caused an error


Created: 2024-07-02

  • Fixed an issue with the outbound field mapper where removal of a column in the source table which was already mapped to an optional column yielded an error

  • Trace events in Streamlit are now logged asyncronously


Created: 2024-07-01

  • Fixed an issue where outbound syncs using the Create strategy, don't retry after initial failures using the latest record value


Created: 2024-06-29

  • Fixed an issue where parsing a list of tables in some environments would raise an error


Created: 2024-06-28

  • Fixed an issue where inbound storage locations that are already pre-quoted would be quoted again

  • Fixed an issue where errors streams may not display in the progress accordion of the sync run detail page

  • Added a "check again" button to the outound source table selector, to re-check for duplicate ID column values

  • Fixed an issue for the inbound sync run view, where some in-progress streams were not visible

  • Handled the scenario where the inbound record count exceeds the initial estimate


Created: 2024-06-24

  • Fixed an issue where expired plugin trials could break the plugin selector


Created: 2024-06-24

  • Plugins reporting total record estimates for streams now show a progress bar in the UI

  • Fixed an issue for inbound syncs, where default inbound storage locations would fail to delete tables when the sync is deleted

  • Using async stored procs has resulted in occasional transaction-related errors that need retrying. We now retry and eventually ignore these during the plugin execution, but raise an error if it's still occurring at the final commit

  • Fixed an issue where inbound streams may show as both "completed" and "successful"


Created: 2024-06-23

  • Fixed issue where outbound staging duplicate detection is not isolated by sync branch

  • Fixed an inbound sync error affecting the most recent plugin runtime

  • New async results processing now only occurs on plugin runtime 0.4.2, where results are guaranteed to be uploaded before stream status updates


Created: 2024-06-22

  • Inbound syncs now have their deadline and cancellation monitored by the sync engine (thanks to the new async query support)

  • Post-processing of completed streams now happens during the sync run, so that cancelled/abandoned syncs save progress more reliably

  • Plugins can now report a total count of records ahead of time

  • Update inline product help

  • Fixed an issue where editing existing syncs could result in the inbound storage location being reset to the default

  • Outbound record staging now detects duplicate identifiers prior to the record state being updated. This reduces the need for manual record cleanup if you accidentally introduce duplicates in your source table


Created: 2024-06-18

  • Added a workaround for a streamlit bug which affects the stream lister


Created: 2024-06-17

  • Removed some excess logging

  • Fixed an issue introduced by the previous release


Created: 2024-06-17

  • Inbound storage location can now include the type of table to use: standard (with or without change tracking) or hybrid

  • Changing the storage location copies any existing data to the new location, and also removes the old table


Created: 2024-06-11

  • Fixed an issue where ngrok connection edits did not check for pending API configuration changes before proceeding to the final connection form


Created: 2024-06-08

  • Switched Streamlit record selectors from experimental_data_editor to data_editor


Created: 2024-06-08

  • Added logging to the stream lister component to determine whether users have deliberately toggled a stream

  • Changed the way the app artifact is added to the system path to improve Streamlit cycle times

  • Reduced the size of the packaged codebase (both streamlit and stored libraries)


Created: 2024-06-07

  • Fixed an issue where an abandoned notification recipient configuration causes a persistent error on the settings page


Created: 2024-06-02

  • UI now uses Streamlit 1.26.0. Inbound stream lister now uses toggles instead of checkboxes


Created: 2024-06-01

  • Updated inline documentation for inbound syncs

  • Plugin registrations can now be deleted from the sync engine UI following the uninstall of a plugin application


Created: 2024-05-24

  • A default custom storage location for inbound syncs can now be configured globally


Created: 2024-05-21

  • Changed the behaviour of dynamic-valued drop-downs so that the first option is not auto-selected. This prevents long initial waits for forms that progressively load options from a remote source

  • Provided ability for users to bring their own GPG/SSH keys during connection creation

  • When prompting the user to deploy the ngrok agent, the address and port are now prompted for, and the specific cli command or service configuration file are provided


Created: 2024-05-17

  • Snowflake connector/snowpark logging level is now set to WARN by default, to reduce noise in event tables. It can be increased for individual customers via a shared table parameter, up to the level configured for the native app

  • Any custom snowflake logging level is applied to plugin apps using runtime 0.3.25 or newer

  • Fixed issue where stream configuration screen could error when editing a sync

  • Added monthly billing events for ngrok tunnels


Created: 2024-05-06

  • Added logging for sync request payloads


Created: 2024-05-02

  • For inbound streams, improved handling of default primary key fields when they are composite


Created: 2024-05-02

  • Slider fields now have their step size correctly configured to match what was set by the plugin

  • Added support for composite id fields on inbound streams


Created: 2024-04-27

  • Fixed error that can occur when selecting a cursor field for an inbound sync

  • Connections using ngrok now have the agent launch command available when you view the connection details


Created: 2024-04-23

  • Fixed an issue where id-only deletions in run results resulted in an empy record rather retaining the previous value of the record

  • Added cursor field and primary key field to the sync home page summary


Created: 2024-04-22

  • Disabled client side encryption on plugin results stage, so that results can be streamed

  • If plugin application has been uninstalled, show a warning message when creating a new connection


Created: 2024-04-17

  • Plugins are now provided with an internal stage to upload run results to


Created: 2024-04-17

  • Fixed an error where ngrok resources are not cleaned up when connections are deleted

  • Fixed error that may occur when editing connections with ngrok tunnels

  • Removed the requirement that plugins return a list of errored streams from the SYNC_INBOUND operation. This simplifies deferring the operation to other languages like java.


Created: 2024-04-10

  • ngrok tunnel limits can now be increased centrally by Omnata


Created: 2024-04-10

  • Fixed an issue where an error occurs if an inbound stream does not report state before ending

  • Added support for ad-hoc consumer-facing functions that can be shipped with plugins and invoked using a connection slug

  • Fixed an issue where an error can occur when customizing streams

  • Fixed an issue where ngrok-based connections may display an error during initial creation


Created: 2024-04-03

  • If the native app doesn't own an inbound raw table, it will no longer error on configuration save when attempting to run grant statements


Created: 2024-03-28

  • When hovering on the help text for a custom cron schedule, a human readable description is now shown instead of the cron expression

  • All streamlit dependencies are now pinned to specific package versions, to ensure Anaconda availability


Created: 2024-03-27

  • Fixed an issue where outbound sync pre-flight checks would fail due to source table not being quoted correctly

  • Provided a fallback configuration option for new streams when the plugin runtime is less than 0.3.19

  • Provided a maintenance process which deletes Sync Record State History older than a configurable number of days, to reduce storage costs at high record volume

  • Fixed an issue where outbound syncs could not have record transformation applied due to table names not being quoted

  • Created all Sequences with the ORDER attribute set, to ensure identifiers increment

  • Included the application's own database in the list of outbound source databases

  • Reverted the UI back to Streamlit 1.22.0


Created: 2024-03-25

  • Fixed an issue where results deduplication did not work correctly for inbound syncs

  • Fixed an issue where the append storage behaviour would create duplicate records


Created: 2024-03-25

  • During post run processing, added some exception logging logic so that transaction rollback errors don't cause the original error to be lost


Created: 2024-03-22

  • On the Syncs list, custom schedules now display the cron string without needing to hover on the help text

  • Inbound syncs no longer offer a choice of Incremental or Full Refresh unless you choose the Customize option. Instead, Auto mode will choose incremental where possible, and fall back on full refresh.

  • Storage behaviour is still a choice, but it is now "Merge changes" or "Keep history" in the UI and an explanatory diagram is included.

  • For full refreshes, the term "Replace" has been deprecated in favour of "Merge"

  • Upgraded the UI to Streamlit 1.26.0

  • Fixed streamlit warning on pre-populated checkboxes on sync configuration forms


Created: 2024-03-21

  • Syncs list now has a column for Sync Schedule

  • Fixed issue where multi-select dropdown with required value allowed the user to continue without any values selected

  • Added some extra columns to the SYNC_RUN view, relating to inbound record counts

  • Fix issue where deletion by criteria with the append storage behaviour could append multiple delete records for the same identifier

  • Added a button on the inbound sync run screen to refresh the list of streams


Created: 2024-03-19

  • The total number of inbound records for a run is now logged pre and post deduplication. A warning is logged if deduplication removed records.

  • Fixed an issue with the new deletion functionality, where the new sync engine version was not compatible with older runtime versions

  • Fixed an issue where non-empty inbound runs were hidden by the "Hide empty completed runs" checkbox


Created: 2024-03-17

  • After clearing state for an inbound sync, the table is truncated when applying the results for the next run


Created: 2024-03-15

  • The "Append" inbound storage behaviour now only stores changed records, making it much more storage efficient when used with the Full Refresh strategy.

  • Inbound sync tables now contain a flag which indicates when the record has been deleted at the source. When using the Full Refresh strategy, this is maintained automatically via the absence of previous records.

  • Inbound sync tables now contain the run id that the most recent change for each record belongs to.

  • Plugins can now provide individual record deletions, and also deletions by criteria for inbound incremental syncs. This requires omnata-plugin-runtime version 0.3.15 or greater.

  • When processing results of inbound syncs at the end of the run, the engine will commit the data and latest state of each stream individually, so that if errors are encountered then progress is not lost.

  • When a sync runs for longer than its configured limit, it is assumed failed and can be re-ran. The previous run will be marked as failed when this occurs.

  • Sync run id is now included in event span attributes, to make it easier to filter event table queries

  • When comparing branch stream configurations, the stream state is no longer included in the comparison.

  • Fixed issue with Sync History tab where incomplete outbound syncs did not render correctly

  • Inbound stream cursor field now has the default value applied to it at initial sync creation, so that plugins don't need to check both values

  • Fully deprecated the internal Sync Run Log table in favour of event tables


Created: 2024-03-07

  • Added the PLUGIN_UPDATE_API_CONFIGURATION stored procedure, which can reset the integrations and secret bindings for plugin udfs/procedures

  • Fixed Streamlit error when switching between full and incremental inbound sync strategy for a single stream

  • Provided a button in the Advanced area of Settings which re-binds all plugin integration and secret objects

  • Ensure default cursor field is applied for incremental syncs

  • Fixed issue where Streamlit could not show information about an SSL certificate because the pyopenssl library was no longer available


Created: 2024-03-06

  • Fixed streamlit UI error on sync home screen when production connection is not yet configured for inbound syncs in branch mode

  • Fixed issue when nominating a production connection for inbound syncs

  • When presenting dbt model configuration for streams, handled the scenario where JSON schema is not available

  • Inbound stream schemas are now fetched when syncs or branches are created or updated, so that the schema is visible in the UI prior to the first run


Created: 2024-03-06

In normalized views, object fields are now supported as columns


Created: 2024-03-05

Fixed issue where refreshing inbound stream schemas manually resulted in an error


Created: 2024-03-03

This is the first version to contain release notes.

Changes in this version:

  • Reset page number when stream search changes

  • Killing sync runs no longer attempts to kill the Snowflake query

  • Added a refresh button to the Syncs page

  • Inbound sync history is now paginated

  • Added a refresh button to the Connections page

  • Fixed error when recreating views for inbound syncs

  • Removed sync engine billing from daily billing run in preparation for switch to plugin-only billing

  • Added consumer facing views of internal app state

Last updated