📘Release History
V3.3
Created: 2024-12-20
Fixed an issue with the CONFIGURE_OMNATA_INBOUND_SYNC procedure where stream configuration changes may not be applied
V3.2
Created: 2024-12-19
Fixed an issue where an error may occur in the UI when configuring outbound syncs
When normalized view creation fails, the view definition will now be logged at the warning level instead of debug
Added an experimental stored proc to simplify automated inbound sync creation/update
V3.1
Created: 2024-12-17
Added a consumer view for plugin information
V3.0
Created: 2024-12-16
The sync engine will now use Hybrid tables for most of its internal state, whenever the consumer account supports them
V2.40
Created: 2024-12-13
Fixed an issue where editing a connection would error if the connection slug was changed at the same time
V2.39
Created: 2024-12-13
Updated core to use latest plugin runtime
Fixed an issue where timestamp formatting parameters overridden at the account level could interfere with sync runs
Fixed an issue where editing inbound syncs could cause an error
V2.38
Created: 2024-12-11
Connections now store whether or not other environments exist, to assist with branching decisions during sync configuration
Introduced Outbound Target Types, an optional top level selection for outbound syncs. Each target type can have its own set of sync strategies and provides an easier way to present different sync configuration parameters. They can also nominate a target parameter, another new concept described in the docs.
Registered Plugins now store supported connectivity options in the sync engine
Added the Manual sync schedule to the UI
Fixed an issue where an error may be raised during sync creation
Marking inbound tables for truncation actually uses a delete statement, to avoid auto-commiting the transaction
Upgraded streamlit to version 1.35.0
Added logging to assist with normalized view creation problems
V2.37
Created: 2024-12-03
Pinned Streamlit's Python runtime to v3.8 so that we can control the migration to 3.11 in January 2025
Added retries to the plugin cancellation check procedure
V2.36
Created: 2024-12-01
Sync Engine log messages are now automatically linked to the current trace and span in the event table
Fixed an issue where applying settings to an inbound branch would clear the sync state
Fixed an issue where refreshing the data in the controls tab would not reload data for a branch
V2.35
Created: 2024-11-28
Added extra telemetry to sync run processor
Fixed an issue where applying branch settings via dbt raised an error in certain scenarios
Plugins can now choose to load results directly into an object column to avoid json parsing overhead
New field detection now works on a sample of records to improve performance
Improved error handling when syncing
Fixed an issue where a stream without a schema may raise an error when building the normalized view
V2.34
Created: 2024-11-21
Fixed an issue where not all inbound streams were truncated as marked
Added INBOUND_NEXT_RUN_FULL_REFRESH columns to consumer data views
V2.33
Created: 2024-11-18
Fixed an issue with checking API limits on ngrok connections
V2.32
Created: 2024-11-15
Fixed an issue where outbound syncs containing specific column names that overlap with the record validator alias caused an error to occur
V2.31
Created: 2024-11-15
Fixed an issue where consumer UDFs did not have the connectivity option passed in with the connection parameters
V2.30
Created: 2024-11-14
Fixed an issue where manual connection tests may raise an error for ngrok/privatelink connections
V2.29
Created: 2024-11-14
Fixed an issue with ngrok tunnel creation
V2.28
Created: 2024-11-13
Exposed the CREATE_SYNC proc to consumers. This is currently undocumented, you must contact support for more information
Fixed an issue where the ngrok connectivity option fails to render the sync configuration form
V2.27
Created: 2024-11-12
Introduced support for Connectivity Options, which allows plugins to signal support for any combination of direct network access, ngrok or privatelink.
Fixed a UI issue where dependant streams raise an error when requesting the list of columns
V2.26
Created: 2024-11-06
Fixed an issue where the sync runner task did not update the user_task_timeout_ms to reflect the new run time limit
V2.25
Created: 2024-11-02
Fixed an issue where outbound records set to skip_always could appear again as a duplicate identifier
Fixed an issue with the SET_INBOUND_SYNC_STATE procedure where it would error if a sync has never completed
V2.24
Created: 2024-10-31
Plugins can now instruct the sync engine to refresh the schema and recreate the normalized view of inbound sync streams at the end of a run
V2.23
Created: 2024-10-30
Fixed an issue when editing syncs, where the local timezone would be shown instead of the overridden value
V2.22
Created: 2024-10-23
Fixed an issue where branch sync runs would error if a run is already running
V2.21
Created: 2024-10-21
Fixed an issue where the timezone of a sync schedule could not be changed after initial sync creation
V2.20
Created: 2024-10-20
Fixed an issue where branch runs on inbound syncs may error while trying to create tables
Fixed an issue where deleting a sync did not always clean up inbound tables and views that reside in the application database
Added an experimental stored proc for deleting branches
V2.19
Created: 2024-10-19
Fixed an issue where applying branch settings could incorrectly fail due to configuration differences when there were none
If a normalized view column references another column, ensured that they were ordered correctly in the view to accomodate this
Fixed an issue where fetching the inbound view definitions would cause a Streamlit error if there was a problem with the json schema
V2.18
Created: 2024-10-16
Changed the trace level in the application manifest to meet new native application framework requirements
Inbound stream schemas can now include joins to other streams. This will accommodate plugins providing calculated values in normalized views that depend on data from other streams
Provided stored procedures for deleting syncs and setting a sync's schedule
V2.17
Created: 2024-10-01
Fixed an issue where old stream state of deselected streams would cause an error in the Streamlit UI
Upgraded several common python packages to match latest on Snowflake Anaconda channel
Fixed an issue where a 'normalized_column' error was shown on the sync home screen
V2.16
Created: 2024-09-23
When creating the daily admin task, the warehouse name is now quoted
V2.15
Created: 2024-09-22
Fixed an issue where notifications would be dispatched even when no events were triggered (this is in addition to the fix in V2.12)
V2.14
Created: 2024-09-20
When creating Snowflake tasks, the warehouse name is now quoted
Fixed an issue where manual full refreshes of incremental streams did not perform flagging of deleted records
V2.13
Created: 2024-09-20
Fixed an issue where the sync configuration overview would error when comparing some branch values to main
V2.12
Created: 2024-09-20
Fixed an issue where notifications would be dispatched even when no events were triggered
Instead of clearing the inbound sync state of a stream, users can now mark a stream for full refresh at the next run, which has the same effect but is more clear in its intent. When doing this, a choice is made as to whether the table is truncated or not.
If an existing inbound stream has its storage behaviour changed from Append to Merge, it will automatically be marked for full refresh with truncate at the next run.
A full refresh schedule may now be defined independently of the sync schedule, so that incremental syncs can have a full refresh performed occasionally. This is currently only configurable via a stored procedure but will feature more prominently in a future release.
Marking a stream for full refresh will cause any streams it depends to also perform a full refresh
Fixed an issue where sync runs that fail catastrophically do not dispatch a notification when the next run occurs
V2.11
Created: 2024-09-17
Fixed an issue where recreating normalized views for custom sync locations would attempt to create databases that already exist
V2.10
Created: 2024-09-15
Fixed an issue where empty outbound runs may fail to count records during post-processing
Reduced the amount of logging performed at the info level during a sync run
V2.9
Created: 2024-09-15
Fixed an issue where changing only the column template in the inbound storage location does not apply the change
V2.8
Created: 2024-09-14
Fixed an issue where outbound records that failed source validation were still included in the run
V2.7
Created: 2024-09-13
Fixed an issue where switching away from the Snowflake scheduler did not pause the existing task
Syncs can now be paused and resumed via stored proc calls
For multi-select dropdown fields, metadata will be stored for each selected form option, keyed on its value
Fixed an issue where the outbound record viewer did not include the selected individual fields
Fixed an issue where sync runs which were cancelled or exceeded their time limit caused an error
V2.6
Created: 2024-09-06
Logging levels can now be restricted at the package level inside plugins
V2.5
Created: 2024-09-05
Telemetry event sharing preferences are now configured in the application manifest, instead of prompting the user at first launch of Streamlit
V2.4
Created: 2024-09-04
Fixed an issue where filtering outbound records by record state in the UI caused an error
Plugin registration now captures information about consumer functions, in preparation for displaying this information in the UI
Reduced some of the busier log events during syncing to the debug level
V2.3
Created: 2024-08-30
Fixed an issue where development session runs would fail to finalize for outbound syncs
Modified the inbound sync results processing to ensure that deduping is applied in append mode for changes
Users with the OMNATA_ADMINISTRATOR application role can now call the PLUGIN_CONNECTION function directly as a convenience
Fixed an issue where forcing recreation of normalized views from the UI did not have any effect
V2.2
Created: 2024-08-26
Fixed an issue where an error may be thrown when no inbound streams are returned by the plugin
Fixed an issue where modifying an outbound sync branch may result in a SQL compilation error
Fixed an issue where inbound post-processing would throw an error in some situations
Added information to the stream lister to indicate when the Customize option should be used
V2.1
Created: 2024-08-24
Fixed an issue where non-ascii streamlit form parameter values caused an encoding error when passing back to the plugin
V2.0
Created: 2024-08-20
Improved inbound sync post-processing performance
Fixed an issue where creating an ngrok tunnel threw a data validation error
V1.120
Created: 2024-08-18
Reduced the number of post-processing queries when no inbound records are returned for a stream
Moved inbound table DDL to the start of the sync run, to prevent transaction commits while post-processing
Logged errors thrown during stream listing
Added retries for Snowflake throwing "210006: Unknown error" when polling sync job status
Updated to new plugin runtime
For custom inbound storage locations, the database and schema will be created if missing
Allow billing tasks warehouse to be selected again, regardless of the state of the billing task
V1.119
Created: 2024-08-12
Added a stored procedure which can delete staged outbound records by apply state. This can be used under the direction of support in the unlikely event that duplicate identifiers make it into the internal record state table.
V1.118
Created: 2024-08-09
Provided two stored procedures:
SET_SYNC_NAME to update a sync name after creation
SET_INBOUND_SYNC_STATE to manually override the state of an inbound sync stream Both of these actions will eventually be available in the UI
V1.117
Created: 2024-08-07
Testing compatibility of plugins migrating to Pydantic v2 ahead of sync engine
Migrated backend and streamlit to Pydantic v2
Fixed an issue where branches failed to reopen correctly after a recent product upgrade
Fixed an issue where outbound record duplicate detection was not performed for some sync strategies
Fixed an issue where editing connections with keypair fields may result in an error
V1.116
Created: 2024-08-01
Inbound storage configuration now allows the column name of normalized views to be specified as a template. This allows for column names formats to be standardised, regardless of the source application.
V1.115
Created: 2024-07-31
Schema for normalized views can now contain Snowflake column expressions, for advanced parsing
V1.114
Created: 2024-07-25
Allowed plugin development sessions to have their sync run closed off without providing results
Outbound syncs now upload result objects as JSON, to avoid issues with the arrow serializer. This is enabled in plugin runtime 0.4.10 and newer
V1.113
Created: 2024-07-24
Fixed an issue where an error was displayed on the outbound records tab after record updates occurred
V1.112
Created: 2024-07-23
Fixed an issue where the hover text on the Syncs list did not show the name of the connection
Provided a process to choose the admin tasks warehouse again after initial onboarding
V1.111
Created: 2024-07-23
Added additional checks to ensure no ngrok API errors include API tokens
Modified the SYNC and SYNC_BRANCH data views to include more configuration information
V1.110
Created: 2024-07-21
Fixed an issue where Snowflake account names containing underscores would break the ngrok tunnel configuration process
Adding multiple ngrok connections will now reuse the same auth token to simplify agent configuration
V1.109
Created: 2024-07-19
When changing the outbound sync strategy during initial configuration, the form definition is reloaded in case there are variations between strategies
Outbound syncs will now have access to the previous state of an updated record, in case there are situations where the plugin needs to know exactly what changed between syncs
Errors during stream listing are now handled gracefully and can be retried
V1.108
Created: 2024-07-15
Syncs can no longer be deleted if other sync's schedules depend on them
Fixed a Streamlit error that can occur when rendering outbound records table
Changed the behaviour of inbound syncs, so that full refresh+append streams can have previously deleted records restored back to an undeleted state
Errors that occur while populating form data sources from plugins, are now caught and displayed within the form boundary
Fixed some Streamlit dependencies who's pinned versions had been removed from Anaconda
V1.107
Created: 2024-07-10
Improved handling of transient transaction-related errors when syncing
V1.106
Created: 2024-07-09
Changed inbound stream lister to ignore source defined primary keys if it's set to an empty list
Added a button to refresh the fields list for primary key/cursor selection
When saving inbound configuration, store any newer json schemas in case the sync is edited again before the next sync
V1.105
Created: 2024-07-08
Modified PLUGIN_CONNECTION to only return connection metadata for completed connections. This ensures that consumer UDFS do not try to use partially completed connections
V1.104
Created: 2024-07-08
Fixed an issue where quoted columns in outbound syncs caused an error during record staging
V1.103
Created: 2024-07-05
Add retries to the plugin message receiver on the Sync Engine
Modified stream configuration UI to always require a primary key, even with the Append storage behaviour
V1.102
Created: 2024-07-03
Fixed an issue where inbound syncs were able to be created without all streams having a ID Field chosen
V1.101
Created: 2024-07-03
Modified the function which fetchings connection information for consumer functions, to make it more compatible with UDTFs
For inbound syncs, reduced the amount of concurrent post-processing work to reduce transaction errors
Fixed an issue where viewing inbound sync records in the UI caused an error
V1.100
Created: 2024-07-02
Fixed an issue with the outbound field mapper where removal of a column in the source table which was already mapped to an optional column yielded an error
Trace events in Streamlit are now logged asyncronously
V1.99
Created: 2024-07-01
Fixed an issue where outbound syncs using the Create strategy, don't retry after initial failures using the latest record value
V1.98
Created: 2024-06-29
Fixed an issue where parsing a list of tables in some environments would raise an error
V1.97
Created: 2024-06-28
Fixed an issue where inbound storage locations that are already pre-quoted would be quoted again
Fixed an issue where errors streams may not display in the progress accordion of the sync run detail page
Added a "check again" button to the outound source table selector, to re-check for duplicate ID column values
Fixed an issue for the inbound sync run view, where some in-progress streams were not visible
Handled the scenario where the inbound record count exceeds the initial estimate
V1.96
Created: 2024-06-24
Fixed an issue where expired plugin trials could break the plugin selector
V1.95
Created: 2024-06-24
Plugins reporting total record estimates for streams now show a progress bar in the UI
Fixed an issue for inbound syncs, where default inbound storage locations would fail to delete tables when the sync is deleted
Using async stored procs has resulted in occasional transaction-related errors that need retrying. We now retry and eventually ignore these during the plugin execution, but raise an error if it's still occurring at the final commit
Fixed an issue where inbound streams may show as both "completed" and "successful"
V1.94
Created: 2024-06-23
Fixed issue where outbound staging duplicate detection is not isolated by sync branch
Fixed an inbound sync error affecting the most recent plugin runtime
New async results processing now only occurs on plugin runtime 0.4.2, where results are guaranteed to be uploaded before stream status updates
V1.93
Created: 2024-06-22
Inbound syncs now have their deadline and cancellation monitored by the sync engine (thanks to the new async query support)
Post-processing of completed streams now happens during the sync run, so that cancelled/abandoned syncs save progress more reliably
Plugins can now report a total count of records ahead of time
Update inline product help
Fixed an issue where editing existing syncs could result in the inbound storage location being reset to the default
Outbound record staging now detects duplicate identifiers prior to the record state being updated. This reduces the need for manual record cleanup if you accidentally introduce duplicates in your source table
V1.92
Created: 2024-06-18
Added a workaround for a streamlit bug which affects the stream lister
V1.91
Created: 2024-06-17
Removed some excess logging
Fixed an issue introduced by the previous release
V1.90
Created: 2024-06-17
Inbound storage location can now include the type of table to use: standard (with or without change tracking) or hybrid
Changing the storage location copies any existing data to the new location, and also removes the old table
V1.89
Created: 2024-06-11
Fixed an issue where ngrok connection edits did not check for pending API configuration changes before proceeding to the final connection form
V1.88
Created: 2024-06-08
Switched Streamlit record selectors from experimental_data_editor to data_editor
V1.87
Created: 2024-06-08
Added logging to the stream lister component to determine whether users have deliberately toggled a stream
Changed the way the app artifact is added to the system path to improve Streamlit cycle times
Reduced the size of the packaged codebase (both streamlit and stored libraries)
V1.86
Created: 2024-06-07
Fixed an issue where an abandoned notification recipient configuration causes a persistent error on the settings page
V1.85
Created: 2024-06-02
UI now uses Streamlit 1.26.0. Inbound stream lister now uses toggles instead of checkboxes
V1.84
Created: 2024-06-01
Updated inline documentation for inbound syncs
Plugin registrations can now be deleted from the sync engine UI following the uninstall of a plugin application
V1.83
Created: 2024-05-24
A default custom storage location for inbound syncs can now be configured globally
V1.82
Created: 2024-05-21
Changed the behaviour of dynamic-valued drop-downs so that the first option is not auto-selected. This prevents long initial waits for forms that progressively load options from a remote source
Provided ability for users to bring their own GPG/SSH keys during connection creation
When prompting the user to deploy the ngrok agent, the address and port are now prompted for, and the specific cli command or service configuration file are provided
V1.81
Created: 2024-05-17
Snowflake connector/snowpark logging level is now set to WARN by default, to reduce noise in event tables. It can be increased for individual customers via a shared table parameter, up to the level configured for the native app
Any custom snowflake logging level is applied to plugin apps using runtime 0.3.25 or newer
Fixed issue where stream configuration screen could error when editing a sync
Added monthly billing events for ngrok tunnels
V1.79
Created: 2024-05-06
Added logging for sync request payloads
V1.78
Created: 2024-05-02
For inbound streams, improved handling of default primary key fields when they are composite
V1.77
Created: 2024-05-02
Slider fields now have their step size correctly configured to match what was set by the plugin
Added support for composite id fields on inbound streams
V1.76
Created: 2024-04-27
Fixed error that can occur when selecting a cursor field for an inbound sync
Connections using ngrok now have the agent launch command available when you view the connection details
V1.75
Created: 2024-04-23
Fixed an issue where id-only deletions in run results resulted in an empy record rather retaining the previous value of the record
Added cursor field and primary key field to the sync home page summary
V1.74
Created: 2024-04-22
Disabled client side encryption on plugin results stage, so that results can be streamed
If plugin application has been uninstalled, show a warning message when creating a new connection
V1.73
Created: 2024-04-17
Plugins are now provided with an internal stage to upload run results to
V1.72
Created: 2024-04-17
Fixed an error where ngrok resources are not cleaned up when connections are deleted
Fixed error that may occur when editing connections with ngrok tunnels
Removed the requirement that plugins return a list of errored streams from the SYNC_INBOUND operation. This simplifies deferring the operation to other languages like java.
V1.71
Created: 2024-04-10
ngrok tunnel limits can now be increased centrally by Omnata
V1.70
Created: 2024-04-10
Fixed an issue where an error occurs if an inbound stream does not report state before ending
Added support for ad-hoc consumer-facing functions that can be shipped with plugins and invoked using a connection slug
Fixed an issue where an error can occur when customizing streams
Fixed an issue where ngrok-based connections may display an error during initial creation
V1.69
Created: 2024-04-03
If the native app doesn't own an inbound raw table, it will no longer error on configuration save when attempting to run grant statements
V1.68
Created: 2024-03-28
When hovering on the help text for a custom cron schedule, a human readable description is now shown instead of the cron expression
All streamlit dependencies are now pinned to specific package versions, to ensure Anaconda availability
V1.67
Created: 2024-03-27
Fixed an issue where outbound sync pre-flight checks would fail due to source table not being quoted correctly
Provided a fallback configuration option for new streams when the plugin runtime is less than 0.3.19
Provided a maintenance process which deletes Sync Record State History older than a configurable number of days, to reduce storage costs at high record volume
Fixed an issue where outbound syncs could not have record transformation applied due to table names not being quoted
Created all Sequences with the ORDER attribute set, to ensure identifiers increment
Included the application's own database in the list of outbound source databases
Reverted the UI back to Streamlit 1.22.0
V1.66
Created: 2024-03-25
Fixed an issue where results deduplication did not work correctly for inbound syncs
Fixed an issue where the append storage behaviour would create duplicate records
V1.65
Created: 2024-03-25
During post run processing, added some exception logging logic so that transaction rollback errors don't cause the original error to be lost
V1.64
Created: 2024-03-22
On the Syncs list, custom schedules now display the cron string without needing to hover on the help text
Inbound syncs no longer offer a choice of Incremental or Full Refresh unless you choose the Customize option. Instead, Auto mode will choose incremental where possible, and fall back on full refresh.
Storage behaviour is still a choice, but it is now "Merge changes" or "Keep history" in the UI and an explanatory diagram is included.
For full refreshes, the term "Replace" has been deprecated in favour of "Merge"
Upgraded the UI to Streamlit 1.26.0
Fixed streamlit warning on pre-populated checkboxes on sync configuration forms
V1.63
Created: 2024-03-21
Syncs list now has a column for Sync Schedule
Fixed issue where multi-select dropdown with required value allowed the user to continue without any values selected
Added some extra columns to the SYNC_RUN view, relating to inbound record counts
Fix issue where deletion by criteria with the append storage behaviour could append multiple delete records for the same identifier
Added a button on the inbound sync run screen to refresh the list of streams
V1.62
Created: 2024-03-19
The total number of inbound records for a run is now logged pre and post deduplication. A warning is logged if deduplication removed records.
Fixed an issue with the new deletion functionality, where the new sync engine version was not compatible with older runtime versions
Fixed an issue where non-empty inbound runs were hidden by the "Hide empty completed runs" checkbox
V1.61
Created: 2024-03-17
After clearing state for an inbound sync, the table is truncated when applying the results for the next run
V1.60
Created: 2024-03-15
The "Append" inbound storage behaviour now only stores changed records, making it much more storage efficient when used with the Full Refresh strategy.
Inbound sync tables now contain a flag which indicates when the record has been deleted at the source. When using the Full Refresh strategy, this is maintained automatically via the absence of previous records.
Inbound sync tables now contain the run id that the most recent change for each record belongs to.
Plugins can now provide individual record deletions, and also deletions by criteria for inbound incremental syncs. This requires omnata-plugin-runtime version 0.3.15 or greater.
When processing results of inbound syncs at the end of the run, the engine will commit the data and latest state of each stream individually, so that if errors are encountered then progress is not lost.
When a sync runs for longer than its configured limit, it is assumed failed and can be re-ran. The previous run will be marked as failed when this occurs.
Sync run id is now included in event span attributes, to make it easier to filter event table queries
When comparing branch stream configurations, the stream state is no longer included in the comparison.
Fixed issue with Sync History tab where incomplete outbound syncs did not render correctly
Inbound stream cursor field now has the default value applied to it at initial sync creation, so that plugins don't need to check both values
Fully deprecated the internal Sync Run Log table in favour of event tables
V1.59
Created: 2024-03-07
Added the PLUGIN_UPDATE_API_CONFIGURATION stored procedure, which can reset the integrations and secret bindings for plugin udfs/procedures
Fixed Streamlit error when switching between full and incremental inbound sync strategy for a single stream
Provided a button in the Advanced area of Settings which re-binds all plugin integration and secret objects
Ensure default cursor field is applied for incremental syncs
Fixed issue where Streamlit could not show information about an SSL certificate because the pyopenssl library was no longer available
V1.58
Created: 2024-03-06
Fixed streamlit UI error on sync home screen when production connection is not yet configured for inbound syncs in branch mode
Fixed issue when nominating a production connection for inbound syncs
When presenting dbt model configuration for streams, handled the scenario where JSON schema is not available
Inbound stream schemas are now fetched when syncs or branches are created or updated, so that the schema is visible in the UI prior to the first run
V1.57
Created: 2024-03-06
In normalized views, object fields are now supported as columns
V1.56
Created: 2024-03-05
Fixed issue where refreshing inbound stream schemas manually resulted in an error
V1.55
Created: 2024-03-03
This is the first version to contain release notes.
Changes in this version:
Reset page number when stream search changes
Killing sync runs no longer attempts to kill the Snowflake query
Added a refresh button to the Syncs page
Inbound sync history is now paginated
Added a refresh button to the Connections page
Fixed error when recreating views for inbound syncs
Removed sync engine billing from daily billing run in preparation for switch to plugin-only billing
Added consumer facing views of internal app state
Last updated