# Release History *** ### V5.28 #### Created: 2026-04-14 * Fixed an issue where normalized views with CTEs may fail due to ambiguous column names * Fixed display of source tables in multi-account syncs * Added default case for start sync label button * Recommendations tile for notifications now leads directly to notifications tab *** ### V5.27 #### Created: 2026-04-14 * Fixed an issue where some sync engine upgrades would fail after "create managed task" privileges were revoked *** ### V5.26 #### Created: 2026-04-05 * Added an extra second to the beginning of the results table time travel queries due to ongoing intermittent "Time travel data is not available" errors *** ### V5.25 #### Created: 2026-04-05 * Fixed various issues relating to stored proc parameter generation when editing syncs and branches. There were several cases where syncs/branches could not be saved due to invalid/incomplete object keying for multi-env/branching scenarios * Adjusted top padding to allow for changes in recent Streamlit versions * Change the icon size in home page * Fixed an issue where changing the sync name could affect sync editing * Added branch creation option for Single-Environment syncs * Automatically enabled branching for existing syncs with production connections * Fixed an issue where outbound target types may retain values from other plugins * When editing an existing sync, the choice of connections is now limited to the same type (production or non-production) * In multi-account outbound syncs, the source tables for other accounts are now clearly marked as disabled until the main source table is completed, to avoid confusion * Fixed an issue with case sensitivity of account names in multi-account syncs * Fixed an issue where not supplying an inbound storage location when invoking CONFIGURE\_OMNATA\_INBOUND\_SYNC would revert it to the default *** ### V5.24 #### Created: 2026-04-01 * Fixed behaviour of "Automatically add new objects to the sync". It now re-renders correctly upon toggle, and excluded streams are populated after being fetched *** ### V5.23 #### Created: 2026-04-01 * Fixed an issue where the configuration diff view did not scroll correctly * Remove duplicate label when deleting a branch * Fixed the button label and helper text at the end of the sync configuration wizard to correctly reflect the operation being performed *** ### V5.22 #### Created: 2026-03-27 * Fixed an issue where inbound single-account syncs with branching generated an invalid stored procedure call *** ### V5.21 #### Created: 2026-03-27 * Fixed an issue where "Re-sync all next run" button was not correctly branch-scoped * Fixed an issue where excluded streams were not correctly passed into configuration stored proc *** ### V5.20 #### Created: 2026-03-26 * Inbound views with joined streams containing self-references now leverage subqueries to avoid invalid view definitions *** ### V5.19 #### Created: 2026-03-25 * Added a guard against setting invalid inbound storage locations in single account branching mode * Outbound dbt syncs now correctly use the branch\_name jinja variable *** ### V5.18 #### Created: 2026-03-25 * Fixed an issue where previously created dbt syncs did not display single account branching parameters correctly on the main sync *** ### V5.17 #### Created: 2026-03-25 * Fixed issues relating to stored procedure generation during sync configuration *** ### V5.16 #### Created: 2026-03-24 * Sync configuration rollback will now use the diff view to show the effective changes * The SYNC\_CONFIGURATION\_ROLLBACK proc can now be invoked directly if needed *** ### V5.15 #### Created: 2026-03-23 * Fixed an issue where the inbound stream view name on the config tab would incorrectly resolve sync\_branch as "None" instead of "main" for the main sync *** ### V5.14 #### Created: 2026-03-23 * Fix internal clean-up process when very large numbers of tables exist. * Fixed a regression in the previous release where already-selected streams were deselected when using the "Show only selected" toggle *** ### V5.13 #### Created: 2026-03-22 * Fixed an issue with the previous release relating to applying criteria deletes * Fixed an issue where newly deleted streams were not automatically removed from the sync during editing *** ### V5.12 #### Created: 2026-03-21 * Fixed an issue where deleting a branch would incorrectly clear the outbound sync record state for the whole sync instead of just the branch * Fixed an issue where checkpointing may miss records if the background processor task takes too long to start * Fixed an issue where the new sync configuration stored procs may not create a Snowflake task correctly for some deployment options *** ### V5.11 #### Created: 2026-03-20 * Sync level storage location changes will now advise the user that the data will be moved * Fixed an issue where existing syncs containing slugs with underscores could not be edited *** ### V5.10 #### Created: 2026-03-19 * When selecting primary key fields for inbound syncs, the fields will be removed from the drop-down as they are selected to be part of the key * Fixed an issue where editing a sync caused the storage location to revert to the account default *** ### V5.9 #### Created: 2026-03-18 * Fixed an error that may occur when saving a branch edit * Branch dialog popup will no longer be shown if a branch already exists *** ### V5.8 #### Created: 2026-03-18 * Fixed a Pydantic validation error during outbound sync configuration *** ### V5.7 #### Created: 2026-03-17 * Added a message to reinforce when there are no optional outbound mappings * Added the sync/branch name when deleting a sync * Fixed an issue where sync configuration would error for outbound syncs without field mappings defined *** ### V5.6 #### Created: 2026-03-17 * The Edit Sync button menu now matches the appearance of other menus * Fixed an issue where duplicate select-box option labels interfered with selection * Fixed a SQL parameters error that may occur when copying config between branches * Added the Snowflake account name and sync ID to the email notification template *** ### V5.5 #### Created: 2026-03-16 * Fixed an issue where editing inbound syncs did not allow progression past stream selection without modifying the selection *** ### V5.4 #### Created: 2026-03-13 * Fixed an issue where unnecessary fields were included in the stored proc call to configure inbound syncs (#460) * Fixed an issue where the excluded\_streams parameter was not cast to an array *** ### V5.3 #### Created: 2026-03-12 * Fixed an issue where the new inbound sync configuration stored proc calls did not contain bulk stream configuration options * Fixed an issue where changing plugins in the initial sync configuration left residual settings * Auto-mapper will now preserve existing field mappings if someone re-opens the mapper * Fixed an issue where the automatic outbound field mapper would not include field metadata * Fixed an issue where the inbound storage location did not display the correct radio choice when editing existing syncs *** ### V5.2 #### Created: 2026-03-07 * Reverted APPLY\_INBOUND\_SYNC\_SETTINGS changes for compatibility with existing dbt package *** ### V5.1 #### Created: 2026-03-06 * Inbound streams using the full refresh strategy will now be truncated according to any configured full refresh schedule * Fixed an issue where plugins were incorrectly shown as lacking application roles * Marked existing production connections as having other environments, to accomodate branching changes * Fixed an issue where existing inbound syncs did not offer branch creation *** ### V5.0 #### Created: 2026-03-06 * Introduced official support for multiple Snowflake accounts. The configuration for other accounts can be authored centrally and then distributed via a stored procedure call. * Introduced two new stored procedures: CONFIGURE\_OMNATA\_INBOUND\_SYNC and CONFIGURE\_OMNATA\_OUTBOUND\_SYNC. These stored procedures are more suited to external orchestration than their predecessors and can create or update a sync using the same call pattern. The sync configuration process provides SQL text for these calls. * Deprecated the dbt-omnata-sync dbt package in favour of plain SQL statements in model files. * The rigid branching lifecycle has been removed. Configuration can be copied between the main sync and branches without requiring a ‘merge’ operation. * Introduced a "branch owner" and "sync owner" concept for dbt syncs, so that multiple concurrent developers don't interfere with each other's sync configuration. * Auto-mapped outbound syncs will have a field mapping generated to capture the original order of the columns. This will allow plugins to leverage this ordering during syncing. *** ### V4.34 #### Created: 2026-03-02 * Outbound records that fail to delete will be retried * Added logging for dbt scheduler use, so that we can assist current users with migration to new process *** ### V4.33 #### Created: 2026-02-24 * Outbound records that fail to delete, should be retried *** ### V4.32 #### Created: 2026-02-17 * Tuning tab layout changes to simplify and improve readability * Fixed an issue where configured tuning parameters did not display correctly *** ### V4.31 #### Created: 2026-02-14 * Fixed an issue where excess column pruning would occur when the same stream was joined to multiple times with different aliases *** ### V4.30 #### Created: 2026-02-12 * Added stack trace to JSON-schema pruning warnings *** ### V4.29 #### Created: 2026-02-07 * Field mapping metadata will now update from the plugin when editing outbound syncs *** ### V4.28 #### Created: 2026-02-05 * Fixed an issue where inbound syncs error due to missing full refresh flags *** ### V4.27 #### Created: 2026-02-05 * Fixed an issue completing ngrok connection edit/creation following Python 3.11 upgrade *** ### V4.26 #### Created: 2026-02-04 * Removed the legacy Streamlit UI * JSON schema pruning warnings during sync runs will now include stack traces * All backend APIs now use Python 3.11 instead of 3.10, bumped python packages to latest Anaconda versions * Debounce text field updates in FlexibleTextField *** ### V4.25 #### Created: 2026-02-02 * The main streamlit UI will clear the session storage at startup, to ensure browser tab cloning does not interfere with action handler processing (#412) * Fixed an issue where auto-fetching of new stream schema did not work with dependant streams (#413) *** ### V4.24 #### Created: 2026-01-24 * Fixed an issue where criteria deletes did not apply if there were no records changes for the stream * Sync runs now start with previous state set as the ending state, to prevent missing streams from losing their state * Reorganised background processing for inbound syncs, to minimise locking on the current SYNC\_RUN object * Fixed an issue where the manual full refresh flag was not cleared during checkpointing unless the truncate option was selected *** ### V4.23 #### Created: 2026-01-21 * Fixed an issue where newly discovered streams were not included in the sync because they lacked sync strategy and storage settings * Fixed an issue where stream primary key values were not updated during editing * Fixed up various descriptions and helper texts *** ### V4.22 #### Created: 2026-01-20 * Improved the new UI experience by handling query interruptions, running a warehouse ping every minute, and reducing dynamic python imports * Fixed an issue where changing custom storage location on sync level did not refresh the current page *** ### V4.21 #### Created: 2026-01-20 * Added a new stored procedure for forcefully marking outbound records as successful *** ### V4.20 #### Created: 2026-01-19 * Fixed an issue with the previous release relating to criteria deletes and checkpointing *** ### V4.19 #### Created: 2026-01-19 * MARK\_RECORDS\_FOR\_RESYNC can now be invoked directly by consumers * Manual (UI) sync runs will now be billed, but every production connection direction for the 24hr period will offset a non-production connection for the same plugin direction *** ### V4.18 #### Created: 2025-12-08 * Added additional console logging for action handler allocation failures * Fixed an issue where criteria deletes were not applied during checkpointing *** ### V4.17 #### Created: 2025-12-06 * Bumped Streamlit version to 1.50.0 *** ### V4.16 #### Created: 2025-12-04 * Fixed an issue where the outbound field mapping auto-suggest popup did not scroll correctly when a large number of mappings are present *** ### V4.15 #### Created: 2025-12-04 * Fixed an issue where syncs could not be deleted after its plugin app has been uninstalled *** ### V4.14 #### Created: 2025-12-03 * Added ngrok config and launch commands to connection details drawer * Due to intermittent action handler issues in the new UI, added extra logging and a timeout for the action handler availability wait * Added multi-environment branch toggle during connection creation/edit *** ### V4.13 #### Created: 2025-11-24 * Modified the sync tuning parameters migration stored proc to leverage bulk atomic Snowflake operations *** ### V4.12 #### Created: 2025-11-20 * Fixed an issue where slider values set via text did not apply correctly * Fixed an issue where records were not truncated during full refresh if checkpointing was not supported for the stream * For dbt-scheduled syncs, branch deployment settings will now show dbt model configuration for the branch * Added a stored procedure for plugins to migrate tuning parameters *** ### V4.11 #### Created: 2025-11-14 * Slider form field changes no longer apply until slider is released, to prevent data source reloads * Radio fields with dynamic data sources now only reload when dependent fields change * Inbound stream lister now reloads when any form field changes value * During sync creation, the sync runner task will be created before inbound table creation is attempted, in case privilege issues cause an error *** ### V4.10 #### Created: 2025-11-13 * Fixed an issue where the stream lister would re-fetch streams when form fields were changed that did not have reload\_on\_change set to true * Added a new option to the sync edit button, to allow quick rollback of configuration to previous snapshots * When switching the daily billing task back to serverless, it will be automatically resumed * Corrected some log messages in the inbound sync processor * Fixed an issue where form fields with dynamic data sources did not auto-select the default form option *** ### V4.9 #### Created: 2025-11-10 * Moved metrics objects into a module to avoid OpenTelemetry warnings * Added setup script logging to identify accounts which still do not support Hybrid Tables * The daily billing task check is no longer performed at sync creation *** ### V4.8 #### Created: 2025-11-06 * Fixed an issue with the daily billing serverless warehouse fallback * Added an advanced section for forcefully updating sync run states *** ### V4.7 #### Created: 2025-11-05 * If the configured daily billing task warehouse has the USAGE privilege revoked from the sync engine application, it will be configured to use a serverless warehouse at next upgrade * SYNC\_RUN records are now created via a direct INSERT statement insert of a Snowpark merge operation * Fixed an issue where unhandled exceptions during sync runs may not be logged to the event table in certain scenarios * Scrollbar and layout adjustments * Adjusted garbage collection time of UI query cache *** ### V4.6 #### Created: 2025-11-04 * Fixed an issue where the inbound storage location could not be reverted at the sync level back to the standard location * Fixed an issue where streams confiogured for full refresh and marked for truncate did not have checkpointing performed * Fixed an issue with logging null metric values in the daily billing process *** ### V4.5 #### Created: 2025-10-30 * Fixed an issue where editing notification rules in the new UI caused an error in the settings menu of the legacy UI * Sync runs will now emit record count custom metrics * Billing metrics will now be output during the daily billing run * Fixed an issue where editing a connection who's slug contains underscores did not allow progression *** ### V4.3 #### Created: 2025-10-24 Merge branch into dev-v4 *** ### V4.2 #### Created: 2025-10-24 * Improved the normalized view column pruning process by leveraging graphlib *** ### V4.1 #### Created: 2025-10-23 * Metrics are now recorded in the event table and shared * View column sorting now leverages graphlib as Python 3.8 support is no longer required * Plugins can now offer tuning parameters which are separate, directly-editable set of fields that tune the performance of a sync * Syncs containing only a branch and no main configuration can no longer be ran from the syncs list *** ### V4.0 #### Created: 2025-10-20 * Provided a new consumer procedure named CLEANUP\_INBOUND\_OBJECTS which can remove orphaned inbound stream objects *** ### V3.129 #### Created: 2025-10-16 * The new UI (streamlit custom component) is now the default * Fixed an issue where dbt models for outbound syncs did not include the source model in the sql file * Consolidated the setup script by removing migration logic used in previous upgrades *** ### V3.128 #### Created: 2025-10-16 General Improvements: * Inbounds syncs with streams marked for full refresh will now have the flag removed after the first checkpoint or at stream post-processing, instead of at the end of the run * Added a check to assert that post-run hook procedures are created with owner's rights UI Improvements: * Add field refresh capability to Sync Strategy configuration * Added refresh buttons to Primary Key and Cursor Field selectors, allowing users to re-fetch updated field lists when source table schemas change. * Fixed an issue where plugin form field data sources would attempt to populated before dependent fields were given values * A branch can now have its schedule modified if it is not defined *** ### V3.127 #### Created: 2025-10-11 * Normalized views which already exist will no longer have select permission re-added unless recreation is forced *** ### V3.126 #### Created: 2025-10-06 * Custom storage locations can now be set during sync configuration * Updated plugin runtime to improve detection of view column references * Added a parameter to RECREATE\_INBOUND\_NORMALIZED\_VIEWS to allow view creation errors to be ignored *** ### V3.125 #### Created: 2025-09-27 General improvements: * Fixed an issue where branch parameters were not rendered correctly in the old UI New UI: * Connection page refreshes after creating/editing connection * Added UI to differentiate between branch and main in history runs *** ### V3.124 #### Created: 2025-09-25 * Plugins which support checkpointing but do not provide any state will have all of their records committed during stream finalization * Outbound syncs will now perform their source table access test inside the sync run, for better visibility of failures *** ### V3.123 #### Created: 2025-09-24 * Fixed a logging issue in the inbound processor *** ### V3.122 #### Created: 2025-09-23 * Added extra debug logging to inbound sync processing loop * Snowflake query cache is now cleared if the page reloads, to reduce cache invalidation traps *** ### V3.121 #### Created: 2025-09-22 * Fixed a setup script error when trying to install the application in a reader account * Fixed an issue where setting the inbound sync location would cause an error if only an unmerged branch was configured *** ### V3.120 #### Created: 2025-09-21 * Remediated an issue where hybrid table migration did not include Recipient records * Fixed an issue where partially completed ngrok connections did not redirect back to the settings menu correctly * Fixed an issue where inbound sync records did not handle query errors correctly *** ### V3.119 #### Created: 2025-09-18 * During upgrades, orphaned sequence objects from old runs will be cleaned up * Fixed an issue where the sync branch revision number did not increment correctly in the versions table * REFRESH\_INBOUND\_STREAM\_SCHEMAS will only store the streams it requested, if a list of streams was provided * SET\_INBOUND\_SYNC\_LOCATION will no longer create landing tables for streams without schemas * Added validation to branching behaviour flag parameters of stored procedures * Adjusted sync run orphaned temp objects cleanup to be after 7 days *** ### V3.118 #### Created: 2025-09-17 * Added consumer views for Recipients and Notification Rules * Fixed an issue where updating notification rules did not update the recipient * Added a check for duplicate rule name when creating a rule * Improved the inbound branching diff view for readability * Fixed an issue where modifying the sync timeout for scheduled deployment type would not save * Fixed issues where branch details would not display on the sync home page *** ### V3.117 #### Created: 2025-09-16 * Fixed an issue in the old Streamlit UI where inbound sync tables did not render correctly in the overview * Inbound sync storage locations with custom database and schema will have the 'usage' privilege granted to the OMNATA\_ADMINISTRATOR application role * When granting 'select' access on tables and views, if an error relating to managed access schemas is thrown, the grant will be retried without the grant option *** ### V3.116 #### Created: 2025-09-15 * Fixed query caching errors on the sync run view page * Added numeric validation to sync run ID filter * Inbound storage location can no longer be set at the branch level, this is to reduce confusion * When configuring a sync-level storage location, the user is no longer warned about clashes based on the sync slug uniqueness * Storage location validation is now performed via a Validate button instead of automatically. Validation checks no longer prevent the storage location from being applied. * Fixed an issue where attempts to delete the previous storage location's tables or views could fail if the database or schema was no longer visible * Fixed an issue where branches would not run on their configured schedule * Fixed an issue where inbound storage examples would fall back on fake values if the current sync was initially created as a branch * Fixed an issue where notification rule changes could not be applied *** ### V3.115 #### Created: 2025-09-11 * Added logging to gauge ongoing usage of the old Streamlit UI * Improved search function for sync run detail modal * Fixed an issue where outbound records with a sync action of delete did not react to a resync request * In inbound normalized views, join columns that do not directly or indirectly reference other streams are now included in the initial CTE * Stream aliases can be used in referencedFields instead of stream names, allowing more flexibility for objects with self-referencing relationships * Fixed an issue where the global inbound storage location did not correctly detect that the standard location was in effect * Removed deprecated ngrok monthly billing functionality *** ### V3.114 #### Created: 2025-09-10 Engine Improvements: * Added a check for null identifiers in outbound sync source tables * Fixed an issue where changing the global storage location did not create the normalized view database and schema if they were missing UI Improvements: * Standardized date formats * Added a warning to inform users that Source Columns should match betweeen branch and main when using Alternate Source Branching. * Refined the layout of the Auto Field Mapper for usability. * Outbound records select all now selects all and applies bulk actions based on criteria * Rate limits can now be set at the sync level * When a branch is abandoned or merged, a "Reopen branch" button will be presented and the configuration will be sourced from the main sync *** ### V3.113 #### Created: 2025-09-04 * Added an inbound record count summary to inbound sync runs table and sync run detail. * Fixed a number of small UI bugs and formatting issues. * Fixed sync run durations (both previous run and next run) showing incorrect values, incl. tooltips * Added sync list tooltip for schedule to align with sync view status next run duration * Exposed the SYNC\_CONFIGURATION\_VERSION table as a consumer view *** ### V3.111 #### Created: 2025-09-02 * Improved background processing loop completion detection * Fixed an issue where automated delete detection did not occur on manual full refresh runs * When viewing an inbound sync run, streams without a total or total estimated record count will display progress as a question mark * Sync name and Sync tags fields will no longer be shown during sync edit or branch creation/edit * Manual schedules will now show orchestration instructions for running the sync * Fixed an issue where not all streams were selectable on the inbound records tab *** ### V3.110 #### Created: 2025-08-29 * When marking remaining streams as completed or abandoned, added a guard against transactional inconsistency *** ### V3.109 #### Created: 2025-08-28 General improvements: * Improved record counting wait time calculations for background processor * Improved reliability of concurrent Snowflake table access from python objects * Sync post hooks will now wait for completion to prevent them being terminated New UI: * Improved the display of sync state * The records tab for outbound syncs now sources record state directly from the current outbound state, and historical data is no longer available in this view * The records tab for inbound syncs now sources record data directly from the raw table of a selected stream, and historical data is no longer available in this view * The records tab is no longer impacted by record retention settings * Added a Last Refresh Component with Query Counter on Home, Plugins, Sync, and History Pages. * Modified the Sync Run Modal to display Active Stream State and added info for the user about checkpointing. *** ### V3.108 #### Created: 2025-08-27 General Improvements: * Fixed an issue with inbound checkpointing where inbound tables were not correctly marked as truncated during a run * Removed SQL utility from the settings menu New UI: * Refresh buttons now display loading icons during data refreshes * Improved the performance of the home screen *** ### V3.107 #### Created: 2025-08-25 * Added Outbound Data tab back in showing source tables for branching modes. * The new UI now has a very minimal set of Streamlit python packages. All advanced functionality such as RSA/GPG key generation, Jinja template evaluation, etc have been moved into stored procs and functions * When setting inbound storage location at the sync level, there is now a clear distinction between standard, global and custom * Stream checkpointing is now performed in all incremental streams where supported by the plugin * Fixed an issue in the new UI where manually running a sync and viewing runs did not incorporate the current branching context * Inbound sync state in the new UI is now branch-aware *** ### V3.106 #### Created: 2025-08-19 * Updated the version of a pinned python package for streamlit, as the pinned version had been removed from the Snowflake Anaconda channel *** ### V3.105 #### Created: 2025-08-19 General improvements: * Introduced stream checkpointing, where stream progress can be saved in case of run timeout or global errors. This feature is enabled via plugins leveraging a new version of the plugin runtime * Detection of existing inbound schemas and tables now uses quoted identifiers * At sync creation, all streams are marked for full refresh with truncate in order to leverage checkpointing New UI: * Fixed an issue where the time limit of an existing sync could not be edited via the Deployment tab * Fixed an issue where the deployment settings would not update after being edited * Prevented the connection environment from being in an invalid state * Preserved other\_environments\_exist parameter when editing connections *** ### V3.104 #### Created: 2025-08-15 * Fixed an issue where a JSON parsing error would display when requesting form field parameters from the plugin *** ### V3.103 #### Created: 2025-08-14 * Fixed an issue where branch runs did not appear in the Sync runs table * On the sync status tab, the latest run start will now respect the current branch selection *** ### V3.102 #### Created: 2025-08-09 * Fixed an issue where notifications were not fired for certain kinds of sync failures. The global exception handler will now fire notifications after marking the run as failed. * Improved the OAuth connection experience in the new UI for authorization code flow *** ### V3.101 #### Created: 2025-08-06 New UI improvements: * Fixed an issue where an error would occur when toggling "Show Only Selected" in stream lister * Various display and layout changes * Improved some of the filter layouts and cleaned up their display for narrower resolutions. * Added a new helper text component for plugin form fields. * Fixed the Sync Strategy icons. * Added more guidance and clarified steps for OAuth and ngrok connection flows. * Fixed an issue where the current record history retention settings did not display correctly * Fixed an issue where choosing an alternate source table for outbound syncs would allocate the alternate table to the main sync and vice versa * Fixed scrollable screen height inconsistencies in the sync config and sync views. * Improved the branching explanation text. * When returning from connection creation, the plugin home page will automatically refresh * Fixed an issue where branches were not automatically activated upon creation * Applied uniform sizing of Alerts across all pages. * Sync name and slug can now be edited inline * When editing a sync, field mappings will be correctly pre-populated *** ### V3.100 #### Created: 2025-07-31 General: * Added an explicit parameter to CREATE\_SYNC for the outbound single environment branching mode * Fixed an issue where newly added inbound syncs with default "Keep History" setting were unexpectedly using "Full Refresh" instead of "Incremental" * Fixed an issue where ngrok provisioning did not work after a sync engine application upgrade * Fixed an issue where sync runs would fail due an error with a temporary results table sequence name New UI: * Fix and refactor syncs list filters and the display of syncs * History: added tags and connections filter * Fix history view now has default display that defaults to 7 days from current date * Added health filter in history overview tab * Added extra filters to History Runs * Sync state displays the entire json structure of the state * Disabled connection option and auth method editing in edit connection flow *** ### V3.99 #### Created: 2025-07-28 General: * Plugins can now optionally narrow down syncs by connection ID when fetching New UI: * Added bulk tag deletes from sync whenever user deletes a tag that has sync attached * Tags can now be assigned during sync creation * ngrok onboarding now has a "Run as a service" configuration option * Implemented theme-aware code block backgrounds and text colours, improved contrast ratios for accessibility *** ### V3.98 #### Created: 2025-07-22 General: * Fixed an issue where the PLUGIN\_FETCH\_SYNCS proc only returned storage information for the first stream * Inbound tables and views will now be granted to the application role with the grant option, so that if they are outside of the application database they can be granted to other roles without taking ownership New UI: * Add on-demand schema fetching when "View Fields" drawer is opened, and now displays schema data types * Added bulk tag layout and functionalities * Stream lister Data Type Drawer now fetches and displays schema data types *** ### V3.97 #### Created: 2025-07-19 General Improvements: * Introduced stored procedures that plugins can use to retrieve sync, branch and connection information. These will be accessible from devkit version 0.10.15 New UI Improvements: * Replace localStorage with IndexedDB and compression for sync configuration storage *** ### V3.96 #### Created: 2025-07-18 General Improvements: * Fixed an issue where syncs using background processing could error when invoking the plugin New UI Improvements: * Fixed outbound single-environment branching mode to respect is\_branching\_toggle property * Added Tags support * Fixed Sync List Grid inconsistent layout *** ### V3.95 #### Created: 2025-07-15 * The sync home back button will now always navigate back to the plugin's home page * Removed unnecessary stream information from stream lister screen * Fixed an issue where refreshing the sync status did not re-fetch branches * Fixed an issue where existing ngrok API integrations did not re-bind during sync engine upgrades * Fixed an issue where stream listing errors were masked by an unhelpful error *** ### V3.94 #### Created: 2025-07-11 General: * Added support for the 'mandatory' flag on an inbound stream * Fixed an issue where inbound stream counting did not occur * Fixed an issue where normalized view CTE columns would reference their own alias, resulting in an error New UI: * Resolve UI flow bugs preventing successful connection completion * Improved error handling and user feedback in connection creation process * Syncs list table (both plugin > sync and syncsList) now sorts in alphabetical order * Created Home Page components for App Recommendations, Runs with Issues and Recent Sync Runs components, and added these to the Home Page * Corrected the wording on Stream State Chips to refer to Streams and change the Sync Run Pending Chip to display the RunState in the context of Sync Runs * Fixed Connections Filter in Syncs list * Fixed an issue where changing connections in the inbound sync wizard would persist streams from the previous connection *** ### V3.93 #### Created: 2025-07-05 * Plugin icons now display in the History Runs tab * Added refresh button and skeleton animations to Sync run detail * The ngrok API external integration object is now named based on the application database name, so that multiple app installs are properly supported * Fixed an issue where the normalized view column re-ordering did not result in the correct order * Fixed an issue where background processing could get stuck waiting for a stream which had already completed *** ### V3.92 #### Created: 2025-07-02 * After running a sync, the sync runs list will automatically reload * Fixed an issue where outbound sync parameter form fields were incorrectly filtered on target type when not using branching * Changed the layout of the ngrok step helpers, improved ngrok steps for readability, and made some other display changes in the create and edit connection flows. * When building normalized views from json schemas, a second pass of column re-ordering will occur *** ### V3.91 #### Created: 2025-06-27 * Fixed an issue where background processing could incorrectly skip post-processing * After editing a sync or branch, there will no longer be a delay before the sync home page contents update * Changed the caching and refresh behaviour of sync, connection and plugin lists *** ### V3.90 #### Created: 2025-06-26 General changes: * Fixed an issue where failed inbound syncs using background processing may have a health state of Pending New UI changes: * Improved scrolling behaviour in new UI, added transparency to footer * Outbound sync config screen now shows outbound target type where applicable * Add support for Jinja template mapping type with syntax highlighting * Enhance field mapping UI with better data type icons and labels * Improve error handling and null safety for field mappings * Added suggested mappings to outbound sync configurations * Fixed an issue where inbound streams were not visible when no sync parameters exist * Fixed an issue where unsupported sync directions were clickable during sync configuration *** ### V3.89 #### Created: 2025-06-24 * Ensured that dbt model configuration in the new UI matches the old * Fixed an issue where default values for form field parameters were not applied * Fixed an issue where query results for sync run information would be accessed beyond the 24 hr window, causing an error * Plugins using the outbound target type migration functionality are now provided with the sync strategy, and it will no longer be applied to inbound syncs *** ### V3.88 #### Created: 2025-06-23 * Added support for daily billing of ngrok tunnels * Fixed an issue where Outbound sync strategy details did not display correctly in dark mode * Slider fields will now show label values * Fixed an issue where editing connections did not preserve the same OTHER\_ENVIRONMENTS\_EXIST setting *** ### V3.87 #### Created: 2025-06-20 * Fixed another background processing error involving empty streams New UI changes: * Layout improvements * Fixed an issue in the new UI where editing an existing sync did not always pre-populate existing values * Fixed the clipping of some buttons *** ### V3.86 #### Created: 2025-06-17 * If an inbound sync using background processing raises a global error, any streams without a status set will be marked as abandoned * Implemented a gradual backoff policy for record counting during inbound syncs that use background processing * Fixed an issue with the background processor where syncs exceeding timeout would have their health status set to pending instead of delayed * Enabled query devtools in production for new UI *** ### V3.85 #### Created: 2025-06-15 * Fixed an inbound sync bug introduced in the previous release *** ### V3.84 #### Created: 2025-06-14 * Fixed an issue where newly discovered streams may cause a sync error if they do not have a cursor field defined and the sync strategy is incremental. These streams will now be skipped with a warning log message. * Fixed an issue where sync runs with background processing enabled may have their global error masked with another error *** ### V3.83 #### Created: 2025-06-13 * Introduced a private preview of background processing via a separate task, which will be introduced gradually. When enabled, background processing of inbound sync runs will occur in an entirely separate snowflake session, avoiding errors caused by async queries (including the infamous "Unknown Error" when retrieving results). *** ### V3.82 #### Created: 2025-06-11 * Fixed an issue with the ngrok initial onboarding process *** ### V3.81 #### Created: 2025-06-10 * Exposed the DELETE\_CONNECTION stored proc to consumers *** ### V3.80 #### Created: 2025-06-10 * Exposed the SET\_SYNC\_CONNECTION stored procedure to consumers *** ### V3.79 #### Created: 2025-06-05 * Updated the ngrok agent instructions to use the correct version 3 format for cloud endpoints *** ### V3.78 #### Created: 2025-06-04 * Fixed an issue where editing existing edge-based ngrok connections would result in a SQL compilation error *** ### V3.77 #### Created: 2025-06-02 * When enqueuing a sync run, total time detection now works for branches * Fixed an issue where some connection creation attempts failed with a network rule error *** ### V3.76 #### Created: 2025-05-30 * Connection completion will now preserve existing network addresses created during the creation process *** ### V3.75 #### Created: 2025-05-30 * Fixed an issue where the ngrok cloud endpoint URL was not permitted for access *** ### V3.74 #### Created: 2025-05-30 * Fixed an issue where the main sync schedule was not correctly configured after a branch merge * Changed the ngrok provisioning model to use a random identifier for the app install, instead of the Snowflake account name * Removed warning messages caused by null span attributes in telemetry * Added list of fields to logged warnings about inbound streams missing schema fields *** ### V3.73 #### Created: 2025-05-29 * Fixed a further issue with ngrok connection onboarding *** ### V3.72 #### Created: 2025-05-29 * The SYNC\_RUN table will be migrated to a hybrid table for all supported accounts * Fixed an issue where ngrok connections could not be provisioned *** ### V3.71 #### Created: 2025-05-28 * Fixed an issue where new form field values couldn't be created in the UI * Introduced a stored procedure for configuring app setting values *** ### V3.70 #### Created: 2025-05-27 * Fixed an issue where ngrok external access integration checks on secret binding could not be performed * Fixed an issue affecting branch syncs where there is no schedule set on the main sync *** ### V3.69 #### Created: 2025-05-26 * Fixed an issue where ngrok connections could not be migrated to cloud endpoints due to object privileges * Fixed an issue where outbound syncs with only a branch configuration could not be ran * For internal Omnata tables, set the DEFAULT\_DLL\_COLLATION at the schema level to ensure hybrid tables can be created *** ### V3.68 #### Created: 2025-05-23 * Fixed an issue where sync runner tasks were not created for branches * Migrated ngrok functionality to use cloud endpoints instead of edges * Added sync run post hooks as a private preview feature * Removed the confirmation prompting of network addresses during connection creation * Fixed an issue where branches could not be created with alternate target parameters *** ### V3.67 #### Created: 2025-05-16 * Added a new application role named VIEWS\_ONLY which only has permission to see the internal data views *** ### V3.66 #### Created: 2025-05-15 * The MARK\_STREAMS\_FOR\_FULL\_REFRESH procedure now accepts NULL for the streams parameter, which will mark all streams for full refresh *** ### V3.65 #### Created: 2025-05-13 * When using the dbt scheduler, configuring the schedule in any context applies to all branches and the main sync * Added extra columns to the SYNC table to improve branching configuration visibility *** ### V3.64 #### Created: 2025-05-08 * Added a workaround for the "Failed to process a Websocket message" Streamlit error in the new UI *** ### V3.63 #### Created: 2025-05-06 * Migrated numeric primary keys to have precision of 38, to work around a Hybrid Tables table scanning bug *** ### V3.62 #### Created: 2025-04-30 * Added support for the new Snowflake Stage form field * Fixed an issue where ssh/gpg keypair fields included backticks in the UI * Fixed an issue where sync run stages were not dropped after the sync run. Added a step to the setup script to clean up stages and tables for old runs *** ### V3.61 #### Created: 2025-04-24 * Plugins can now provide json schema fields with expressions which reference the plugin database * Added a guard for an edge case in normalized view recreation where creation of an existing database would be attempted *** ### V3.60 #### Created: 2025-04-23 * The REFRESH\_INBOUND\_STREAM\_SCHEMAS stored proc is now exposed to consumers *** ### V3.59 #### Created: 2025-04-23 * The Sync Run table will be migrated to a hybrid table on a per-account basis * Fixed an issue with setting the record history retention value in the UI * Added a compatibility fallback option for inbound stream bulk configuration on existing plugin runtime versions *** ### V3.58 #### Created: 2025-04-06 * Plugin external access re-binding will now occur after connection deletion or when connection creation is cancelled *** ### V3.57 #### Created: 2025-04-04 * Fixed an issue where the hybrid table migration logic could cause previously deleted syncs/connections to reappear *** ### V3.56 #### Created: 2025-04-04 * Fixed an issue where some branching mode syncs using dbt scheduler would error during sync creation *** ### V3.55 #### Created: 2025-04-03 * Further improvements to cyclic dependency detection in normalized views *** ### V3.54 #### Created: 2025-04-03 * Improved normalized view pruning *** ### V3.53 #### Created: 2025-04-02 * When building normalized views, fields with cyclic dependencies will result in an error. Also added extra debug logging to the view generation process. *** ### V3.52 #### Created: 2025-04-01 * Fixed an issue where dbt scheduling configuration errored when viewing a non-configured branch *** ### V3.51 #### Created: 2025-03-29 * Fixed an error which may appear in the Streamlit deployment section * Fixed an issue where connections couldn't be deleted if they had an edit in progress *** ### V3.50 #### Created: 2025-03-27 * Improved the pruning process for related fields in a normalized view. Unnecessary columns have been removed from CTEs and the process itself is more memory-efficient *** ### V3.49 #### Created: 2025-03-24 * Allowed dropdown form fields to store an empty value if they are marked as not required *** ### V3.48 #### Created: 2025-03-21 * CTEs in normalized views now have unreferenced columns pruned out *** ### V3.47 #### Created: 2025-03-21 * Fixed an issue where syncs created as branches could not be deleted before the initial merge * Removed the Sync Engine check for pending plugin API configuration to improve performance, as re-binding stored proc integrations and secrets appears to work in the setup script now *** ### V3.46 #### Created: 2025-03-19 * Object columns in normalized views will now be included alongside their convenience drill-downs * Fixed an issue creating branch syncs, introduced by the previous release *** ### V3.45 #### Created: 2025-03-18 * Exposed the SET\_SYNC\_SLUG stored procedure to consumers * Fixed an issue where inbound syncs created as branches could not be merged * Stream fields can now declare dependencies on other fields explicitly. This will enable more robust pruning of unsupported fields when building normalized views. *** ### V3.44 #### Created: 2025-03-16 * Fixed an issue where branch record state/reopen behaviour stored incorrect values. Also fixed incorrectly written data * Fixed an issue where columns which reference other columns in their expression break the view if the column name is transformed. This is avoided by not applying the name transformation in the CTEs, leaving it until the final select clause * Fixed an error in the settings menu caused by the recent Streamlit version upgrade * Branches can now be scheduled to run instead of manually triggered * Merging of sync branches can now include the main sync connection and scheduling information, in preparation for the new UI flow *** ### V3.43 #### Created: 2025-03-10 * When creating a sync initially as a branch, the main sync's source table and connection can now be set at the same time * During creation, outbound branch-only sync parameters can also be set separately to the sync parameters for the branch * Branch-mode syncs can now be created initially with an alternate table and connection set *** ### V3.42 #### Created: 2025-02-28 * If a warehouse is not specified at sync creation, the permission to create a serverless task is checked before creating any sync objects * Serverless warehouse selection is now detemined by selecting "" from the warehouse list * Broadened the error retrying on sync polling to include all "Unknown error" exceptions, as the error numbers are proliferating * Reverted change to skip privilege checks at initial onboarding, as users can ignore them at install *** ### V3.41 #### Created: 2025-02-26 * Changed raw table creation to not use batched async queries due to "Unknown error" being logged *** ### V3.40 #### Created: 2025-02-26 * Fixed an issue where running an already-running sync may result in an unhelpful error message * Fixed an issue where notification rules could not be viewed when applied to specific syncs, plugins or connections *** ### V3.39 #### Created: 2025-02-25 * Fixed an issue where scheduler tasks were being created without a warehouse assigned, when one was specified *** ### V3.38 #### Created: 2025-02-22 * Fixed an issue with outbound date transformation *** ### V3.37 #### Created: 2025-02-21 * Added logic in the setup script to handle internal Snowflake employee accounts which are in regions that don't support hybrid tables * Fixed an issue where an Anaconda package version was removed from the Snowflake channel, breaking the Streamlit UI *** ### V3.36 #### Created: 2025-02-20 * Fixed an issue where existing installed plugins didn't have their outbound record transformation parameters updated when upgrading *** ### V3.35 #### Created: 2025-02-20 * The daily billing task is now created as a serverless task * The initial onboarding step of granting a warehouse is no longer used * Syncs can now be created with a serverless task instead of requiring a warehouse * Fixed an issue where raw tables may attempt to be created multiple times during initial view creation *** ### V3.34 #### Created: 2025-02-19 * Fixed a regression involving outbound record date transformation functionality *** ### V3.33 #### Created: 2025-02-18 * If a plugin changes to a different source defined cursor, the streams will automatically use it * If the default cursor field changes for a stream with a source-defined cursor, the Streamlit configuration page will update the stream's cursor field *** ### V3.32 #### Created: 2025-02-15 * Disabled minification of javascript in new frontend due to Snowflake security requirements *** ### V3.31 #### Created: 2025-02-15 * Fixed the action handler component for the new UI *** ### V3.30 #### Created: 2025-02-14 * If a plugin no longer supports incremental mode for an inbound sync, it will be changed to full refresh at next configuration *** ### V3.29 #### Created: 2025-02-14 * Fixed an issue affecting outbound syncs with timestamp columns on older plugin runtime versions *** ### V3.28 #### Created: 2025-02-12 * Migrated Streamlit to 1.35.0 on Python 3.11 now that the Snowflake 9.2 release has occurred * Reinstated the new UI * Fixed a regression from the previous release involving the outbound record timestamp transformation functionality *** ### V3.27 #### Created: 2025-02-12 * During outbound record transformation, timestamp and date transformation will use an explicit format in case the account level defaults have been overridden. Plugins can override this format if desired, in runtime version 0.10.1 or greater. *** ### V3.26 #### Created: 2025-02-12 * Adjusted setup script in response to an INFORMATION\_SCHEMA behaviour change to hybrid tables metadata *** ### V3.25 #### Created: 2025-02-11 * Exposed the SET\_INBOUND\_SYNC\_LOCATION stored proc to consumers * Fixed an issue where editing a connection could use an incorrect connectivity option * Connectivity option is now displayed on the connection view screen *** ### V3.24 #### Created: 2025-02-08 * Added more setup script handling for another specific hybrid tables migration condition *** ### V3.23 #### Created: 2025-02-07 * Added setup script handling for a specific hybrid tables migration condition *** ### V3.22 #### Created: 2025-02-07 * Fixed a deployment script issue impacting some Azure and GCP customers *** ### V3.21 #### Created: 2025-02-07 * Fixed an issue where plugins using the new table-based stream lister did not return error messages correctly * Added advanced controls to revert hybrid tables deployments * Converted the SYNC\_RUN table back to a base table, due to billing concerns as a hybrid table *** ### V3.20 #### Created: 2025-02-07 * Fixed an issue with the UI where stream listing would still require the streams list to fit into a 16MB single result instead of a table * Rolled Streamlit back to the old version as Python 3.8 is still required by default *** ### V3.19 #### Created: 2025-02-06 * Added further setup script handling to cover failed hybrid table deployments * Temporarily removed new UI Streamlit due to problems with the external package * Upgraded main Streamlit to 1.39.0 * Relaxed pandas requirements to allow recent Python versions *** ### V3.18 #### Created: 2025-02-05 * Fixed an issue where normalized views did not have select grants applied after creation *** ### V3.17 #### Created: 2025-02-05 * Allowed the RECREATE\_INBOUND\_NORMALIZED\_VIEWS procedure to be invoked by a consumer * Initial Private Preview build of new UI *** ### V3.16 #### Created: 2025-01-31 * Added some further Hybrid table data migration logic to retry instances which errored during the first deployment *** ### V3.15 #### Created: 2025-01-31 * Added some more Hybrid table data migration logic to retry instances which errored during the first deployment *** ### V3.14 #### Created: 2025-01-30 * Added a temporary workaround for an issue affecting manual hybrid tables data migration of the SYNC table *** ### V3.13 #### Created: 2025-01-30 * Fixed an issue where Hybrid Table migration was attempted on trial Snowflake accounts, which caused the deployment to fail *** ### V3.12 #### Created: 2025-01-30 * Fixed an issue where Sync Branches were not correctly migrated to Hybrid Tables due to foreign key violations *** ### V3.11 #### Created: 2025-01-29 * If an inbound normalized view already exists, the view creation query result is now logged regardless * Plugins can now return the stream list as a table instead of a single object, avoiding the 16MB limit. This capability is being introduced in devkit version 0.9.11 * Default sync record history retention can now be set to 0 days. When this is selected, sync runs will skip the record history retention storage step * Added an advanced utility for manually performing Hybrid table data migration *** ### V3.10 #### Created: 2025-01-22 * Fixed an issue where a stream automatically added to an inbound sync may fail to have its normalized view created * Fixed an issue in the CONFIGURE\_OMNATA\_INBOUND\_SYNC proc where primary keys were required for streams that weren't requested *** ### V3.9 #### Created: 2025-01-22 * Fixed an issue affecting the firing of notifications * Updated to the latest plugin runtime, which fixed an issue where numeric scale/precision casting was attempted on non-numeric view columns *** ### V3.8 #### Created: 2025-01-20 * Fixed an issue where notification rules filtered on particular events did not appear in the UI * Added a button for deleting notification roles in the UI *** ### V3.7 #### Created: 2025-01-19 * Fixed an issue where modifying the sync schedule would cause an error in the UI *** ### V3.6 #### Created: 2025-01-17 * Fixed an issue where certain inbound stream field descriptions could break the normalized view *** ### V3.5 #### Created: 2025-01-16 * Fixed an issue where the record history feed did not display correctly * Added usage privileges on stored procs related to connection lifecycle, so that consumers can automate connection creation *** ### V3.4 #### Created: 2025-01-15 * OUTBOUND\_SYNC\_RECORD\_STATE is no longer a hybrid table, due to cost concerns on high volume outbound syncs * Added a new variant of the GET\_INBOUND\_STREAM\_VIEW\_DEFINITION procedure which doesn't require a configured sync * Fixed an issue where normalized views were created with only the default columns present *** ### V3.3 #### Created: 2024-12-20 * Fixed an issue with the CONFIGURE\_OMNATA\_INBOUND\_SYNC procedure where stream configuration changes may not be applied *** ### V3.2 #### Created: 2024-12-19 * Fixed an issue where an error may occur in the UI when configuring outbound syncs * When normalized view creation fails, the view definition will now be logged at the warning level instead of debug * Added an experimental stored proc to simplify automated inbound sync creation/update *** ### V3.1 #### Created: 2024-12-17 * Added a consumer view for plugin information *** ### V3.0 #### Created: 2024-12-16 * The sync engine will now use Hybrid tables for most of its internal state, whenever the consumer account supports them *** ### V2.40 #### Created: 2024-12-13 * Fixed an issue where editing a connection would error if the connection slug was changed at the same time *** ### V2.39 #### Created: 2024-12-13 * Updated core to use latest plugin runtime * Fixed an issue where timestamp formatting parameters overridden at the account level could interfere with sync runs * Fixed an issue where editing inbound syncs could cause an error *** ### V2.38 #### Created: 2024-12-11 * Connections now store whether or not other environments exist, to assist with branching decisions during sync configuration * Introduced Outbound Target Types, an optional top level selection for outbound syncs. Each target type can have its own set of sync strategies and provides an easier way to present different sync configuration parameters. They can also nominate a target parameter, another new concept described in the docs. * Registered Plugins now store supported connectivity options in the sync engine * Added the Manual sync schedule to the UI * Fixed an issue where an error may be raised during sync creation * Marking inbound tables for truncation actually uses a delete statement, to avoid auto-commiting the transaction * Upgraded streamlit to version 1.35.0 * Added logging to assist with normalized view creation problems *** ### V2.37 #### Created: 2024-12-03 * Pinned Streamlit's Python runtime to v3.8 so that we can control the migration to 3.11 in January 2025 * Added retries to the plugin cancellation check procedure *** ### V2.36 #### Created: 2024-12-01 * Sync Engine log messages are now automatically linked to the current trace and span in the event table * Fixed an issue where applying settings to an inbound branch would clear the sync state * Fixed an issue where refreshing the data in the controls tab would not reload data for a branch *** ### V2.35 #### Created: 2024-11-28 * Added extra telemetry to sync run processor * Fixed an issue where applying branch settings via dbt raised an error in certain scenarios * Plugins can now choose to load results directly into an object column to avoid json parsing overhead * New field detection now works on a sample of records to improve performance * Improved error handling when syncing * Fixed an issue where a stream without a schema may raise an error when building the normalized view *** ### V2.34 #### Created: 2024-11-21 * Fixed an issue where not all inbound streams were truncated as marked * Added INBOUND\_NEXT\_RUN\_FULL\_REFRESH columns to consumer data views *** ### V2.33 #### Created: 2024-11-18 * Fixed an issue with checking API limits on ngrok connections *** ### V2.32 #### Created: 2024-11-15 * Fixed an issue where outbound syncs containing specific column names that overlap with the record validator alias caused an error to occur *** ### V2.31 #### Created: 2024-11-15 * Fixed an issue where consumer UDFs did not have the connectivity option passed in with the connection parameters *** ### V2.30 #### Created: 2024-11-14 * Fixed an issue where manual connection tests may raise an error for ngrok/privatelink connections *** ### V2.29 #### Created: 2024-11-14 * Fixed an issue with ngrok tunnel creation *** ### V2.28 #### Created: 2024-11-13 * Exposed the CREATE\_SYNC proc to consumers. This is currently undocumented, you must contact support for more information * Fixed an issue where the ngrok connectivity option fails to render the sync configuration form *** ### V2.27 #### Created: 2024-11-12 * Introduced support for Connectivity Options, which allows plugins to signal support for any combination of direct network access, ngrok or privatelink. * Fixed a UI issue where dependant streams raise an error when requesting the list of columns *** ### V2.26 #### Created: 2024-11-06 * Fixed an issue where the sync runner task did not update the user\_task\_timeout\_ms to reflect the new run time limit *** ### V2.25 #### Created: 2024-11-02 * Fixed an issue where outbound records set to skip\_always could appear again as a duplicate identifier * Fixed an issue with the SET\_INBOUND\_SYNC\_STATE procedure where it would error if a sync has never completed *** ### V2.24 #### Created: 2024-10-31 * Plugins can now instruct the sync engine to refresh the schema and recreate the normalized view of inbound sync streams at the end of a run *** ### V2.23 #### Created: 2024-10-30 * Fixed an issue when editing syncs, where the local timezone would be shown instead of the overridden value *** ### V2.22 #### Created: 2024-10-23 * Fixed an issue where branch sync runs would error if a run is already running *** ### V2.21 #### Created: 2024-10-21 * Fixed an issue where the timezone of a sync schedule could not be changed after initial sync creation *** ### V2.20 #### Created: 2024-10-20 * Fixed an issue where branch runs on inbound syncs may error while trying to create tables * Fixed an issue where deleting a sync did not always clean up inbound tables and views that reside in the application database * Added an experimental stored proc for deleting branches *** ### V2.19 #### Created: 2024-10-19 * Fixed an issue where applying branch settings could incorrectly fail due to configuration differences when there were none * If a normalized view column references another column, ensured that they were ordered correctly in the view to accomodate this * Fixed an issue where fetching the inbound view definitions would cause a Streamlit error if there was a problem with the json schema *** ### V2.18 #### Created: 2024-10-16 * Changed the trace level in the application manifest to meet new native application framework requirements * Inbound stream schemas can now include joins to other streams. This will accommodate plugins providing calculated values in normalized views that depend on data from other streams * Provided stored procedures for deleting syncs and setting a sync's schedule *** ### V2.17 #### Created: 2024-10-01 * Fixed an issue where old stream state of deselected streams would cause an error in the Streamlit UI * Upgraded several common python packages to match latest on Snowflake Anaconda channel * Fixed an issue where a 'normalized\_column' error was shown on the sync home screen *** ### V2.16 #### Created: 2024-09-23 * When creating the daily admin task, the warehouse name is now quoted *** ### V2.15 #### Created: 2024-09-22 * Fixed an issue where notifications would be dispatched even when no events were triggered (this is in addition to the fix in V2.12) *** ### V2.14 #### Created: 2024-09-20 * When creating Snowflake tasks, the warehouse name is now quoted * Fixed an issue where manual full refreshes of incremental streams did not perform flagging of deleted records *** ### V2.13 #### Created: 2024-09-20 * Fixed an issue where the sync configuration overview would error when comparing some branch values to main *** ### V2.12 #### Created: 2024-09-20 * Fixed an issue where notifications would be dispatched even when no events were triggered * Instead of clearing the inbound sync state of a stream, users can now mark a stream for full refresh at the next run, which has the same effect but is more clear in its intent. When doing this, a choice is made as to whether the table is truncated or not. * If an existing inbound stream has its storage behaviour changed from Append to Merge, it will automatically be marked for full refresh with truncate at the next run. * A full refresh schedule may now be defined independently of the sync schedule, so that incremental syncs can have a full refresh performed occasionally. This is currently only configurable via a stored procedure but will feature more prominently in a future release. * Marking a stream for full refresh will cause any streams it depends to also perform a full refresh * Fixed an issue where sync runs that fail catastrophically do not dispatch a notification when the next run occurs *** ### V2.11 #### Created: 2024-09-17 * Fixed an issue where recreating normalized views for custom sync locations would attempt to create databases that already exist *** ### V2.10 #### Created: 2024-09-15 * Fixed an issue where empty outbound runs may fail to count records during post-processing * Reduced the amount of logging performed at the info level during a sync run *** ### V2.9 #### Created: 2024-09-15 * Fixed an issue where changing only the column template in the inbound storage location does not apply the change *** ### V2.8 #### Created: 2024-09-14 * Fixed an issue where outbound records that failed source validation were still included in the run *** ### V2.7 #### Created: 2024-09-13 * Fixed an issue where switching away from the Snowflake scheduler did not pause the existing task * Syncs can now be paused and resumed via stored proc calls * For multi-select dropdown fields, metadata will be stored for each selected form option, keyed on its value * Fixed an issue where the outbound record viewer did not include the selected individual fields * Fixed an issue where sync runs which were cancelled or exceeded their time limit caused an error *** ### V2.6 #### Created: 2024-09-06 * Logging levels can now be restricted at the package level inside plugins *** ### V2.5 #### Created: 2024-09-05 * Telemetry event sharing preferences are now configured in the application manifest, instead of prompting the user at first launch of Streamlit *** ### V2.4 #### Created: 2024-09-04 * Fixed an issue where filtering outbound records by record state in the UI caused an error * Plugin registration now captures information about consumer functions, in preparation for displaying this information in the UI * Reduced some of the busier log events during syncing to the debug level *** ### V2.3 #### Created: 2024-08-30 * Fixed an issue where development session runs would fail to finalize for outbound syncs * Modified the inbound sync results processing to ensure that deduping is applied in append mode for changes * Users with the OMNATA\_ADMINISTRATOR application role can now call the PLUGIN\_CONNECTION function directly as a convenience * Fixed an issue where forcing recreation of normalized views from the UI did not have any effect *** ### V2.2 #### Created: 2024-08-26 * Fixed an issue where an error may be thrown when no inbound streams are returned by the plugin * Fixed an issue where modifying an outbound sync branch may result in a SQL compilation error * Fixed an issue where inbound post-processing would throw an error in some situations * Added information to the stream lister to indicate when the Customize option should be used *** ### V2.1 #### Created: 2024-08-24 * Fixed an issue where non-ascii streamlit form parameter values caused an encoding error when passing back to the plugin *** ### V2.0 #### Created: 2024-08-20 * Improved inbound sync post-processing performance * Fixed an issue where creating an ngrok tunnel threw a data validation error *** ### V1.120 #### Created: 2024-08-18 * Reduced the number of post-processing queries when no inbound records are returned for a stream * Moved inbound table DDL to the start of the sync run, to prevent transaction commits while post-processing * Logged errors thrown during stream listing * Added retries for Snowflake throwing "210006: Unknown error" when polling sync job status * Updated to new plugin runtime * For custom inbound storage locations, the database and schema will be created if missing * Allow billing tasks warehouse to be selected again, regardless of the state of the billing task *** ### V1.119 #### Created: 2024-08-12 * Added a stored procedure which can delete staged outbound records by apply state. This can be used under the direction of support in the unlikely event that duplicate identifiers make it into the internal record state table. *** ### V1.118 #### Created: 2024-08-09 Provided two stored procedures: * SET\_SYNC\_NAME to update a sync name after creation * SET\_INBOUND\_SYNC\_STATE to manually override the state of an inbound sync stream Both of these actions will eventually be available in the UI *** ### V1.117 #### Created: 2024-08-07 Testing compatibility of plugins migrating to Pydantic v2 ahead of sync engine * Migrated backend and streamlit to Pydantic v2 * Fixed an issue where branches failed to reopen correctly after a recent product upgrade * Fixed an issue where outbound record duplicate detection was not performed for some sync strategies * Fixed an issue where editing connections with keypair fields may result in an error *** ### V1.116 #### Created: 2024-08-01 * Inbound storage configuration now allows the column name of normalized views to be specified as a template. This allows for column names formats to be standardised, regardless of the source application. *** ### V1.115 #### Created: 2024-07-31 * Schema for normalized views can now contain Snowflake column expressions, for advanced parsing *** ### V1.114 #### Created: 2024-07-25 * Allowed plugin development sessions to have their sync run closed off without providing results * Outbound syncs now upload result objects as JSON, to avoid issues with the arrow serializer. This is enabled in plugin runtime 0.4.10 and newer *** ### V1.113 #### Created: 2024-07-24 * Fixed an issue where an error was displayed on the outbound records tab after record updates occurred *** ### V1.112 #### Created: 2024-07-23 * Fixed an issue where the hover text on the Syncs list did not show the name of the connection * Provided a process to choose the admin tasks warehouse again after initial onboarding *** ### V1.111 #### Created: 2024-07-23 * Added additional checks to ensure no ngrok API errors include API tokens * Modified the SYNC and SYNC\_BRANCH data views to include more configuration information *** ### V1.110 #### Created: 2024-07-21 * Fixed an issue where Snowflake account names containing underscores would break the ngrok tunnel configuration process * Adding multiple ngrok connections will now reuse the same auth token to simplify agent configuration *** ### V1.109 #### Created: 2024-07-19 * When changing the outbound sync strategy during initial configuration, the form definition is reloaded in case there are variations between strategies * Outbound syncs will now have access to the previous state of an updated record, in case there are situations where the plugin needs to know exactly what changed between syncs * Errors during stream listing are now handled gracefully and can be retried *** ### V1.108 #### Created: 2024-07-15 * Syncs can no longer be deleted if other sync's schedules depend on them * Fixed a Streamlit error that can occur when rendering outbound records table * Changed the behaviour of inbound syncs, so that full refresh+append streams can have previously deleted records restored back to an undeleted state * Errors that occur while populating form data sources from plugins, are now caught and displayed within the form boundary * Fixed some Streamlit dependencies who's pinned versions had been removed from Anaconda *** ### V1.107 #### Created: 2024-07-10 * Improved handling of transient transaction-related errors when syncing *** ### V1.106 #### Created: 2024-07-09 * Changed inbound stream lister to ignore source defined primary keys if it's set to an empty list * Added a button to refresh the fields list for primary key/cursor selection * When saving inbound configuration, store any newer json schemas in case the sync is edited again before the next sync *** ### V1.105 #### Created: 2024-07-08 * Modified PLUGIN\_CONNECTION to only return connection metadata for completed connections. This ensures that consumer UDFS do not try to use partially completed connections *** ### V1.104 #### Created: 2024-07-08 * Fixed an issue where quoted columns in outbound syncs caused an error during record staging *** ### V1.103 #### Created: 2024-07-05 * Add retries to the plugin message receiver on the Sync Engine * Modified stream configuration UI to always require a primary key, even with the Append storage behaviour *** ### V1.102 #### Created: 2024-07-03 * Fixed an issue where inbound syncs were able to be created without all streams having a ID Field chosen *** ### V1.101 #### Created: 2024-07-03 * Modified the function which fetchings connection information for consumer functions, to make it more compatible with UDTFs * For inbound syncs, reduced the amount of concurrent post-processing work to reduce transaction errors * Fixed an issue where viewing inbound sync records in the UI caused an error *** ### V1.100 #### Created: 2024-07-02 * Fixed an issue with the outbound field mapper where removal of a column in the source table which was already mapped to an optional column yielded an error * Trace events in Streamlit are now logged asyncronously *** ### V1.99 #### Created: 2024-07-01 * Fixed an issue where outbound syncs using the Create strategy, don't retry after initial failures using the latest record value *** ### V1.98 #### Created: 2024-06-29 * Fixed an issue where parsing a list of tables in some environments would raise an error *** ### V1.97 #### Created: 2024-06-28 * Fixed an issue where inbound storage locations that are already pre-quoted would be quoted again * Fixed an issue where errors streams may not display in the progress accordion of the sync run detail page * Added a "check again" button to the outound source table selector, to re-check for duplicate ID column values * Fixed an issue for the inbound sync run view, where some in-progress streams were not visible * Handled the scenario where the inbound record count exceeds the initial estimate *** ### V1.96 #### Created: 2024-06-24 * Fixed an issue where expired plugin trials could break the plugin selector *** ### V1.95 #### Created: 2024-06-24 * Plugins reporting total record estimates for streams now show a progress bar in the UI * Fixed an issue for inbound syncs, where default inbound storage locations would fail to delete tables when the sync is deleted * Using async stored procs has resulted in occasional transaction-related errors that need retrying. We now retry and eventually ignore these during the plugin execution, but raise an error if it's still occurring at the final commit * Fixed an issue where inbound streams may show as both "completed" and "successful" *** ### V1.94 #### Created: 2024-06-23 * Fixed issue where outbound staging duplicate detection is not isolated by sync branch * Fixed an inbound sync error affecting the most recent plugin runtime * New async results processing now only occurs on plugin runtime 0.4.2, where results are guaranteed to be uploaded before stream status updates *** ### V1.93 #### Created: 2024-06-22 * Inbound syncs now have their deadline and cancellation monitored by the sync engine (thanks to the new async query support) * Post-processing of completed streams now happens during the sync run, so that cancelled/abandoned syncs save progress more reliably * Plugins can now report a total count of records ahead of time * Update inline product help * Fixed an issue where editing existing syncs could result in the inbound storage location being reset to the default * Outbound record staging now detects duplicate identifiers prior to the record state being updated. This reduces the need for manual record cleanup if you accidentally introduce duplicates in your source table *** ### V1.92 #### Created: 2024-06-18 * Added a workaround for a streamlit bug which affects the stream lister *** ### V1.91 #### Created: 2024-06-17 * Removed some excess logging * Fixed an issue introduced by the previous release *** ### V1.90 #### Created: 2024-06-17 * Inbound storage location can now include the type of table to use: standard (with or without change tracking) or hybrid * Changing the storage location copies any existing data to the new location, and also removes the old table *** ### V1.89 #### Created: 2024-06-11 * Fixed an issue where ngrok connection edits did not check for pending API configuration changes before proceeding to the final connection form *** ### V1.88 #### Created: 2024-06-08 * Switched Streamlit record selectors from experimental\_data\_editor to data\_editor *** ### V1.87 #### Created: 2024-06-08 * Added logging to the stream lister component to determine whether users have deliberately toggled a stream * Changed the way the app artifact is added to the system path to improve Streamlit cycle times * Reduced the size of the packaged codebase (both streamlit and stored libraries) *** ### V1.86 #### Created: 2024-06-07 * Fixed an issue where an abandoned notification recipient configuration causes a persistent error on the settings page *** ### V1.85 #### Created: 2024-06-02 * UI now uses Streamlit 1.26.0. Inbound stream lister now uses toggles instead of checkboxes *** ### V1.84 #### Created: 2024-06-01 * Updated inline documentation for inbound syncs * Plugin registrations can now be deleted from the sync engine UI following the uninstall of a plugin application *** ### V1.83 #### Created: 2024-05-24 * A default custom storage location for inbound syncs can now be configured globally *** ### V1.82 #### Created: 2024-05-21 * Changed the behaviour of dynamic-valued drop-downs so that the first option is not auto-selected. This prevents long initial waits for forms that progressively load options from a remote source * Provided ability for users to bring their own GPG/SSH keys during connection creation * When prompting the user to deploy the ngrok agent, the address and port are now prompted for, and the specific cli command or service configuration file are provided *** ### V1.81 #### Created: 2024-05-17 * Snowflake connector/snowpark logging level is now set to WARN by default, to reduce noise in event tables. It can be increased for individual customers via a shared table parameter, up to the level configured for the native app * Any custom snowflake logging level is applied to plugin apps using runtime 0.3.25 or newer * Fixed issue where stream configuration screen could error when editing a sync * Added monthly billing events for ngrok tunnels *** ### V1.79 #### Created: 2024-05-06 * Added logging for sync request payloads *** ### V1.78 #### Created: 2024-05-02 * For inbound streams, improved handling of default primary key fields when they are composite *** ### V1.77 #### Created: 2024-05-02 * Slider fields now have their step size correctly configured to match what was set by the plugin * Added support for composite id fields on inbound streams *** ### V1.76 #### Created: 2024-04-27 * Fixed error that can occur when selecting a cursor field for an inbound sync * Connections using ngrok now have the agent launch command available when you view the connection details *** ### V1.75 #### Created: 2024-04-23 * Fixed an issue where id-only deletions in run results resulted in an empy record rather retaining the previous value of the record * Added cursor field and primary key field to the sync home page summary *** ### V1.74 #### Created: 2024-04-22 * Disabled client side encryption on plugin results stage, so that results can be streamed * If plugin application has been uninstalled, show a warning message when creating a new connection *** ### V1.73 #### Created: 2024-04-17 * Plugins are now provided with an internal stage to upload run results to *** ### V1.72 #### Created: 2024-04-17 * Fixed an error where ngrok resources are not cleaned up when connections are deleted * Fixed error that may occur when editing connections with ngrok tunnels * Removed the requirement that plugins return a list of errored streams from the SYNC\_INBOUND operation. This simplifies deferring the operation to other languages like java. *** ### V1.71 #### Created: 2024-04-10 * ngrok tunnel limits can now be increased centrally by Omnata *** ### V1.70 #### Created: 2024-04-10 * Fixed an issue where an error occurs if an inbound stream does not report state before ending * Added support for ad-hoc consumer-facing functions that can be shipped with plugins and invoked using a connection slug * Fixed an issue where an error can occur when customizing streams * Fixed an issue where ngrok-based connections may display an error during initial creation *** ### V1.69 #### Created: 2024-04-03 * If the native app doesn't own an inbound raw table, it will no longer error on configuration save when attempting to run grant statements *** ### V1.68 #### Created: 2024-03-28 * When hovering on the help text for a custom cron schedule, a human readable description is now shown instead of the cron expression * All streamlit dependencies are now pinned to specific package versions, to ensure Anaconda availability *** ### V1.67 #### Created: 2024-03-27 * Fixed an issue where outbound sync pre-flight checks would fail due to source table not being quoted correctly * Provided a fallback configuration option for new streams when the plugin runtime is less than 0.3.19 * Provided a maintenance process which deletes Sync Record State History older than a configurable number of days, to reduce storage costs at high record volume * Fixed an issue where outbound syncs could not have record transformation applied due to table names not being quoted * Created all Sequences with the ORDER attribute set, to ensure identifiers increment * Included the application's own database in the list of outbound source databases * Reverted the UI back to Streamlit 1.22.0 *** ### V1.66 #### Created: 2024-03-25 * Fixed an issue where results deduplication did not work correctly for inbound syncs * Fixed an issue where the append storage behaviour would create duplicate records *** ### V1.65 #### Created: 2024-03-25 * During post run processing, added some exception logging logic so that transaction rollback errors don't cause the original error to be lost *** ### V1.64 #### Created: 2024-03-22 * On the Syncs list, custom schedules now display the cron string without needing to hover on the help text * Inbound syncs no longer offer a choice of Incremental or Full Refresh unless you choose the Customize option. Instead, Auto mode will choose incremental where possible, and fall back on full refresh. * Storage behaviour is still a choice, but it is now "Merge changes" or "Keep history" in the UI and an explanatory diagram is included. * For full refreshes, the term "Replace" has been deprecated in favour of "Merge" * Upgraded the UI to Streamlit 1.26.0 * Fixed streamlit warning on pre-populated checkboxes on sync configuration forms *** ### V1.63 #### Created: 2024-03-21 * Syncs list now has a column for Sync Schedule * Fixed issue where multi-select dropdown with required value allowed the user to continue without any values selected * Added some extra columns to the SYNC\_RUN view, relating to inbound record counts * Fix issue where deletion by criteria with the append storage behaviour could append multiple delete records for the same identifier * Added a button on the inbound sync run screen to refresh the list of streams *** ### V1.62 #### Created: 2024-03-19 * The total number of inbound records for a run is now logged pre and post deduplication. A warning is logged if deduplication removed records. * Fixed an issue with the new deletion functionality, where the new sync engine version was not compatible with older runtime versions * Fixed an issue where non-empty inbound runs were hidden by the "Hide empty completed runs" checkbox *** ### V1.61 #### Created: 2024-03-17 * After clearing state for an inbound sync, the table is truncated when applying the results for the next run *** ### V1.60 #### Created: 2024-03-15 * The "Append" inbound storage behaviour now only stores changed records, making it much more storage efficient when used with the Full Refresh strategy. * Inbound sync tables now contain a flag which indicates when the record has been deleted at the source. When using the Full Refresh strategy, this is maintained automatically via the absence of previous records. * Inbound sync tables now contain the run id that the most recent change for each record belongs to. * Plugins can now provide individual record deletions, and also deletions by criteria for inbound incremental syncs. This requires omnata-plugin-runtime version 0.3.15 or greater. * When processing results of inbound syncs at the end of the run, the engine will commit the data and latest state of each stream individually, so that if errors are encountered then progress is not lost. * When a sync runs for longer than its configured limit, it is assumed failed and can be re-ran. The previous run will be marked as failed when this occurs. * Sync run id is now included in event span attributes, to make it easier to filter event table queries * When comparing branch stream configurations, the stream state is no longer included in the comparison. * Fixed issue with Sync History tab where incomplete outbound syncs did not render correctly * Inbound stream cursor field now has the default value applied to it at initial sync creation, so that plugins don't need to check both values * Fully deprecated the internal Sync Run Log table in favour of event tables *** ### V1.59 #### Created: 2024-03-07 * Added the PLUGIN\_UPDATE\_API\_CONFIGURATION stored procedure, which can reset the integrations and secret bindings for plugin udfs/procedures * Fixed Streamlit error when switching between full and incremental inbound sync strategy for a single stream * Provided a button in the Advanced area of Settings which re-binds all plugin integration and secret objects * Ensure default cursor field is applied for incremental syncs * Fixed issue where Streamlit could not show information about an SSL certificate because the pyopenssl library was no longer available *** ### V1.58 #### Created: 2024-03-06 * Fixed streamlit UI error on sync home screen when production connection is not yet configured for inbound syncs in branch mode * Fixed issue when nominating a production connection for inbound syncs * When presenting dbt model configuration for streams, handled the scenario where JSON schema is not available * Inbound stream schemas are now fetched when syncs or branches are created or updated, so that the schema is visible in the UI prior to the first run *** ### V1.57 #### Created: 2024-03-06 In normalized views, object fields are now supported as columns *** ### V1.56 #### Created: 2024-03-05 Fixed issue where refreshing inbound stream schemas manually resulted in an error *** ### V1.55 #### Created: 2024-03-03 This is the first version to contain release notes. Changes in this version: * Reset page number when stream search changes * Killing sync runs no longer attempts to kill the Snowflake query * Added a refresh button to the Syncs page * Inbound sync history is now paginated * Added a refresh button to the Connections page * Fixed error when recreating views for inbound syncs * Removed sync engine billing from daily billing run in preparation for switch to plugin-only billing * Added consumer facing views of internal app state ***