Creating Plugins

Use this guide to author your own Omnata plugin.

In the previous sections, we walked through some example plugins to demonstrate how they work.

Here, we'll cover the development tools that Omnata provides to help simplify the task and make you productive.

In this example, we'll build a plugin which supports outbound syncing. Once you're familiar with this, the inbound functionality works similarly.

Summary

Ultimately, creating a plugin is an simple as uploading a python class with a product logo, and optionally some other python files or libraries. These are packaged and distributed as Snowflake Native Applications.

Omnata provides a development kit which helps by providing:

• a plugin template as a starting point

• a convenient command-line interface (CLI) which takes care of uploading and packaging your plugin code, and registering it with the Omnata Application

• a "development session" concept, so that you can use an interactive session (e.g. a Jupyter notebook) to build the plugin functionality incrementally with a fast feedback loop, and verify that it's behaviour is correct

• automated test case generation, including HTTP recording with secrets redacted

Once it's working correctly, we can migrate it all into the plugin class and tidy it up.

Step 1 - App Familiarisation

Read over your application's API documentation to gain familiarisation with it.

Questions to answer:

• What use cases does the application have for receiving data from Snowflake? What are some examples?

• What style of sync will it be? Event-style, where Snowflake rows are published once? Or record-style, where ongoing changes are replicated?

• How do people typically call the API? Is there a python package we can leverage, or will we make http requests?

• What information do we need to connect to the app? (e.g. API Key, OAuth Client Credentials)

• What information do we need to collect to configure the sync behaviour? (e.g. names of objects/fields in the app, any other options exposed by the app's API)

• What rate limits apply to the API?

Step 2 - Environment setup

To enable iterative prototyping of a plugin, we recommend using Visual Studio Code with the Python extension. This will help you manage your Python environment, and also includes Jupyter support which works nicely with our SDK.

Your Python plugin is able to use any of the Python packages available in the Snowflake Anaconda channel or on PyPi. By adding the package to the requirements.txt file, the Plugin uploader will either reference it on Anaconda or download it from PyPi.

We recommend using a Python environment manager like conda to keep your plugin's environment controlled and minimal.

In order to develop plugins, you'll need to install the Omnata Plugin Devkit:

pip install omnata-plugin-devkit

After this, all plugin development commands can be see by running:

omnata plugin_dev -h

Finally, the Omnata plugin devkit uses snowcli to authenticate to your Snowflake environment, which means you must follow its onboarding instructions first.

Step 3 - Create Plugin from template

We recommend you create a git repository and clone it first, so that you can properly manage changes to your code.

Run the following command to generate a plugin project into the current directory:

omnata plugin_dev init

Start by editing the plugin's get_manifest method, so that it returns a unique name and the real label for your app, as well as the sync directions and strategies you intend to support.

Then you can run the following command to upload your plugin to your Snowflake account:

omnata plugin_dev upload

After this command completes, unless you overrode the defaults:

• Your code will have been uploaded to a stage named OMNATA_PLUGIN_DEVELOPMENT.<Plugin ID>.PLUGIN_CODE

• An application package named <Plugin ID>_PLUGIN will have been created, with a version named DEVELOPMENT

Then you can create/update an instance of your plugin by running:

omnata plugin_dev deploy <package name from above> <application name>

This will create an application from the above application package.

There's one final step which only needs to be done once, initially. Because applications have no visiblity of each other by default, we have to register the plugin application with the Omnata Sync Engine:

omnata plugin_dev register <application name>

Now in the Omnata UI, you should see your plugin in the list.

Step 4 - Connect to your App

Edit the plugin's connection_form method so that it returns the fields you want to present to the user. You can see the form whenever you create a new connection in the UI.

Also edit the plugin's connect method to have it do a direct connection test, or return OAuth parameters to commence an OAuth flow.

Upload the plugin using sample code shown in the plugin_upload.ipynb notebook. There will be warnings to say that the plugin is incomplete, that's ok.

Once it's uploaded, you can go into the Omnata UI, create a new Connection, then select your new app from the list. Ensure that the connection is created without errors.

Step 5 - Define Sync Parameters

Now modify the outbound_configuration_form method to return the fields required for sync configuration. This usually involves at least one fixed value (like the Salesforce object name or the Slack channel name).

To map the Snowflake table data to something inside the app, you choose an appropriate type of Mapper to return:

• FormFieldMappingSelector provides a visual column -> field mapper. Your Plugin provides a list of possible target fields in the app, and the user selects which Snowflake columns to map to them. This mapper is most common in record-based systems like Salesforce CRM.

• FormJinjaTemplate provides a text-based template, which can have Snowflake column values interpolated. This mapper is most common in event-based systems like Slack.

Once you've defined the form, upload the plugin again to apply the changes. Then, create a Sync in Omnata. Just choose a manual schedule so that it never runs automatically.

Step 6 - Sample data

Now, create a table with sample data. This will be used to test the plugin.

Step 7 - Build via a development session

Use the example code provided in the plugin_development.ipynb notebook to create a development session.

A development session simplifies the hardest part of plugin development; writing the code that talks to the App. It does this by connecting to your Snowflake account and retrieving what's normally provided to the sync_outbound method during a scheduled sync.

Instead of having to continually upload new versions of your plugin and re-run syncs until the sync_outbound method works perfectly start to finish, you can simply use the parameters and sync_request objects in your Jupyter notebook to gradually construct the code.

As part of this, you'll capture the outcome of the API operation back into a DataFrame to return to Omnata. This DataFrame contains the columns:

• IDENTIFIER - Provided in the original apply request

• SUCCESS - A boolean flag to indicate true/false

• APP_IDENTIFIER - (Optional) an identifier for this record on the app side

• RESULT - A JSON object containing any useful/relevant information about the record or specific errors encountered. Error messages returned under the "error" property will be displayed in the UI for troubleshooting.

The response DataFrame can be validated with:

dev_sess.validate_response(sync_request)

This will return a Success or Failure message back to the terminal

Once you get a success message, you can complete the scenario:

dev_session.complete_scenario()

Step 8 - Generate test cases automatically

Once your Jupyter notebook code is working, and you can see data appearing in your app correctly, you should move the code from your Jupyter notebook into the apply_records method of your plugin.

Once this is done you can re-run your development session start to finish going directly against your completed plugin class, as shown in plugin_test.ipynb.

If it all works as expected, you can generate a behave test:

dev_sess.generate_behave_test()

This will output a feature file, as well as a VCR.py cassette containing your HTTP requests and responses, which is referenced in the behave test.

The test case generator assumes that everything worked correctly, but you need to confirm this yourself!

In other words, the test case is useful for ensuring you don't inadvertently break this scenario in future, but it gives no guarantees of original correctness.

It also relies on recordings of the app responses, which assumes the app's behaviour will remain consistent.

This behave test can be ran at any time, without needing to connect to Snowflake or your App.

You can continue to record and generate new test cases to cover other sync scenarios and account for different behaviours in the app.

Step 9 - Implement inbound syncing

Similar to outbound, implement the inbound_configuration_form and sync_inbound functions to enable inbound syncing.

Step 10 - Complete your plugin

With the plugin fully working locally, you can upload it again.

Now Omnata can start syncing data for you by scheduling the sync you created.