Skip to main content

Snowflake Documentation

Secured data sharing

Unravel leverages the Snowflake data sharing with secure data sharing option for data transfer over shared connections outside the Virtual Private Cloud (VPC) or through private links. This involves employing a serverless/warehouse task to independently replicate metadata from the schema in the customer Snowflake account, allowing you to share metadata directly with the Unravel Snowflake account in the same region. The data is polled and shared every 1 hour by default. The polling can be configured.

For secure data share from Snowflake, complete the prerequisites and configure the Secure data share option.

The solution provides the following benefits:

  • No requirement for a direct connection to Snowflake

  • Rapid setup and reduced complexity for security permissions

  • Availability of all Unravel features with configurable latency

Architecture

Unravel Snowflake Architecture - Secure Share

Snowflake-architecture.png

The Unravel Snowflake Architecture - Secure share is a comprehensive framework designed to integrate with the Snowflake data warehouse, providing users with robust data analytics and management capabilities. The architecture is divided into three main layers: User Layer, Core Layer, and Data Source Layer, each serving distinct purposes and functionalities.

  • User Layer: This layer interfaces directly with users and clients, offering web application access, REST API interactions, and external notifications through platforms such as Email, Slack, and Microsoft Teams. It ensures secure and authenticated access using username/password for web users and revocable tokens for API clients.

  • Core Layer: Deployed on virtual machines or cloud services like AWS, Azure, and SaaS, this layer includes critical components such as NGINX for load balancing, a notification service, various analytical and service modules, and a datastore. The Core Layer facilitates seamless data management and insight generation through components like the Intelligent Insight Generator (IIG) and Snowflake Worker. It supports development environments like Python and Java and runs on CentOS.

  • Data Source Layer: This layer securely connects to the Snowflake data warehouse using a Snowflake-JDBC secure link with local authentication. It handles both shared and customer-specific data, ensuring that data operations are performed within the same region (us-east-1) to optimize performance and reduce latency.

    The architecture emphasizes secure communication pathways, including external connections, HTTP/HTTPS traffic, and internal communications, ensuring data integrity and confidentiality. It provides a scalable and efficient platform for data analytics, capable of integrating with various external systems and notifying users through multiple channels.

Sharing Data with Secure Data Sharing

Snowflake-Sharingdata.png

This diagram outlines the process of securely sharing metadata between a customer's Snowflake account and Unravel (or a partner) using the secure data sharing feature. This setup ensures that customer metadata remains within the customer's control while allowing Unravel to access and analyze the metadata as needed.

The workflow involves three main steps:

  • Export Metadata: Customers run scripts to create databases, schemas, and tables, exporting the metadata to transient tables.

  • Share Metadata via Snowshare: The customer uses Snowflake's secure share feature to share the exported metadata with Unravel.

  • Poll Metadata: Unravel polls the shared metadata directly from the customer's storage through Unravel's Snowflake warehouse, without storing it on their end.

    This process ensures secure, efficient, and on-demand access to customer metadata, enabling Unravel to perform analysis and monitoring

Prerequisites for secured Data sharing

The following prerequisites must be fulfilled before you use the secured data-sharing option:

Permissions and Roles

  • Assign the ACCOUNTADMIN role to the user.

  • Grant SELECT permission on the schema SNOWFLAKE.ACCOUNT_USAGE.

Unravel Account ID

Contact Unravel Support for Unravel account ID corresponding to the platform, region, and edition.

Configuring the Secure data sharing option

Unravel provides two methods for secure data sharing between the customer and Unravel:

  • Health Check: One-time data share from the customer to Unravel.

  • Snow-Share Live: Continuous data sharing from the customer to Unravel.

Health Check: One-Time Data Share

Use the Health Check option to perform a one-time secure data transfer from your Snowflake environment to Unravel. This method is typically used for a snapshot of your data at a particular point in time for analysis and recommendations.

To configure the Health Check option:

  1. Run the SQL statement from this link to create procedures and necessary functions.

  2. You must replace it with the correct <Unravel Account ID> based on your Snowflake region and account type. Contact Unravel Support to obtain the appropriate Account ID for your region.

    /**
Step-1 (One time execution for health check for 35 days)
*/
CALL CREATE_TABLES('SECURE_SHARE','SCHEMA_4825');
CALL REPLICATE_ACCOUNT_USAGE('SECURE_SHARE','SCHEMA_4825', 35);
CALL REPLICATE_HISTORY_QUERY('SECURE_SHARE','SCHEMA_4825', 35);
CALL CREATE_SHARED_DB_METADATA('SECURE_SHARE','SCHEMA_4825');
CALL WAREHOUSE_PROC('SECURE_SHARE','SCHEMA_4825');
CALL CREATE_QUERY_PROFILE(dbname => 'SECURE_SHARE', schemaname => 'SCHEMA_4825', credit =>
'1', days => '15'); /**
SHARE tables to given accountId
*/
CALL SHARE_TO_ACCOUNT('<Unravel Account ID>');
Snow-Share Live: Continuous Data Share

Use the Snow-Share Live option to set up continuous, secure data sharing between your Snowflake environment and Unravel.

To configure the snow-share live option:

  1. Keep the following information ready:

    • Unravel account ID (provided by Unravel based on your Snowflake region). You must replace it with the correct <Unravel Account ID> based on your Snowflake region and account type. Contact Unravel Support to obtain the appropriate Account ID for your region.

    • Warehouse name (created in your Snowflake account). You must replace SET WAREHOUSE_NAME = '<WAREHOUSE_NAME>'; with the created warehouse name.

  2. Run the SQL statement from this link to create procedures and necessary functions.

  3. You can also update the following optional parameters if required. The default values will be applicable if these are not updated.

    • DATABASE_TO_SHARE: Name of the database to share.

    • SCHEMA_TO_SHARE: Name of the schema to share.

    • SHARE_NAME: Name of the secure share to create.

    • PROFILE_QUERY_CREDIT: Credits allocated for query profiling.

    • ACCOUNT_ID: Your Snowflake account ID.

    • R_DAYS: Number of days of real-time queries to poll.

    • H_DAYS: Number of days of historical queries to poll.

    • DAYS_TO_KEEP: Number of days to retain query and access history table data.

    • Task schedules for:

      • REPLICATE_METADATA

      • REPLICATE_STORAGE_METADATA

      • REPLICATE_HISTORY_QUERY

      • CREATE_PROFILE_TABLE

      • REPLICATE_WAREHOUSE_AND_REALTIME_QUERY

      • CLEANUP_DATA_TASK

Backend steps to configure Snowflake account

Follow these steps to set up Snowflake integration with Unravel using the manager command. All settings are applied by an administrator.

  1. Stop Unravel services.

    <Unravel installation directory>/unravel/manager stop

  2. Add Snowflake account

    <Unravel installation directory>/unravel/manager config snowflake add

    The configuration settings listed under Configuration Parameters are applied by an administrator when your account is set up. You don't need to enter or change these values during regular use.

  3. Apply the configuration.

    <Unravel installation directory>/unravel/manager config apply

  4. Start Unravel services.

    <Unravel installation directory>/unravel/manager start
Configuration Parameters
Account information

Parameter

Property

Description

Account

com.unraveldata.sf.account.id

Identifies the Snowflake account integrated with Unravel.

Internal id

Internal reference ID for the environment.

Display name

com.unraveldata.sf.account.displayname

Name used to identify the Snowflake account in Unravel.

Cloud provider

com.unraveldata.sf.cloud_provider

Cloud platform that hosts the Snowflake account (for example, AWS, Azure, or GCP).

Region

com.unraveldata.sf.region

Cloud region where the Snowflake account is located.

Pricing plan

com.unraveldata.sf.pricing_plan

Type of Snowflake subscription: on-demand or pre-purchased.

Authentication and access

Parameter

Property

Description

Auth type

com.unraveldata.sf.auth.type

Specifies the authentication method for connecting to Snowflake.

Username

com.unraveldata.sf.user

Snowflake user that acts for Unravel.

Password

com.unraveldata.sf.password

Password if using basic authentication.

Private key

com.unraveldata.sf.private.key

Private key used for keypair authentication.

Role

com.unraveldata.sf.user.role

Role that has the required permissions for monitoring and analytics.

Resource and data management

Parameter

Property

Description

Warehouse

com.unraveldata.sf.warehouse

Compute warehouse that Unravel uses for running jobs.

Warehouse manage database

com.unraveldata.sf.warehouse.manage.db

Database for managing warehouse resources.

Warehouse manage schema

com.unraveldata.sf.warehouse.manage.schema

Schema for management tasks in the warehouse database.

Database

com.unraveldata.sf.QA_EU.database

Snowflake database monitored and analyzed by Unravel.

Schema

com.unraveldata.sf.schema

Schema within the database that Unravel monitors.

Data load and analytics

Parameter

Property

Description

Load mode

com.unraveldata.sf.data.load.mode

How data is ingested. Options are live, snapshot, dump, or replication.

Credit rate

com.unraveldata.sf.credits.to.isocurrency

Amount of Snowflake compute credits used for Unravel tasks.

Information schema

com.unraveldata.sf.information.schema

Reserved for future use. This property is not currently used. In a future release, it may enable polling metadata from the INFORMATION_SCHEMA in Snowflake.

Look back days

com.unraveldata.sf.look.back.days

Number of days in the past that Unravel analyzes.

Priority

com.unraveldata.sf.priority

Priority for job execution: low, medium, or high. Accepted values are L,M, or H.

Cost and transfer rates

Parameter

Property

Description

Storage rate (USD)

com.unraveldata.sf.storage.to.isocurrency

Cost for Snowflake-managed storage, if applicable.

Transfer rate same cloud and region (USD)

com.unraveldata.sf.datatransfer.scsr.cost.to.isocurrency

Cost to transfer data within the same cloud region.

Transfer rate, same cloud and continent (USD)

com.unraveldata.sf.datatransfer.scsc.cost.to.isocurrency

Cost to transfer data within the same cloud, across regions.

Transfer rate, same cloud, different continent (USD)

com.unraveldata.sf.datatransfer.scdc.cost.to.isocurrency

Cost to transfer data within the same cloud, across continents.

Transfer rate, different cloud (USD)

com.unraveldata.sf.datatransfer.dc.cost.to.isocurrency

Cost to transfer data between different cloud providers.

In this section: