ADF, Synapse, CosmosDB, ETL

Required for a project at work. Ordered by exam objectives (not a fan of this) + short explanation. Useful info about constraints set by cloud services (azure) and how it may act as a failure mode. Good tips to save costs etc

Design and implement data storage
1. Azure Data Storage Options - blob vs adls2?
  1. blob
    1. cheapest
    2. allows “hot to cool to archive” storage and other lifecycle management
    3. flat namespace - no heirarchy
    4. slow query performance
  2. ADLS gen2
    1. on top of blob storage
    2. hierarchical namespace is allowed, faster query performance
    3. Hadoop compatible - HDFS
    4. unix like file permission
    5. more scalable, dont need to move data for analysis
2. Recommended file types for storage
  1. CSV
    1. comma, pipe, tab
    2. plain text format
  2. json
    1. used in web services
    2. nested plaintext
  3. Apache Avro
    1. binary format
    2. great for write
    3. schema support - enforce properties, data types etc
  4. Apache Parquet
    1. binary
    2. columnar
    3. read optimised
    4. schema/metadata support
3. Design for efficient querying
  1. how many bytes does your query read or write? (monitoring, gathering log data), how cpu intensive is the query? What level of indexing is your data supporting? (non-relational db all json properties are indexed)
  2. Field indexing - primary key, partition key(horizontal partitioning, what key or property is used to create the partition), row key (each row has its own key, usually primary key)
  3. Demo - created a storage account - uploaded files - manage acl(unix permission) - added the ADLS query acceleration feature
Design for Data Pruning
1. Design a folder structure that represents levels of data transformation
  1. data pruning - remove irrelevant data to improve query perf. done in T of ETL. Missing rows, invalid data
  2. data partitioning -
    1. adls2 allows query acceleration using HFS.
    2. relational db - Synapse sql pool - horizontally partition tables (e.g date) - can drop partitions if they are not needed
    3. non -relational db - spark - happens in memory - can repartition in the context of in memory data frames
2. Design a distribution strategy
  1. azure synapse dedicated sql pools - 60 distributions for parallel processing MPP - when a table is created you specify the distribution method that instructs synapse row by row where each row goes. Ie
    1. Hash - tables >2gb, i/o ops - synapse creates a hash of the column, put each row in a seperate distribution based on the hash - combination of hash value and a compression of data at each node using a special data structure - makes it really fast to satisfy large queries
    2. round robin - better when temporary data
    3. replicated >2GB - replicate all rows to every table - satisfy quickly
3. Design a data archiving solution
  1. azure storage access tiers - charged by REST API txn + data size
    1. hot tier - discount on REST txn
    2. cool - discount on storage ( >30days)
    3. archive (>180 days) - discount on both
  2. Automate the collection of archiveable data - demo
    1. synapse analytics workspace - data warehousing of relational and non-relational data
      1. dedicated sql pools are expensive - turn off whenever possible - built in serverless is the default
      2. transact sql query - partitioning a table - specify column names and constraints, “CREATE clustered column store index”, “DISTRIBUTE hash(ColumnName) PARTITION ([orderdatekey] range for values [20000101, 20150201])
    2. azure storage - cant edit for archive, can do it for cool, hot
    3. lifecycle management policies - allow for the creation of rules - if then cases to move to stages into archive/cool or even delete
Design a Partition Strategy
1. For files
  1. blob storage - range partioning using lexical sequence
  2. adls gen2 - better performance
    1. Plan the directory structure
    2. choose right file type - avro parquet - binaries faster than parsing through plain text for some domains
    3. implement adf to move files into their directories
2. for analytical workloads
  1. horizontal partitioning (sharding) - divide row data across distributions
  2. vertical partiooning - divide into column subset - (like views)
  3. functional partitioning - security and business requirements take priority over speed
3. for efficiency and performance
  1. Optimise data distribution to minimise cross partition data transfers
  2. size the cluster appropriately to support parallelism (single partitoin shouldnt be too hot)
  3. Refactor queries to optimise performance
4. Design a partition strategy for azure synapse analytics
  1. SQL pool types
    1. serverless
    2. dedicated - increased reliability and performance - can automate pausing these pools using azure funcs
    3. sql pool distributions - hash, round robin, replicated
      1. maximise data loading efficiency - round robin faster than hash
5. identify when partitioning is needed in adls2 - demo
  1. data rates - are limited per storage account
    1. ingress - charged
    2. egress - charged to remvoe data out of the account
  2. scalability and performance rate - target request rates for a single block - design checklist - naming convention (lexical), networking (region storage and replication) to be closer to users
  3. azure sql db is a transaction processing system - day to day. synapse - long term . common patterns - use db for daily work and ingest into synapse every night
  4. tasks like redefining the range - we might have to empty the partitoins, reconstruct the table, and then copy it back - hence there are some limitations post partition of a table that might need clumsy workarounds (e.g redefining range requires emptying partitions)
  5. CTAS create table as select - create a staging table - same hash, index, - this allows partition switching - refresh statistics of the target table, drop staging tables
Design the Serving Layer - data modeling
1. Design star schemas
  1. Collection of content (fact) tables with high volume - which is why hash distribution is recommended
  2. Dimension (context) table - smaller size - use replicated distribution
  3. Snowflake schema - variation of star schema
2. Desing slowly changing dimensions (SCDs) - infrequently changing data - email address
  1. synapse supports scd but it needs to be manually implemented
  2. SCD1 - old column value is overwritten -
  3. CSD 2 - maintain value change history - preserve original record (+ isCurrent=False optionally) + new record
  4. SCD 3 - partial column history - original email + CurrentEmail - columns that indicate history and updatee the current like scd1
3. Design a Dimensional Heirarchy - e.g org chart
4. Design a solution for temporal data - store data as it existed at a spcific point of time
  1. azure sql temporal tables - create table WITH SYSTEM_VERSIONING = ON
5. Design for incremental loading - ingest data in smaller increments over time
6. Design analytical stores - HTAP - hybrid txn and analytical processing - analytical store - speed data access and reduce contention - cosmos db - nosql - can be used as a transactional store and analytical store - the analytical store can be integrated with transactional store by azure synapse link- Azure Synapse Link is the The integration layer that lets you query the analytical store directly from Synapse Analytics (Spark or serverless SQL) without moving data with manual ETL. It surfaces the analytical store to Synapse for near real-time analytics
7. Design metastore in azure synapse analytics and azure db - meta store is data catalog is called the apache hive metastore that descibes data model and structure - can be created using the azure synapse spark pool - databricks is another method of doing the same
8. Demo - “populate slowly changine dimensions”
Implement Physical data storage structures
1. Implement compression - copy data task supports data compression - bzip2,gzip, deflate, zipdeflate
2. implement partitioning - allows snapse to make parallel queries
3. implement sharding - ""
4. implement different table geometries with synapse analytics pools - distributions, indexes(how rows are accessed) - Index types - clustered columnstore - best for large fact tables; Heap - best for temporary data loading scenario; clustered - best for row based (traditional) lookups
5. implement data redundancy - protecting data from outages by duplicating data on replica stores - locally, zone, geo, geo-zone redundant storage (GzRS)
  1. sql database redundancy - geo replication with manual failover or failover groups(provided by ms)
  2. cosmos db redundancy - multi-master (read+write) global account replication - continuos backup (but it doesnt allow synapse link if continuous backup is enabled)
6. implement distributions - hash, roundrobin, replicated
7. implement data archiving - demo - access tiers - hot cool archive, lifecyle management - demonstrated features, limitations and switching between manual and automatic failover
Implement logical data structures -
1. building a temporal data solution - built change tracking tables ie system versioned table
2. build a slowly changing dimensions -SCD questions - do we need to track the changes? how long of a history do we keep? -
3. Buikd a logical folder structure - ADLS2 provides query acceleration (SDK extensions)
4. build external tables - define tables in azure sql in which data exists in which data exists in external storage like adls gen2 or blob storage - direct reads from external data source - supports plain text and binary source files - this feature is called PolyBase - Create EXTERNAL TABLE - T-SQL statemennts - ms recommends the more concise COPY statement
5. IMplement file and folder structures for efficient querying and data pruning
  1. ingest data into azure synapse sql pools visa extenral data sources
    1. Polybase
    2. COPY
  2. partitions, switch level operations on entire partitions
IMplement the serving layer
1. deliver data in a relational star schema
  1. dedicate vs built in sql pool
  2. foreign key constraints are not supported in synapse sql pools - logic can be simulated by insert update delete triggers and integrity checks (!)
2. Parquet - binary, column based data format - language agnostic -
3. maintain metadata - shared metadata stores are catalogs that track the structure of your data wwarehouses between synapse sql and spark pools. azure db uses apache hive metastore
4. implement a dimensional hierarchy - pipeline (full step by step orchestration) vs dataflow (etl/elt operation)
  1. integration runtime - azure vm needs self hosted, external resource needs self hosted (an agent is needed there)
  2. create source - created dataset object, create new runtime, select integration runtime, complete dataset
  3. create sink - select csv, create new linked service, browse through the file system, name the file in the sink, validate and publish to data factory
Ingest and Transform data
1. Transform data by Apache spark
  1. In memory processing, faster than hadoop (disk i/o slows it)
  2. has two APIs
    1. RDDS - main data structure, supports parallel ops
    2. dataframes - in-mem table structure that support ops
2. Transform data by T-SQL
  1. spark supports ansi-sql
3. Transform data with adf
4. Transform data with synapse - create elt/etl, triggers (manual, schedule, event-driven) - seperate than adf
5. Transform data by stream analytics - AStreamAnalytics-stream processing engine - operate on in-flight data
  1. ingestion options - StreamAnalyticsQL
    1. event hub - general purpose event processror that throttles up or down upto mullion/events per sec
    2. iot hub - specifically for sensors
    3. kafka - competitor of event hub
6. demo - databricks - create a cluster, transform etc
Work with transformed data
1. Cleanse data - deduplication, missing/null values - default values, trailing whitespace, standardize value, remove PII and sensitive data
2. Split data - conditional split adf activity - file split activity in adf (create synapse sql pool tables with distributed method)
3. shred json - parsing json and placing into tables - using spark.read.json(“.json”), also with T-sql with polybase “select from openrowset”
4. Encode and decode data - ascii, utf-8 , synapse sql - collation determines, spark - encode decode methods, also adf using pipelines
Troubleshoot data transformers
1. Configure error handling - failure branching options - failure, skipped, adf sink activity writes error messages to external services 1. normalise (data standardiasation) and denormalise
2. Pivot - create multiple columns from the unique row values in a single column - denormalisation, agregates data
3. Unpivot - unnormalised data into normalise representation 2. Perform EDA - synapse spark pool - right click and load to dataframe, adf has data preview
Design a batch processing solution
1. Develop batch processing using adf, adls, sparl, synapse, polybase, databricks - work best on the basis of triggers - manual, schedule, event driven
2. Create data pipelines - stitch etl/elt processes and triggers -
3. Design and implement incremental data loads - re-use adf linked service
4. Design and develop slowly changing dimensions
5. handle security and compliance - RBAC using azure compliance, azure security benchmark initiative (container that contrains one or more individual policies)-
  1. e.g ky vault secrets shouls have expiration
  2. policy has a json - allowedValues - e,g BuiltIn policy - if account is storage access and it does not contain a network control access list then trigger an effect “Deny”/“Audit”
6. Scale resources - azure abstraction units (per service) event hub = throuhgput unit, adf - data integration unit, stream analytics - streaming unit
7. demo - e.g every 24 hrs csv is added to a blob - should trigger pipeline and add to a table to sql
  1. Copy Data activity - activity is a json
    1. source dataset - blob storage, delimited storeage
    2. create new linked service
    3. sink - azure sql db
    4. create new linked service
    5. run debug, before publish
    6. add trigger - type-storage events, watch “container name”, publish
Develop a batch processing solution
1. COnfigure the batch size - azure batch simplifies large scale HPC batch jobs - [special VM]
  1. Choose VM size and image for workload
  2. number of tasks per VM node
  3. VM size availibility in region
  4. azure batch VM quota limits (soft-raise via support vs hard)
2. azure batch is preferred by “mundane” jobs, data factory, databricks for ETL jobs, VMSS for more complex workloads (higher config effort-load balancer etc)
  1. task - a command line operation - each job contains n number of task
  2. report stdout of task run
3. Data pipeline testing - combine adf with ci/cd and trigger pipelines on commits
4. notebook/pipeline integration - can load jupyter notebook into adf by configuring data sources
5. handle duplicate data - adf aggregate to filter out duplicates
6. handle missing data - during transformation, filter out or replace with substitute
7. Handle late-arriving data - data that arrives during transformation or serving or ingestion - avoid data loss by running adf pipelines on a schedule
Configure a batch processing solution
1. Upsert data - update+insert - sql - if row value exists, update; if row value does not exist, insert ;; for cosmos db - previously were immutable - delete + insert new was used - but now http patch available
2. Regress to previous state
  1. roll back to a stable state - if txn fails, return withdrawal and deposit operation
  2. relational db - core part
  3. nosql - can mimic this behaviour using server-side javascript (in cosmos db) to create atomic txns (!)
  4. adf - can implement logic in adf - verify data consistency verification, skipping incompatible rows and missing files(?) - to mimic transaction and rollback
3. design and config exception
  1. try..catch
  2. azure batch - has specific error codes and logs
  3. event hubs - eventhubsexception
4. Configure batch retention - task retentions is 7 days by default - unless - task is deleted and compute node is removed
5. Revisit design - Lambda archtiecture - combine hot path (streaming data) and cold path (batch data)
6. Debug spark jobs using spark ui - streaming tab - shows related metrics like input rate, scheduling delay, processing time; verify expected count of elements is transferred to sink, provide appropriate mapping
Design a stream processing solution
1. use steam analytics, azure db, event hubs - real time data ingestion, often through lambda architectures.
2. process data using spark structured streaming - incoming streaming data populated into a continuosly growing table (called dataframes)
  1. structured streaming queries will rreturn different results each time theryre run
  2. Writing modes
    1. complete - entire result table written to data sink
    2. append - only new rows written to sink
    3. update - rows that have changed are updated
3. monitor for performance and functional regressions
  1. event hub - incoming outgoing message flows - metrics - utilisation of event ingestion and processing services are to rightsize them
  2. stream analytics - input output events
  3. alerts
    1. signal
    2. condition - triggered based on signal
    3. action group - fired when condition is true (e.g email, message)
  4. design and create windowed aggregates - create groupings and execute evaluation windows -
    1. tumbling, - every 10 seconds tell me the count of tweets per time zone - provide unit, and a number - repeate at a cadence and dont overlap
    2. hopping, - every 5 seconds, give me the count of tweets over the last 10 seconds - provide unit, duration, offset) - if offset = duration hopping is tumbling window - hopping window overlaps
    3. sliding, - alert me every time a topic is mentioned more than 3 times every 10 seconds - provide unit, duration - moves when content of the window actually changes
    4. session, - groups event that arrive at similar times and filters out period of times where there is no data - tell me the count of tweets that occur within 5 seconds of each other
    5. snapshot window - group events that have the same timestamps
  5. handle schema drift - change in event source
4. demo -
  1. event hub - configure throughput units, partitions - similar for IoT hub (add a virtual device) - provide connection string to RPi (IoT device)
    1. cosmos db - sink
    2. stream analytics job - (clusters for grouping) - rquires input (iot hub), function(op[tional logic), query (filter data), output (select cosmos)
    3. select 8 from into [db] from [iothubinput]
    4. start job - will take incoming data into output
    5. unit is streaming units
Process data in a stream processing solutoin
1. Process time series data e.g iot sensor data - time event time(iot hubs are scalable, mostly not a bottleneck), processing time ()
2. Process across partitions - event hub partitions enable service to process multiple data streams - partitions load balance the stream (single) if needed
3. process within one parition - customise processing in application code - EventHubConsumerClient can change how streaming data is handled
4. Configure watermakrs = keep a record of the last timestamp processed by the streaming solution
  1. nosql dbs like cosmos are used as sinks
  2. azure stream analytics does checkpointing periodically by default
5. scale resources - event hubs throughout usnits - 1mb/sec is 1000 events/second - called processing units for premium skus
6. data pipeline tests - adf + ci/cd - checkpoint, rollback, time sampling, window functions activites can be validated
7. optimise pipelines for analytical/transactional purpose
  1. apply file compression
  2. network dealy from source into azure
  3. refactoring user -defind functions (UDFs)
  4. refactoring indexes - sql db, cosmos db (custom indexes) -
    1. indexing policy - automaic indexing (/8 means index every element in the json) - include or exclude paths (certain properties) by modifying a json within each container
    2. event hubs - multiple consumers can connect and consume event data (consumer groups) - azure event hub capture - take a copy of streaming data into blob storage (select time/storage in mbs)
Troubleshoot stream processing solution
1. Handle interruptions
  1. network connectivity issues
  2. planned unplanned maintenance
  3. Bugs
  4. Solution - Azure availability zone - high availability within a region and not across regions
2. upsert data - stream analytics - various compatibility levels
  1. 1.0, 1.1 do a property level insert
  2. 1.2 - supports amqp message protocol
  3. 1.2 > replace document operation
3. Replay archived stream data - even hub stores 7 days worth of data - client libraries can be used to “replay” streams
4. Design a stream processing solution
  1. Phases
    1. Real time ingestion - event hub
    2. stream processing - stream analytics
    3. analytical data store - cosmosdb for nosql allows for mirrored data store (that is transactional)
    4. reporting - powerbi etc
  2. demo - azure event hubs capture and event re-ingestion feature - specify serialisation format (avro, parquet, delta lake) - showed community project
Manage Batches and Pipelines
1. Trigger batches
  1. azure functions - each function can only have one trigger
  2. trigger templates - http, timer, event grid, bus queue
2. Handle failed batch loads - azure batch error, pool error - quota issue, node error - start task failure, job - dependency problem, task error - exit code non-zero
3. Validate batch loads - can verify that file exists before proceeding
4. manage data pipelines - add, edit linked service, configure source control
5. schedule data pipelines triggers - manual, event, time
6. manage spark jobs in a pipeline -
  1. azure databricks, trigger in adf to schedule - demo
    1. create function app - compute layer
      1. blob trigger - execute fn when file arrives
      2. cosmosdb - output of fn app passed here
      3. code - can be tested in the portal
    2. adf
Design security for data policies
1. encryption at rest - data not in use located in azure data centers - storage service encrypytion, Azure keyvault; encryption in transit - TLS/ssl over https
2. design a data auditing strategy
  1. diagnostic setting - log in log analytics, archive in azure storage, stream to event hub
  2. azure sql auditing - all db products have it - who access what when
3. design a data masking strategy - azure sql database dynamic data masking (DDM) - hide sensitive data (admins exempty) default, credit card masks, email masks - no encryption, just app view edited
4. design for data privacy - e discovery, PII - azure sql data discvery and classififcation - apply labels to PII and financial data
Design security for data standard
1. design a data retention policy - safely delete data after its out of scope - azure storage lifecycle mangement policy - access tiers, archive
  1. azure sql database long term policy is for 10 years
2. design to purge data based on business requirements - adf delete activity based on timestamp; TRUNCATE t-sql statement
3. design azure rbac and posix like ACL (access control list) for adls2 - least privilege principle
4. design row-level and column-level security - sql db and synapse pool - T-SQL CREATE SECURITY POLICY for rows; T-SQL GRANT statement applies to columns
5. e,g blob storage - add rules to trim, delete, transition hot→cool, transition to archive after 180 (minimum discount days
6. sql db - always encrypted - right click, encrypt column- generates key that is stored in key vault - need to connect to db with a particular mode “always encrypted” to sql server, enter credentials and then view the decrypted column
Implement data security protection
1. implement data masking - identifying when it is needed (support : “can you validate the card by provided the last 4 digits”)
2. encrypt data at rest and in motion - customer and service managed key - some offer double encryption,
3. implement row level and column level security
4. implement rbac - azure AD allows for MFA; azure AD PIM , EPM - higher permissions
5. implement acl for adls2 - acls allow file and folder level security
6. implement data retention policy
7. implement a data auditing strategy
8. demo - using azure public clouds resources on their defualt domain names we are trusting azures PKIs, some services dont allow bringing our own TLS. azure function - custom domain- demonstrate domain ownership by creating a resource record in dns zone and azure does a look up and we can use it for that azure service. then we use a certificate to create a binding.
Implement data security access
1. manage identities, keys and secrets across data platforms -
  1. azure ad identities (users, groups), app registration service principals, managed identities (system assigned identities that can obtain access tokens on their own behalf like VMs/webapps, free standing user managed identites for pipelines).
  2. In azure key vault - three types of objects -
    1. encryption keys - used primarily with azure data products, for customer managed keys, for at rest data encryption needs
    2. secrets - secure string data like passwords api keys, connection strings
    3. certificates- adding TLS binding and https
2. implement secure endpoints : private and public
  1. azure is a public commercial cloud - public endpoints
  2. private endpoints - private ip in virtual network
  3. hybrid approach - ip access list to screen public internet + service endpoint to integrate for private access
3. implement resource tokens in azure databricks
  1. Personal access token - single token that represents entire identity - need to be protected , expiration date
  2. Azure ad tokens are better
4. load a dataframe with sensitive information - databricks has fernet library to mask data in dataframes
5. write encrypted data to tables or parquet files - can write data in binary format as well
6. manage sensitive information - azure sql database data dscovery and classificaiton
  1. private endpoint - resource firewall (conditional forwarding) , dns forwarding + azure dns resolver
  2. microsoft defender for cloud - recommendations are good - e.g transparent data encryptions should be enabled - standard tier is worth it that are data specific - comparison to compliance benchmark e.g FedRAMP
Monitor data storage
1. Implement logging used by azure monitor -
  1. metrics - time series data
  2. logs - external services, hybrid cloud
  3. log analytics - queried using kusto
2. configure monitoring services
  1. infrastructure monitoring
3. application monitoring - client side latency metrics, who is the visutir
  1. subcription - inventory and cost
  2. tenant - aad succesful anf failed login
  3. data stored in Data Sinks - can be processed further
4. measure performance of data movement
  1. adf monitoring tab - throughput per unit time
  2. pipeline runs
  3. triger runs
  4. adf keeps them for 45 days, log analytics keeps it for 2 yrs
5. monitor and update statistics about data across a system
  1. synapse has cost based optimiser (time)
  2. ALTER DATABASE [] SET AUTO_CREATE_STATISTICS ON;
6. monitor data pipeline performance - monitor tab
7. measure query performance - benchmark sql query performance
8. demo - “performance preview” provides automatic tuning suggestions - delegate some potentially destructive options to sql databse - “force plan”(sql calculates and enforces query execution plan), “create index”, “drop index”
Monitor data processing
1. Monitor cluster processing
2. understand custom logging options
  1. log analytics supports custom logs
  2. install azure monitor agent to monitor remote machines
  3. procedure - upload sample log, customise metadata, can use kusto to monitor hybrid cloud apps
3. schedule and monitor pipeline tests
4. interpret azure monitor metrics and logs - metric explorer, log analytics, alerts based on metrics and logs
5. interpret a spark DAG - db engine creates a plan while running a query to figure out which batch tasks can run in parallel - DAG is a job execution plan - DAG statistics -
Tune data storage
1. Compact small files - data analytics products are optimised for read; Apache spark “bin” packing that improves query performance by packing together small files into large ones ; adf has merge option in the copy activity (small files to single monolithic files)
2. Rewrite usser defined files - refactor repititive scripts into server side UDFs. instead of having external dependencies 1. add logic directly to synapse CREATE FUNCTION T_SQL 2. cosmos db imlrmrnts js files that return scalar single value results
3. handle skew (uneven data distribution) - put thought in partitioning data by chooosing a better partitioning key that results in even distribution of data; at compute layer - enable statistics to improve query plam - let it select higher performing query execution plan
4. handle spill data - might not hold all data in memory and spills to disk
  1. symptom :synapse sql runs out of space in tempDB
  2. solution:
    1. resize compute layer - allocatore more
    2. reduce partition size - e.g quarter instead of a year
5. Tume shuffle partitions - spark shuffles data between nodes - might be preferable to reduce this (to improve trouble shooting e.g)
6. find shuffle in a pipeline - synapse sql and spark have EXPLAIN statements that explain query execution plans
7. optimise resource management - cost optimiseation, pause synapse pools (not billed when deallocated)
8. ps - VMs have bidding mechanisms, stateless ephemeral VMs can be obtained for a high discount, with the downside of being stopped/deallocated if a higher bidder appears
Optimise and troubleshoot data processing
1. tune queries by using indexers
  1. sql pool- CCI is the default, clustered index specifies the column - good for use in where index ; heap index- better for staging (temp) tables
  2. spark - no native index - hyperspace api - works with json csv parquet - api level indexing system - speeds up query resolution
2. tune queries by using cache
  1. sql pool - ALTER DATABASE SET RESULT_SET_CACHING ON
  2. spark - cache() persist() cache intermediate results of RDDs, dataframes and datasets
3. optimse pipelines for analytical or txn purpose - data processing system (oltp,olap)
4. HTAP - hybrid between T and A - synapse link for cosmos db - synchronised copy between txn and analytical

आयुष 🌱

Explorer

ADF, Synapse, CosmosDB, ETL - DP-203

Graph View