I need to monitor the Integration Runtime performance for a pipeline that uses stored procedure activity to load data from ADLS to Snowflake. The stored procedure in this case resides in snowflake itself and it’s called from Snowflake Tenant to read parquet files and load to Iceberg tables.
just like Mapping Data Flows and Copy Data Activities provide details of Data read, write and throughput respectively I need to see the performance in this case.
I have diagnostics settings enabled which at max captures the Queue time for activities in a pipeline.
Any sort of suggestions/workarounds appreciated
Thanks
just like Mapping Data Flows and Copy Data Activities provide details of Data read, write and throughput respectively I need to see the performance in this case.
I have diagnostics settings enabled which at max captures the Queue time for activities in a pipeline.
I need to monitor the Integration Runtime performance for a pipeline that uses stored procedure activity to load data from ADLS to Snowflake. The stored procedure in this case resides in snowflake itself and it’s called from Snowflake Tenant to read parquet files and load to Iceberg tables.
just like Mapping Data Flows and Copy Data Activities provide details of Data read, write and throughput respectively I need to see the performance in this case.
I have diagnostics settings enabled which at max captures the Queue time for activities in a pipeline.
Any sort of suggestions/workarounds appreciated
Thanks
just like Mapping Data Flows and Copy Data Activities provide details of Data read, write and throughput respectively I need to see the performance in this case.
I have diagnostics settings enabled which at max captures the Queue time for activities in a pipeline.
The performance details in the copy activity or dataflow activity are the properties of those activities in ADF.
Currently, Stored procedure activity doesn't have the monitoring properties for the performance and throughput. AFAIK, the above activities are data movement activities in which the transformations or the copy will be done using IR cores in ADF. Coming to stored procedure, ADF only calls the stored procedure with required values, but the actual script and the result will reside in the external Snowflake database. This might be the reason that stored procedure activity doesn't have this property in ADF.
But you can monitor the Integration run time duration on the pipeline activities as shown below.
Go to Monitor -> Integration run times -> Select integration run time -> activities.
Here, you can see the pipeline run duration and duration deviation.
Note that this can only done on the triggered pipeline runs and not on the debug pipeline runs.
