An infographic shows Snowflake failures on Azure triggering queued alerts through Logic Apps to engineering teams.
Snowpipe will skip a malformed file and keep loading. That keeps the pipeline alive, but a skipped file can leave your tables short of data with no obvious sign. A data architect writing in Analytics Insight describes a Snowflake-on-Azure design that turns each skipped file into an alert. This piece covers how that design works. It also covers what Snowflake's and Microsoft's documentation say about where it can still fail.

An infographic shows Snowflake failures on Azure triggering queued alerts through Logic Apps to engineering teams. The problem: a pipeline that succeeds while the data is incomplete​

Snowflake's documentation says that by default, Snowpipe doesn't load any file that contains an error (ON_ERROR = SKIP_FILE) and continues to load the other files. Those files show a Load failed status in the copy history. Nothing pages anyone. The author says teams usually found out when BI users or analysts noticed the numbers looked off. The troubleshooting was therefore reactive.

The author's takeaway is that failures which don't break a pipeline can still break the data. A wider point is that "silent" needs qualifying. Snowflake does offer a notification feature, but you have to configure it. It is not on by default.

The architecture the author reports​

The author did not try to change Snowpipe's behavior. Instead they sent the failure signal out of Snowflake. The chain, as the author describes it:

  1. Snowflake sends an outbound notification when a pipe fails a file.
  2. An Azure Event Grid custom topic receives the event.
  3. An Event Grid subscription routes it to an Azure Storage Queue. The author reports putting this queue in the same storage account that hosts the source file container.
  4. An Azure Logic App triggers when messages are in the queue. It parses the JSON and loops over the failures. It then calls an HTTP action to notify the data engineering team, and finally deletes the queue message.

The author gives three reasons for the queue and Event Grid layers:

  • An earlier, more direct integration was tightly coupled, so a downstream failure affected notification delivery.
  • The queue gave them a buffer for bursts of failures.
  • The queue showed how many events were waiting.

These are the team's own findings. The article gives no throughput, latency or failure-rate figures.

Why an HTTP action instead of email​

The team used an HTTP action rather than the built-in Send Email action because they were required to. They send alerts through an internal API or messaging gateway. The author lists the usual reasons for such a rule:

  • Data can leave controlled boundaries through email connectors.
  • Auditing and compliance controls can be bypassed.
  • Email connectors often use user-scoped credentials.
  • Rate limits and content rules are harder to enforce.

Read these as the team's concerns, not as flaws in email connectors. An HTTP action is only as controlled as the endpoint behind it. Authentication, network configuration and logging still have to be built. The design does have one real advantage. Changing the notification channel (email, Teams, an incident system) means changing the internal service and leaves the Logic App alone.

What Snowflake's documentation says about the feature​

Snowflake's documentation backs the central idea. Snowpipe can send a message to your cloud messaging service when a file fails to load. Each message identifies the pipe, the target table and the stage location, and lists the failed files with the first error in each. Snowflake also lists Microsoft Azure Event Grid among the supported destinations.

The setup requirements are specific:

  • The error integration must be an outbound queue notification integration. Email and webhook integrations are not supported for this purpose.
  • The integration is attached to the pipe with the ERROR_INTEGRATION property. Setting it needs OPERATE or OWNERSHIP on the pipe and USAGE on the integration.
  • The role that owns the pipe must keep USAGE on the integration. Snowflake's pipe management documentation warns that without it the pipe keeps loading but stops sending error notifications. Be careful when transferring pipe ownership.
  • Snowflake's error-notification guide describes a test. Stage a file the pipe can't load, such as JSON missing a closing brace, and confirm a notification reaches your topic within a few minutes.

Where the "every failure alerts" claim breaks down​

The article says every Snowpipe failure should produce an event. Snowflake's documentation is narrower. The notification depends on your ON_ERROR setting:

  • With the default SKIP_FILE, you get a notification for every file that contains an error. Snowflake says to get a notification for every file that contains an error, use the default setting, ON_ERROR = SKIP_FILE.
  • With CONTINUE, Snowpipe loads the valid rows and sends a notification only if no rows from the file load. A partly loaded file therefore raises no alert.
  • With SKIPFILE<num> or SKIPFILE<num>%, Snowpipe sends a notification only for files that reach the error limit.

Snowflake's documentation also lists problems that error notifications don't report:

  • Files that fail after repeated internal errors. Check the copy history for those.
  • A pipe that stops or stalls, or a notification channel Snowflake can't read. Snowflake suggests recording Snowpipe events in an event table and alerting on them.
  • Files that never reach Snowpipe. If storage events stop arriving, no file fails, so no notification is sent. Snowflake suggests a scheduled alert that calls SYSTEM$PIPE_STATUS and checks how long ago the pipe last loaded a file.

For Azure, the troubleshooting guide points to fields such as lastReceivedMessageTimestamp and channelErrorMessage. So the Logic App design covers one class of failure, files that failed to load. A complete setup also needs alerts for stalled pipes and missing upstream events.

Reliability caveats for the Event Grid and queue layers​

The author calls the queue a reliability layer. It helps, but Microsoft's documentation shows limits.

Event Grid delivery is best-effort. Microsoft says Event Grid retries delivery on an exponential backoff schedule, starting at 10 seconds and stretching to every 12 hours up to 24 hours. It adds that it may skip some retries when an endpoint looks consistently unhealthy. Some configuration errors, such as a deleted endpoint, are not retried. If dead-lettering isn't configured, those events are dropped. Dead-lettering is off by default and needs a storage account you specify when creating the subscription. Check this setting on the subscription that feeds your queue.

Queue retrieval isn't deletion. Azure Queue Storage hides a retrieved message for a visibility timeout, 30 seconds by default. The message stays until the consumer deletes it with a valid pop receipt. If processing doesn't finish in time, the message becomes visible again. Microsoft's Azure Queues connector says the same of its Logic Apps trigger. It returns up to 32 messages that remain on the queue until the delete action is used. The author's final "delete after success" step is therefore the right pattern. It also means a slow or failing run can cause the same alert to be sent again.

Duplicates are expected. Snowflake delivers each notification at least once, so a consumer may get repeats. The documented fix is to use the messageId field to ignore duplicates. Event Grid can also produce duplicates. Treat the Logic App as something that must tolerate repeated messages, for example by tracking message IDs.

Payload parsing. The author reports that raw messages were inconsistent enough that strict JSON parsing and schema normalization were necessary. Snowflake documents fields such as messageId, messageType, timestamp, pipeName, tableName and a messages array with the file name and first error for each failed file. Note too that Snowflake changed some Azure Event Grid notification fields (subject and eventType) in its 2024_01 behavior change bundle. Parsers written before that change may need checking.

A practical checklist​

If you are building something similar, the documentation points to these checks:

  1. Confirm the pipe's ON_ERROR setting matches the alerting you expect.
  2. Confirm the pipe has an error integration. DESCRIBE PIPE shows it.
  3. Confirm the integration is enabled and the pipe owner holds USAGE on it.
  4. Query NOTIFICATION_HISTORY if messages don't arrive. FAILURE means Snowflake gave up sending. SUCCESS means Snowflake published to the topic, so check the Event Grid subscription and your consumers.
  5. Configure dead-lettering and watch delivery status on the Event Grid subscription.
  6. Monitor queue depth. The author notes the queue gave visibility into pending events.
  7. Make the Logic App idempotent, keyed on messageId.
  8. Alert on the Logic App's own failures. The author's advice is to use the Alerts section under the Monitoring tab. The author's phrase for it is "monitor the monitor."
  9. Add separate checks for pipe health and for absent upstream events.

Remediation still needs a plan​

An alert tells you a file failed. It does not load the file. Snowflake's pipe management guidance covers recovery:

  • ALTER PIPE ... REFRESH can queue files staged within the last 7 days. It treats failed files as processed, so it won't reload them.
  • Snowpipe ignores a file whose path and name match its load metadata, including failed files. A corrected file staged under the same name within 14 days is ignored. Stage it under a new name or load it with COPY INTO.
  • Before reloading a file that partly loaded, delete its rows from the target table first, or the reload duplicates them.

The trade-offs​

The author names the trade-off honestly. Letting Snowpipe continue keeps ingestion available, but it moves the responsibility for data integrity to the surrounding layers. The author says the design is a poor fit for ultra-low-latency notification, very high event volume, or complex cross-system enrichment. Those are the team's own judgments. They are not product limits from Snowflake or Microsoft.

There is also a question of weight. Logic Apps, Event Grid and a queue are several components for one alert. The author admits the added configuration. The alternative Snowflake itself documents is an alert on Snowpipe events, which can send email without this stack. That may suit smaller shops that don't need the HTTP-gateway governance.

One housekeeping note: the article's byline and author biography name different people, and this piece doesn't resolve that. The architecture outcomes are the author's account of their own deployment. They are not independently verified.

Bottom line​

The core lesson holds up against the documentation. Snowpipe's resilience means a skipped file won't announce itself. The Snowflake-to-Event-Grid-to-queue-to-Logic-App chain is a workable way to catch those files, as long as you pin down ON_ERROR, enable dead-lettering, handle duplicates, and add health checks for the failures that never generate a notification.

 

References

  1. Snowpipe error notifications | Snowflake Documentation docs.snowflake.com
  2. Manage Snowpipe | Snowflake Documentation docs.snowflake.com
  3. Azure Event Grid Delivery and Retry Explained - Azure Event Grid | Microsoft Learn learn.microsoft.com