We are experiencing an issue while polling the Amazon Ads API to check the status of reports. The polling mechanism was working fine before, but for the last 3 days, reports are not completing on time, and our retry logic stops before fetching the data. ** API & Retry Mechanism Details:** API Endpoint: /:reportId Actual Behavior: Since the last 3 days, reports remain stuck in PENDING for an unusually long time.
Expected Behavior: Reports should eventually move from PENDING to COMPLETED, allowing us to retrieve the data.
**Error Log: Amazon Ads API Logs **
Timestamp: 2025-01-31T14:29:22.724+05:30
{
"timestamp": "2025-01-31 08:59:22,724",
"level": "INFO",
"message": "Attempting to execute process to get reports.",
"logger": "ingestion",
"func_name": "execute",
"line_no": 256,
"file_name": "amazon_ads.py",
"thread_name": "MainThread",
"process": 1
}
Timestamp: 2025-01-31T14:29:22.724+05:30
{
"timestamp": "2025-01-31 08:59:22,724",
"level": "INFO",
"message": "Attempting to generate access token.",
"logger": "ingestion",
"func_name": "get_access_token",
"line_no": 83,
"file_name": "amazon_ads.py",
"thread_name": "MainThread",
"process": 1
}
Timestamp: 2025-01-31T14:29:22.812+05:30
{
"timestamp": "2025-01-31 08:59:22,812",
"level": "INFO",
"message": "Iteration started",
"logger": "ingestion",
"func_name": "retry_until_condition_satisfied_reportVersion",
"line_no": 203,
"file_name": "request_util.py",
"thread_name": "MainThread",
"process": 1
}
Timestamp: 2025-01-31T14:29:22.947+05:30
{
"timestamp": "2025-01-31 08:59:22,946",
"level": "INFO",
"message": "Reports are not yet completed at server end.",
"logger": "ingestion",
"func_name": "retry_until_condition_satisfied_reportVersion",
"line_no": 209,
"file_name": "request_util.py",
"thread_name": "MainThread",
"process": 1
}
Retry Logic Using tenacity:
@classmethod
@tenacity.retry(
retry=tenacity.retry_if_exception_type(RetryConditionUnsatisfiedError)
or tenacity.retry_if_exception_type(RetryException),
stop=tenacity.stop_after_attempt(MAX_RETRY_ATTEMPT),
wait=tenacity.wait_exponential(multiplier=MULTIPLIER, max=MAX_WAIT_TIME),
reraise=False,
retry_error_callback=on_retry_error,
)
def retry_until_condition_satisfied(
cls,
method: str,
url: str,
condition: t.Callable[..., bool],
params: t.Dict[str, t.Any] = {},
retry_error_codes: t.List[int] = [],
) -> t.Any:
logger.info("Iteration started")
response = requests.request(method, url, **params)
if response.status_code in API_SUCCESS_STATUS_CODES:
if condition(response=response):
return response
logger.info("Reports are not yet completed at server end.")
raise RetryConditionUnsatisfiedError(
{
"errorMessage": "Reports are not yet completed at server end.",
"errorCode": response.status_code,
"response": response,
},
)
if len(retry_error_codes) > 0:
RETRY_ERRORS.extend(retry_error_codes)
if response.status_code in RETRY_ERRORS:
raise RetryException(
{
"errorCode": response.status_code,
"response": response,
"errorMessage": f"API failed with {response.status_code} :: Error Text: {response.text}",
},
)
raise RequestError({"errorCode": response.status_code, "response": response})
** Issues Faced in the Last 3 Days:**
Reports No Longer Move to COMPLETED: Previously, the function retried until the report was ready. Now, the report remains in PENDING, and retries fail.
Possible Amazon API Changes? We haven't changed anything in our implementation. Has Amazon changed report processing times or introduced new limitations?
Unclear Wait Time Impact on Retries: Our settings should allow retries with an exponential backoff up to 244 seconds max. Could something in tenacity’s retry mechanism be preventing it from retrying long enough?