amazon web services - AWS Lambda Python Segmentation Fault on S3 File Upload - Stack Overflow

I am encountering a segmentation fault while running an AWS Lambda function The function fetches data from multiple sources, processes it, and uploads the results to an S3 bucket The data fetching and processing is good, but the function crashes during the file upload step with the following error:

RequestId: 11692707-3f6a-49be-b829-62a0e9e0b18c Error: Runtime exited with error: signal: segmentation fault
Runtime.ExitError
END RequestId: 11692707-3f6a-49be-b829-62a0e9e0b18c
REPORT RequestId: 11692707-3f6a-49be-b829-62a0e9e0b18c  Duration: 1193.46 ms    Billed Duration: 1194 ms    Memory Size: 2056 MB    Max Memory Used: 249 MB Init Duration: 1350.67 ms   
XRAY TraceId: 1-6787fda3-0501431b1684b42e1bb76bcd   SegmentId: 5dee1c4d067cb6b2 Sampled: true

Lambda memory allocation: 2056 MB

Runtime: Python 3.9

The Lambda logs show that credentials are correctly loaded from the environment variables, and the upload begins, but it crashes shortly after. I’ve tried increasing memory allocation and execution time without success

def upload_to_s3(dataframe, bucket_name, file_name):
    """Upload DataFrame as CSV to S3."""
    logger.info(f"Uploading file to S3 bucket {bucket_name} with key {file_name}...")
    retries = 3
    for attempt in range(retries):
        try:
            with tempfile.NamedTemporaryFile(delete=False) as temp_file:
                dataframe.to_csv(temp_file.name, index=False)
                temp_file.seek(0)
                s3 = boto3.client('s3')
                with open(temp_file.name, 'rb') as f:
                    s3.put_object(Bucket=bucket_name, Key=file_name, Body=f)
            os.remove(temp_file.name)
            logger.info(f"File uploaded to S3: {bucket_name}/{file_name}")
            break
        except ClientError as e:
            if attempt == retries - 1:
                logger.error(f"S3 upload failed after {retries} attempts: {e}")
                raise
            else:
                logger.warning(f"S3 upload attempt {attempt + 1} failed, retrying...")
                time.sleep(2)

Any insights on diagnosing or resolving this segmentation fault during S3 uploads? Could it be related to boto3, Lambda environment limits, or a dependency issue?

Any help is appreciated.

I am encountering a segmentation fault while running an AWS Lambda function The function fetches data from multiple sources, processes it, and uploads the results to an S3 bucket The data fetching and processing is good, but the function crashes during the file upload step with the following error:

RequestId: 11692707-3f6a-49be-b829-62a0e9e0b18c Error: Runtime exited with error: signal: segmentation fault
Runtime.ExitError
END RequestId: 11692707-3f6a-49be-b829-62a0e9e0b18c
REPORT RequestId: 11692707-3f6a-49be-b829-62a0e9e0b18c  Duration: 1193.46 ms    Billed Duration: 1194 ms    Memory Size: 2056 MB    Max Memory Used: 249 MB Init Duration: 1350.67 ms   
XRAY TraceId: 1-6787fda3-0501431b1684b42e1bb76bcd   SegmentId: 5dee1c4d067cb6b2 Sampled: true

Lambda memory allocation: 2056 MB

Runtime: Python 3.9

The Lambda logs show that credentials are correctly loaded from the environment variables, and the upload begins, but it crashes shortly after. I’ve tried increasing memory allocation and execution time without success

def upload_to_s3(dataframe, bucket_name, file_name):
    """Upload DataFrame as CSV to S3."""
    logger.info(f"Uploading file to S3 bucket {bucket_name} with key {file_name}...")
    retries = 3
    for attempt in range(retries):
        try:
            with tempfile.NamedTemporaryFile(delete=False) as temp_file:
                dataframe.to_csv(temp_file.name, index=False)
                temp_file.seek(0)
                s3 = boto3.client('s3')
                with open(temp_file.name, 'rb') as f:
                    s3.put_object(Bucket=bucket_name, Key=file_name, Body=f)
            os.remove(temp_file.name)
            logger.info(f"File uploaded to S3: {bucket_name}/{file_name}")
            break
        except ClientError as e:
            if attempt == retries - 1:
                logger.error(f"S3 upload failed after {retries} attempts: {e}")
                raise
            else:
                logger.warning(f"S3 upload attempt {attempt + 1} failed, retrying...")
                time.sleep(2)

Any insights on diagnosing or resolving this segmentation fault during S3 uploads? Could it be related to boto3, Lambda environment limits, or a dependency issue?

Any help is appreciated.

• python
• amazon-web-services
• amazon-s3
• aws-lambda
• segmentation-fault

Share Improve this question Follow edited Jan 16 at 15:12 aaa 65522 gold badges22 silver badges1414 bronze badges asked Jan 16 at 13:48 GNANESHWAR REDDY YENNAGNANESHWAR REDDY YENNA 2111 silver badge33 bronze badges 4

• What if you try a newer Python runtime? – stdunbar Commented Jan 16 at 22:09
• Yeah I have tried with different runtimes... – GNANESHWAR REDDY YENNA Commented Jan 17 at 8:09
• Instead I used requests package... But if anyone can address it, reach me out. – GNANESHWAR REDDY YENNA Commented Jan 17 at 8:11
• You should add logging to understand which call is triggering the segfault (is it Pandas to_csv or boto3 put_object) but I suspect the to_csv call and this might be caused by problematic data in the dataframe (example of that here). – jarmod Commented Jan 29 at 16:33

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 0

Have you checked what your csv file sizes are?

Since the entire CSV file is being written to /tmp before being read back for upload, there is a possibility that the temp storage is getting exceeded. By default lambda gives you 512 MB of storage but it can go upto 10 GB now. Lambda Ephemeral Storage Update