Getting form results using python request - Stack Overflow

admin2025-04-21 2

Admittedly, this is my first time using requests. The form I'm attempting to use is here: /

Here is my code:

import requests

headers = {
    'Accept': 'application/json, text/javascript, */*; q=0.01',
    'Content-Type': 'application/json; charset=UTF-8',
    'User-Agent': 'Mozilla/5.0',
    'X-Requested-With': 'XMLHttpRequest'
}


payload = {
        "dtPageVars":
        {
            "draw":1,
            "columns":
            [
                {"data":"postedDate","name":"","searchable":"true","orderable":"false","search":{"value":"","regex":"false"}},
                {"data":"eventDate","name":"","searchable":"true","orderable":"false","search":{"value":"","regex":"false"}},
                {"data":"voidedDate","name":"","searchable":"true","orderable":"false","search":{"value":"","regex":"false"}},
                {"data":"event","name":"","searchable":"true","orderable":"false","search":{"value":"","regex":"false"}},
                {"data":"basis","name":"","searchable":"true","orderable":"false","search":{"value":"","regex":"false"}},
                {"data":"action","name":"","searchable":"true","orderable":"false","search":{"value":"","regex":"false"}},
                {"data":"entryNumber","name":"","searchable":"true","orderable":"false","search":{"value":"","regex":"false"}},
                {"data":"portOfEntry","name":"","searchable":"true","orderable":"false","search":{"value":"","regex":"false"}},
                {"data":"entryDate","name":"","searchable":"true","orderable":"false","search":{"value":"","regex":"false"}},
                {"data":"entryType","name":"","searchable":"true","orderable":"false","search":{"value":"","regex":"false"}},
                {"data":"teamNumber","name":"","searchable":"true","orderable":"false","search":{"value":"","regex":"false"}}
            ],
            "order":[],
            "start":0,
            "length":100,
            "search":
            {
                "value":"",
                "regex":"false"
            }
        },
        "searchFields":
        {
            "portOfEntry":"0101",
            "entryType":"01"
        }
}

url = '/'
searchUrl = ''

response = requests.post(searchUrl, headers=headers, data=payload)

print(response.status_code)

I keep getting 500 status code but I am not sure why. Could it be the headers I'm using? I've tried many different combinations of headers without luck. Is it possible to use requests for this form or am I better off finding a different approach?

Any help would truly be appreciated. Thank you.

Admittedly, this is my first time using requests. The form I'm attempting to use is here: https://trade.cbp.dhs.gov/ace/liquidation/LBNotice/

Here is my code:

import requests

headers = {
    'Accept': 'application/json, text/javascript, */*; q=0.01',
    'Content-Type': 'application/json; charset=UTF-8',
    'User-Agent': 'Mozilla/5.0',
    'X-Requested-With': 'XMLHttpRequest'
}


payload = {
        "dtPageVars":
        {
            "draw":1,
            "columns":
            [
                {"data":"postedDate","name":"","searchable":"true","orderable":"false","search":{"value":"","regex":"false"}},
                {"data":"eventDate","name":"","searchable":"true","orderable":"false","search":{"value":"","regex":"false"}},
                {"data":"voidedDate","name":"","searchable":"true","orderable":"false","search":{"value":"","regex":"false"}},
                {"data":"event","name":"","searchable":"true","orderable":"false","search":{"value":"","regex":"false"}},
                {"data":"basis","name":"","searchable":"true","orderable":"false","search":{"value":"","regex":"false"}},
                {"data":"action","name":"","searchable":"true","orderable":"false","search":{"value":"","regex":"false"}},
                {"data":"entryNumber","name":"","searchable":"true","orderable":"false","search":{"value":"","regex":"false"}},
                {"data":"portOfEntry","name":"","searchable":"true","orderable":"false","search":{"value":"","regex":"false"}},
                {"data":"entryDate","name":"","searchable":"true","orderable":"false","search":{"value":"","regex":"false"}},
                {"data":"entryType","name":"","searchable":"true","orderable":"false","search":{"value":"","regex":"false"}},
                {"data":"teamNumber","name":"","searchable":"true","orderable":"false","search":{"value":"","regex":"false"}}
            ],
            "order":[],
            "start":0,
            "length":100,
            "search":
            {
                "value":"",
                "regex":"false"
            }
        },
        "searchFields":
        {
            "portOfEntry":"0101",
            "entryType":"01"
        }
}

url = 'https://trade.cbp.dhs.gov/ace/liquidation/LBNotice/'
searchUrl = 'https://trade.cbp.dhs.gov/ace/liquidation/LBNotice/search'

response = requests.post(searchUrl, headers=headers, data=payload)

print(response.status_code)

Any help would truly be appreciated. Thank you.

Share Improve this question asked Jan 22 at 23:20 Ryne 585 bronze badges

Where did you get that payload? Did you enter a search and capture the results? Was the data in JSON form? There are a lot of ways to send POST data, but you have to help requests do the right thing. – Tim Roberts Commented Jan 22 at 23:55
Please also provide the error you see int the logs. 500 can be for many reasons. Also I suggest you to make a request manually in browser and check what goes into request data in Network tab for the request to check the discrepancies with your request – barbariania Commented Jan 22 at 23:57
Yes, I did get my payload from the network tab. It was in JSON just as I have it in my code. The status code I get is just 500. Which I know is not very useful lol. I was hoping I was doing something just clearly wrong. Or there is something specific I need to do to get a proper response from this specific form. At this point, I would be so happy to see anything other than 500. – Ryne Commented Jan 23 at 2:03

Add a comment |

2 Answers 2

Sorted by: Reset to default 1

Cause

In headers you are telling the server you are sending it a JSON data but in request body, you are sending Form data. There is inconsistency.

Solution: send json data

response = requests.post(url, json=payload)

Remove headers too.

Explanation

In requests library, if you pass a dictionary to data parameter, your dictionary of data will automatically be form-encoded when the request is made. But based on your header, it seems you want to send a JSON data: 'Content-Type': 'application/json; charset=UTF-8',

headers = {
    'Accept': 'application/json, text/javascript, */*; q=0.01',
    'Content-Type': 'application/json; charset=UTF-8',
    'User-Agent': 'Mozilla/5.0',
    'X-Requested-With': 'XMLHttpRequest'
}

So, basically In headers you are telling the server you are sending it a JSON data but in request body, you are sending Form data. If you don't want to send a form data (if you want to send JSON or just string data), you can send a string data directly.

If you pass in a string instead of a dict, that data will be posted directly.

requests.post(url, data=json.dumps(payload))

json.dumps(payload) essentially turns your dictionary into (JSON) string. But it doesn't change/set headers. You need to set headers manually.

There is an easier way:

If you need a header set to application/json and you don’t want to encode the dict yourself, you can pass the dcit to json parameter of requests.post method instead of data parameter.

response = requests.post(url, json=payload)

It will both update/set your headers to application/json and stringifies your dictionary.

Tip

If you do not need to specify custom headers, you don't need to add headers to the request, i.e., you can go with default options.

For the site you're accessing, you don't need to provide any headers.

The keyword argument to requests.post should be json and not data. This requests documentation will help you to understand why this is the case.

You can also reduce the amount of code by dynamically constructing the payload.

This will give you your desired output:

import requests
import json

payload = {
    "dtPageVars": {"draw": 1, "order": [], "start": 0, "length": 100},
    "searchFields": {"portOfEntry": "0101", "entryType": "01"},
}
search = {"value": "", "regex": "false"}
dtypes = [
    "postedDate",
    "eventDate",
    "voidedDate",
    "event",
    "basis",
    "action",
    "entryNumber",
    "portOfEntry",
    "entryDate",
    "entryType",
    "teamNumber",
]
payload["dtPageVars"]["search"] = search
payload["dtPageVars"]["columns"] = [
    {
        "data": dt,
        "name": "",
        "searchable": "true",
        "orderable": "false",
        "search": search,
    }
    for dt in dtypes
]

searchUrl = "https://trade.cbp.dhs.gov/ace/liquidation/LBNotice/search"

with requests.post(searchUrl, json=payload) as response:
    response.raise_for_status()
    print(json.dumps(response.json(), indent=2))

转载请注明原文地址:http://anycun.com/QandA/1745223841a90433.html

Getting form results using python request - Stack Overflow

2 Answers 2

Cause

Explanation

Tip

Getting form results using python requestStack Overflow