как обойти JavaScript is required при скрапинге?
сайт bsky.app требует включить js при отправке запроса, как это можно обойти? пробовал cloudscraper, не помогло...
headers = {
"cache-control": "max-age=0",
"sec-ch-ua": '"Google Chrome";v="131", "Chromium";v="131", "Not_A Brand";v="24"',
"sec-ch-ua-mobile": "?0",
"sec-ch-ua-platform": '"Windows"',
"origin": "https://bsky.social",
"content-type": "application/x-www-form-urlencoded",
"user-agent": resp_solver.get("user_agent", "Mozilla/5.0"),
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8",
"sec-fetch-site": "same-origin",
"sec-fetch-mode": "navigate",
"sec-fetch-user": "?1",
"sec-fetch-dest": "iframe",
"referer": "https://bsky.social/gate/signup?handle=example.bsky.social&state=-Y05tkuEQhfUUFw&colorScheme=dim",
"accept-encoding": "gzip, deflate, br",
"accept-language": "ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7"
}
data = {
"g-recaptcha-response": resp_solver["solution"],
"h-captcha-response": resp_solver["solution"]
}
response = scraper.post(
"https://bsky.social/gate/signup?handle=fesefg31res.bsky.social&state=-Y111111111&colorScheme=dim",
headers=headers,
data=data
)
и получаю такой ответ HTML: (часть ответа)
<noscript>
<h1 lang="en">JavaScript Required</h1>
<p lang="en">This is a heavily interactive web application, and JavaScript is required. Simple HTML interfaces are possible, but that is not what this is.
<p lang="en">Learn more about Bluesky at <a href="https://bsky.social">bsky.social</a> and <a href="https://atproto.com">atproto.com</a>.
<p>This is the home page.
</noscript>