Our Documentation

Start implementing Smarketize into your projects with ease! All by using our documentation.

Getting Started

Welcome to the official Smarketize integration documentation. Our aim is to make this as simple as possible. Web scraping shouldn't be hard and neither should our implementation.

Please note our documentaion will be updated and improved with time. If you're having problems and unable to resolve the issue please contact our support.

To get started you'll need to know four things:

  • Your API key
  • The URL you want to scrape
  • Do you want to render JS?
  • Do you want to use a premium/residential proxy?

Don't know your API key? Access the user dashboard. You'll find your API key here. api key example Copy it and integrate it into your application. Keep this key private and secure.
How to do this is beyond the scope of this documentation. A brief example from Twilio by using environmental variables can be watched here.

Now that's out of the way you'll need to encode your target URL. This is extremely important and the API will not work if you do not. The encoding method required is a Base64 "url-friendly" version. You need to convert the target url to Base64 and then eliminate multiple characters ['/', '+', '='] and replace them ['_', '-', '']. 😅 Thankfully, we do provide examples for multiple programming languages.

Now you'll simply send a GET request to our API. Adding the required url, render and premium parameters.


https://api.smarketize.net/api/browser/api_key=sSi229Sklasdl213jasd812398&url=aHR0cHM6Ly9zbWFya2V0aXplLm5ldA&render=true&premium=true
            

And you'll be given the rendered response code.

<html>
<head>
<title>...</title>
...
</head>
<body>
    ...
</body>
</html>

If the request was unsuccessful a JSON response will be returned. You can check if a request is invalid if it contains a "success" key. Here are some examples of the errors you will encounter.

You are NOT charged for unsuccessful requests.

{"success": false, "response": "The reason the request failed will be listed here"}
{"success": false, "response": "Incorrect API key provided."}
{"success": false, "response": "There was an error fetching the website: https://smarketize.net Please try again. You have NOT been charged for this."} // Typically means the proxy or website timed out. Try again.

Didn't get a response?

404? You forgot a parameter or are not using the correct API url.

Please note this is a brief overview. You read more about encoding, proxies and the rendering options on our documentation.

Encoding Your URL

As stated earlier you'll need to encode your URL. An example is if we encode https://smarketize.net into our "url-friendly" format it will return: aHR0cHM6Ly9zbWFya2V0aXplLm5ldA

Don't worry, you have two options:

  1. Encode the target url on our user dashboard
  2. Encode the target url yourself

Listed below you'll find the examples for your programming language.

Python

An example of Base64 encoding and creating a URL friendly version for Python.


import base64

url = "https://smarketize.net"

def base64url_encode():
    base64url = base64.urlsafe_b64encode(url.encode()).decode()
    base64url = base64url.replace("=", "")
    return base64url
                    
NodeJS

An example of Base64 encoding and creating a URL friendly version for NodeJS.


let url = "https://smarketize.net";

function base64EncodeUrl() {
    // Buffer may be depreciated use with caution
    let buffer = new Buffer(url);
    let text = buffer.toString('base64');
    // Make the encoding "url-safe"
    // Could be fully regex if preferred
    text = text.replace("/+/g", "-").replace("/=/g", "").split("/").join("_");
    return text;
}
                                
PHP

An example of Base64 encoding and creating a URL friendly version for PHP.


$url = "https://smarketize.net";

function base64url_encode() {
    $b64string = base64_encode($url);
    // Converts the base64 to base64url safe characters
    $url = strtr($b64string, "+/", "-_");
    return rtrim($url, "="); // Needs to be removed when in a URL
}
                    
C#

An example of Base64 encoding and creating a URL friendly version for C#.


private string url = "https://smarketize.net";

private static string EncodeWebsiteURL()
{
    var inputBytes = System.Text.Encoding.UTF8.GetBytes(url);
    // Converts a "url-safe" base64 encoding
    return Convert.ToBase64String(inputBytes)
    .Replace("+", "-")
    .Replace("/", "_")
    .Replace("=", "");
}
                    
Java

An example of Base64 encoding and creating a URL friendly version for Java.


private String url = "https://smarketize.net";

private static String base64EncodeUrl() {
    return Base64.getUrlEncoder()
        .withoutPadding()
        .encodeToString(url.getBytes(StandardCharsets.UTF_8));
}
                    

Please note all these are just examples. The implementation could be improved. You're also able to implement your own functions or existing ones. The goal of these functions is to replace /, +, and = from the Base64 encoded URL.

Final Options

Well, you're basically just about finished with our current documentation. The only thing left is to decide on whether you're wanting to render javascript and use a residential(premium) proxy or not. We also quickly break down the costs for your requests.

To send a request to the Smarketize API you will use the following format: https://api.smarketize.net/api/browser/api_key={YOUR_API_KEY}&url={ENCODED_URL}&render={RENDER_JAVASCRIPT}&premium={USE_RESIDENTIAL_PROXY}

You would replace the {parameters} with your own values. For example, if I wanted to fetch the Smarketize website including the rendered Javascript and fetch the target url with a residential (premium) proxy:
https://api.smarketize.net/api/browser/api_key=sSi229Sklasdl213jasd812398&url=aHR0cHM6Ly9zbWFya2V0aXplLm5ldA&render=true&premium=true
The amount of credits for this request: 25

If you wanted to render the websites Javascript but NOT use a residential (premium) proxy:
https://api.smarketize.net/api/browser/api_key=sSi229Sklasdl213jasd812398&url=aHR0cHM6Ly9zbWFya2V0aXplLm5ldA&render=true&premium=false
The amount of credits for this request: 10

If you wanted to render the websites WITHOUT Javascript and NOT use a residential (premium) proxy:
https://api.smarketize.net/api/browser/api_key=sSi229Sklasdl213jasd812398&url=aHR0cHM6Ly9zbWFya2V0aXplLm5ldA&render=false&premium=false
The amount of credits for this request: 1

If you're looking for an easier way to calculate costs and encode a URL you can always use our Request Builder on our user dashboard. Smarketize Request Builder

Rendering Javascript

The render parameter in a GET request is required.

To render a website that uses a lot of Javascript you will use the render=true parameter in your GET request.
If you're not wanting to render the JS on a website you need use the render=false parameter in your GET request.

Proxy Options

The premium parameter in a GET request is required.

To fetch a website with a residential (premium) proxy you will use the premium=true parameter in your GET request.
If you're not wanting to use a residential (premium) proxy you need use the render=false parameter in your GET request.
When you're not using a premium proxy you are using a proxy from our datacenter pool. This pool is much smaller at the moment. If you're experiencing blocks while scraping we highly recommend you use the premium proxies.

Right now our documentation and features are bound to change. If you're experiencing issues please check back here or contact our support for assistance.