The X Twitter Scraper Java SDK provides convenient access to the X Twitter Scraper REST API from applications written in Java.
The X Twitter Scraper Java SDK is similar to the X Twitter Scraper Kotlin SDK but with minor differences that make it more ergonomic for use in Java, such as Optional instead of nullable values, Stream instead of Sequence, and CompletableFuture instead of suspend functions.
It is generated with Stainless.
The REST API documentation can be found on xquik.com. Javadocs are available on javadoc.io.
implementation("com.x_twitter_scraper.api:x-twitter-scraper-java:0.2.0")<dependency>
<groupId>com.x_twitter_scraper.api</groupId>
<artifactId>x-twitter-scraper-java</artifactId>
<version>0.2.0</version>
</dependency>This library requires Java 8 or later.
import com.x_twitter_scraper.api.client.XTwitterScraperClient;
import com.x_twitter_scraper.api.client.okhttp.XTwitterScraperOkHttpClient;
import com.x_twitter_scraper.api.models.x.tweets.TweetSearchParams;
import com.x_twitter_scraper.api.models.x.tweets.TweetSearchResponse;
// Configures using the `xtwitterscraper.apiKey`, `xtwitterscraper.bearerToken` and `xtwitterscraper.baseUrl` system properties
// Or configures using the `X_TWITTER_SCRAPER_API_KEY`, `X_TWITTER_SCRAPER_BEARER_TOKEN` and `X_TWITTER_SCRAPER_BASE_URL` environment variables
XTwitterScraperClient client = XTwitterScraperOkHttpClient.fromEnv();
TweetSearchParams params = TweetSearchParams.builder()
.q("from:elonmusk")
.limit(10L)
.build();
TweetSearchResponse response = client.x().tweets().search(params);Configure the client using system properties or environment variables:
import com.x_twitter_scraper.api.client.XTwitterScraperClient;
import com.x_twitter_scraper.api.client.okhttp.XTwitterScraperOkHttpClient;
// Configures using the `xtwitterscraper.apiKey`, `xtwitterscraper.bearerToken` and `xtwitterscraper.baseUrl` system properties
// Or configures using the `X_TWITTER_SCRAPER_API_KEY`, `X_TWITTER_SCRAPER_BEARER_TOKEN` and `X_TWITTER_SCRAPER_BASE_URL` environment variables
XTwitterScraperClient client = XTwitterScraperOkHttpClient.fromEnv();Or manually:
import com.x_twitter_scraper.api.client.XTwitterScraperClient;
import com.x_twitter_scraper.api.client.okhttp.XTwitterScraperOkHttpClient;
XTwitterScraperClient client = XTwitterScraperOkHttpClient.builder()
.apiKey("My API Key")
.bearerToken("My Bearer Token")
.build();Or using a combination of the two approaches:
import com.x_twitter_scraper.api.client.XTwitterScraperClient;
import com.x_twitter_scraper.api.client.okhttp.XTwitterScraperOkHttpClient;
XTwitterScraperClient client = XTwitterScraperOkHttpClient.builder()
// Configures using the `xtwitterscraper.apiKey`, `xtwitterscraper.bearerToken` and `xtwitterscraper.baseUrl` system properties
// Or configures using the `X_TWITTER_SCRAPER_API_KEY`, `X_TWITTER_SCRAPER_BEARER_TOKEN` and `X_TWITTER_SCRAPER_BASE_URL` environment variables
.fromEnv()
.apiKey("My API Key")
.build();See this table for the available options:
| Setter | System property | Environment variable | Required | Default value |
|---|---|---|---|---|
apiKey |
xtwitterscraper.apiKey |
X_TWITTER_SCRAPER_API_KEY |
false | - |
bearerToken |
xtwitterscraper.bearerToken |
X_TWITTER_SCRAPER_BEARER_TOKEN |
false | - |
baseUrl |
xtwitterscraper.baseUrl |
X_TWITTER_SCRAPER_BASE_URL |
true | "https://xquik.com/api/v1" |
System properties take precedence over environment variables.
Tip
Don't create more than one client in the same application. Each client has a connection pool and thread pools, which are more efficient to share between requests.
To temporarily use a modified client configuration, while reusing the same connection and thread pools, call withOptions() on any client or service:
import com.x_twitter_scraper.api.client.XTwitterScraperClient;
XTwitterScraperClient clientWithOptions = client.withOptions(optionsBuilder -> {
optionsBuilder.baseUrl("https://example.com");
optionsBuilder.maxRetries(42);
});The withOptions() method does not affect the original client or service.
To send a request to the X Twitter Scraper API, build an instance of some Params class and pass it to the corresponding client method. When the response is received, it will be deserialized into an instance of a Java class.
For example, client.x().tweets().search(...) should be called with an instance of TweetSearchParams, and it will return an instance of TweetSearchResponse.
Each class in the SDK has an associated builder or factory method for constructing it.
Each class is immutable once constructed. If the class has an associated builder, then it has a toBuilder() method, which can be used to convert it back to a builder for making a modified copy.
Because each class is immutable, builder modification will never affect already built class instances.
The default client is synchronous. To switch to asynchronous execution, call the async() method:
import com.x_twitter_scraper.api.client.XTwitterScraperClient;
import com.x_twitter_scraper.api.client.okhttp.XTwitterScraperOkHttpClient;
import com.x_twitter_scraper.api.models.x.tweets.TweetSearchParams;
import com.x_twitter_scraper.api.models.x.tweets.TweetSearchResponse;
import java.util.concurrent.CompletableFuture;
// Configures using the `xtwitterscraper.apiKey`, `xtwitterscraper.bearerToken` and `xtwitterscraper.baseUrl` system properties
// Or configures using the `X_TWITTER_SCRAPER_API_KEY`, `X_TWITTER_SCRAPER_BEARER_TOKEN` and `X_TWITTER_SCRAPER_BASE_URL` environment variables
XTwitterScraperClient client = XTwitterScraperOkHttpClient.fromEnv();
TweetSearchParams params = TweetSearchParams.builder()
.q("from:elonmusk")
.limit(10L)
.build();
CompletableFuture<TweetSearchResponse> response = client.async().x().tweets().search(params);Or create an asynchronous client from the beginning:
import com.x_twitter_scraper.api.client.XTwitterScraperClientAsync;
import com.x_twitter_scraper.api.client.okhttp.XTwitterScraperOkHttpClientAsync;
import com.x_twitter_scraper.api.models.x.tweets.TweetSearchParams;
import com.x_twitter_scraper.api.models.x.tweets.TweetSearchResponse;
import java.util.concurrent.CompletableFuture;
// Configures using the `xtwitterscraper.apiKey`, `xtwitterscraper.bearerToken` and `xtwitterscraper.baseUrl` system properties
// Or configures using the `X_TWITTER_SCRAPER_API_KEY`, `X_TWITTER_SCRAPER_BEARER_TOKEN` and `X_TWITTER_SCRAPER_BASE_URL` environment variables
XTwitterScraperClientAsync client = XTwitterScraperOkHttpClientAsync.fromEnv();
TweetSearchParams params = TweetSearchParams.builder()
.q("from:elonmusk")
.limit(10L)
.build();
CompletableFuture<TweetSearchResponse> response = client.x().tweets().search(params);The asynchronous client supports the same options as the synchronous one, except most methods return CompletableFutures.
The SDK defines methods that accept files.
To upload a file, pass a Path:
import com.x_twitter_scraper.api.models.x.media.MediaUploadParams;
import com.x_twitter_scraper.api.models.x.media.MediaUploadResponse;
import java.nio.file.Paths;
MediaUploadParams params = MediaUploadParams.builder()
.account("account")
.file(Paths.get("/path/to/file"))
.build();
MediaUploadResponse response = client.x().media().upload(params);Or an arbitrary InputStream:
import com.x_twitter_scraper.api.models.x.media.MediaUploadParams;
import com.x_twitter_scraper.api.models.x.media.MediaUploadResponse;
import java.net.URL;
MediaUploadParams params = MediaUploadParams.builder()
.account("account")
.file(new URL("https://example.com//path/to/file").openStream())
.build();
MediaUploadResponse response = client.x().media().upload(params);Or a byte[] array:
import com.x_twitter_scraper.api.models.x.media.MediaUploadParams;
import com.x_twitter_scraper.api.models.x.media.MediaUploadResponse;
MediaUploadParams params = MediaUploadParams.builder()
.account("account")
.file("content".getBytes())
.build();
MediaUploadResponse response = client.x().media().upload(params);Note that when passing a non-Path its filename is unknown so it will not be included in the request. To manually set a filename, pass a MultipartField:
import com.x_twitter_scraper.api.core.MultipartField;
import com.x_twitter_scraper.api.models.x.media.MediaUploadParams;
import com.x_twitter_scraper.api.models.x.media.MediaUploadResponse;
import java.io.InputStream;
import java.net.URL;
MediaUploadParams params = MediaUploadParams.builder()
.account("account")
.file(MultipartField.<InputStream>builder()
.value(new URL("https://example.com//path/to/file").openStream())
.filename("/path/to/file")
.build())
.build();
MediaUploadResponse response = client.x().media().upload(params);The SDK defines methods that return binary responses, which are used for API responses that shouldn't necessarily be parsed, like non-JSON data.
These methods return HttpResponse:
import com.x_twitter_scraper.api.core.http.HttpResponse;
import com.x_twitter_scraper.api.models.extractions.ExtractionExportResultsParams;
HttpResponse response = client.extractions().exportResults("id");To save the response content to a file, use the Files.copy(...) method:
import com.x_twitter_scraper.api.core.http.HttpResponse;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.nio.file.StandardCopyOption;
try (HttpResponse response = client.extractions().exportResults(params)) {
Files.copy(
response.body(),
Paths.get(path),
StandardCopyOption.REPLACE_EXISTING
);
} catch (Exception e) {
System.out.println("Something went wrong!");
throw new RuntimeException(e);
}Or transfer the response content to any OutputStream:
import com.x_twitter_scraper.api.core.http.HttpResponse;
import java.nio.file.Files;
import java.nio.file.Paths;
try (HttpResponse response = client.extractions().exportResults(params)) {
response.body().transferTo(Files.newOutputStream(Paths.get(path)));
} catch (Exception e) {
System.out.println("Something went wrong!");
throw new RuntimeException(e);
}The SDK defines methods that deserialize responses into instances of Java classes. However, these methods don't provide access to the response headers, status code, or the raw response body.
To access this data, prefix any HTTP method call on a client or service with withRawResponse():
import com.x_twitter_scraper.api.core.http.Headers;
import com.x_twitter_scraper.api.core.http.HttpResponseFor;
import com.x_twitter_scraper.api.models.x.tweets.TweetSearchParams;
import com.x_twitter_scraper.api.models.x.tweets.TweetSearchResponse;
TweetSearchParams params = TweetSearchParams.builder()
.q("from:elonmusk")
.limit(10L)
.build();
HttpResponseFor<TweetSearchResponse> response = client.x().tweets().withRawResponse().search(params);
int statusCode = response.statusCode();
Headers headers = response.headers();You can still deserialize the response into an instance of a Java class if needed:
import com.x_twitter_scraper.api.models.x.tweets.TweetSearchResponse;
TweetSearchResponse parsedResponse = response.parse();The SDK throws custom unchecked exception types:
-
XTwitterScraperServiceException: Base class for HTTP errors. See this table for which exception subclass is thrown for each HTTP status code:Status Exception 400 BadRequestException401 UnauthorizedException403 PermissionDeniedException404 NotFoundException422 UnprocessableEntityException429 RateLimitException5xx InternalServerExceptionothers UnexpectedStatusCodeException -
XTwitterScraperIoException: I/O networking errors. -
XTwitterScraperRetryableException: Generic error indicating a failure that could be retried by the client. -
XTwitterScraperInvalidDataException: Failure to interpret successfully parsed data. For example, when accessing a property that's supposed to be required, but the API unexpectedly omitted it from the response. -
XTwitterScraperException: Base class for all exceptions. Most errors will result in one of the previously mentioned ones, but completely generic errors may be thrown using the base class.
The SDK uses the standard OkHttp logging interceptor.
Enable logging by setting the X_TWITTER_SCRAPER_LOG environment variable to info:
export X_TWITTER_SCRAPER_LOG=infoOr to debug for more verbose logging:
export X_TWITTER_SCRAPER_LOG=debugAlthough the SDK uses reflection, it is still usable with ProGuard and R8 because x-twitter-scraper-java-core is published with a configuration file containing keep rules.
ProGuard and R8 should automatically detect and use the published rules, but you can also manually copy the keep rules if necessary.
The SDK depends on Jackson for JSON serialization/deserialization. It is compatible with version 2.13.4 or higher, but depends on version 2.18.2 by default.
The SDK throws an exception if it detects an incompatible Jackson version at runtime (e.g. if the default version was overridden in your Maven or Gradle config).
If the SDK threw an exception, but you're certain the version is compatible, then disable the version check using the checkJacksonVersionCompatibility on XTwitterScraperOkHttpClient or XTwitterScraperOkHttpClientAsync.
Caution
We make no guarantee that the SDK works correctly when the Jackson version check is disabled.
Also note that there are bugs in older Jackson versions that can affect the SDK. We don't work around all Jackson bugs (example) and expect users to upgrade Jackson for those instead.
The SDK automatically retries 2 times by default, with a short exponential backoff between requests.
Only the following error types are retried:
- Connection errors (for example, due to a network connectivity problem)
- 408 Request Timeout
- 409 Conflict
- 429 Rate Limit
- 5xx Internal
The API may also explicitly instruct the SDK to retry or not retry a request.
To set a custom number of retries, configure the client using the maxRetries method:
import com.x_twitter_scraper.api.client.XTwitterScraperClient;
import com.x_twitter_scraper.api.client.okhttp.XTwitterScraperOkHttpClient;
XTwitterScraperClient client = XTwitterScraperOkHttpClient.builder()
.fromEnv()
.maxRetries(4)
.build();Requests time out after 1 minute by default.
To set a custom timeout, configure the method call using the timeout method:
import com.x_twitter_scraper.api.models.x.tweets.TweetSearchResponse;
TweetSearchResponse response = client.x().tweets().search(
params, RequestOptions.builder().timeout(Duration.ofSeconds(30)).build()
);Or configure the default for all method calls at the client level:
import com.x_twitter_scraper.api.client.XTwitterScraperClient;
import com.x_twitter_scraper.api.client.okhttp.XTwitterScraperOkHttpClient;
import java.time.Duration;
XTwitterScraperClient client = XTwitterScraperOkHttpClient.builder()
.fromEnv()
.timeout(Duration.ofSeconds(30))
.build();To route requests through a proxy, configure the client using the proxy method:
import com.x_twitter_scraper.api.client.XTwitterScraperClient;
import com.x_twitter_scraper.api.client.okhttp.XTwitterScraperOkHttpClient;
import java.net.InetSocketAddress;
import java.net.Proxy;
XTwitterScraperClient client = XTwitterScraperOkHttpClient.builder()
.fromEnv()
.proxy(new Proxy(
Proxy.Type.HTTP, new InetSocketAddress(
"https://example.com", 8080
)
))
.build();To customize the underlying OkHttp connection pool, configure the client using the maxIdleConnections and keepAliveDuration methods:
import com.x_twitter_scraper.api.client.XTwitterScraperClient;
import com.x_twitter_scraper.api.client.okhttp.XTwitterScraperOkHttpClient;
import java.time.Duration;
XTwitterScraperClient client = XTwitterScraperOkHttpClient.builder()
.fromEnv()
// If `maxIdleConnections` is set, then `keepAliveDuration` must be set, and vice versa.
.maxIdleConnections(10)
.keepAliveDuration(Duration.ofMinutes(2))
.build();If both options are unset, OkHttp's default connection pool settings are used.
Note
Most applications should not call these methods, and instead use the system defaults. The defaults include special optimizations that can be lost if the implementations are modified.
To configure how HTTPS connections are secured, configure the client using the sslSocketFactory, trustManager, and hostnameVerifier methods:
import com.x_twitter_scraper.api.client.XTwitterScraperClient;
import com.x_twitter_scraper.api.client.okhttp.XTwitterScraperOkHttpClient;
XTwitterScraperClient client = XTwitterScraperOkHttpClient.builder()
.fromEnv()
// If `sslSocketFactory` is set, then `trustManager` must be set, and vice versa.
.sslSocketFactory(yourSSLSocketFactory)
.trustManager(yourTrustManager)
.hostnameVerifier(yourHostnameVerifier)
.build();The SDK consists of three artifacts:
x-twitter-scraper-java-core- Contains core SDK logic
- Does not depend on OkHttp
- Exposes
XTwitterScraperClient,XTwitterScraperClientAsync,XTwitterScraperClientImpl, andXTwitterScraperClientAsyncImpl, all of which can work with any HTTP client
x-twitter-scraper-java-client-okhttp- Depends on OkHttp
- Exposes
XTwitterScraperOkHttpClientandXTwitterScraperOkHttpClientAsync, which provide a way to constructXTwitterScraperClientImplandXTwitterScraperClientAsyncImpl, respectively, using OkHttp
x-twitter-scraper-java- Depends on and exposes the APIs of both
x-twitter-scraper-java-coreandx-twitter-scraper-java-client-okhttp - Does not have its own logic
- Depends on and exposes the APIs of both
This structure allows replacing the SDK's default HTTP client without pulling in unnecessary dependencies.
Customized OkHttpClient
Tip
Try the available network options before replacing the default client.
To use a customized OkHttpClient:
- Replace your
x-twitter-scraper-javadependency withx-twitter-scraper-java-core - Copy
x-twitter-scraper-java-client-okhttp'sOkHttpClientclass into your code and customize it - Construct
XTwitterScraperClientImplorXTwitterScraperClientAsyncImpl, similarly toXTwitterScraperOkHttpClientorXTwitterScraperOkHttpClientAsync, using your customized client
To use a completely custom HTTP client:
- Replace your
x-twitter-scraper-javadependency withx-twitter-scraper-java-core - Write a class that implements the
HttpClientinterface - Construct
XTwitterScraperClientImplorXTwitterScraperClientAsyncImpl, similarly toXTwitterScraperOkHttpClientorXTwitterScraperOkHttpClientAsync, using your new client class
The SDK is typed for convenient usage of the documented API. However, it also supports working with undocumented or not yet supported parts of the API.
To set undocumented parameters, call the putAdditionalHeader, putAdditionalQueryParam, or putAdditionalBodyProperty methods on any Params class:
import com.x_twitter_scraper.api.core.JsonValue;
import com.x_twitter_scraper.api.models.x.tweets.TweetSearchParams;
TweetSearchParams params = TweetSearchParams.builder()
.putAdditionalHeader("Secret-Header", "42")
.putAdditionalQueryParam("secret_query_param", "42")
.putAdditionalBodyProperty("secretProperty", JsonValue.from("42"))
.build();These can be accessed on the built object later using the _additionalHeaders(), _additionalQueryParams(), and _additionalBodyProperties() methods.
To set undocumented parameters on nested headers, query params, or body classes, call the putAdditionalProperty method on the nested class:
import com.x_twitter_scraper.api.core.JsonValue;
import com.x_twitter_scraper.api.models.integrations.IntegrationCreateParams;
IntegrationCreateParams params = IntegrationCreateParams.builder()
.config(IntegrationCreateParams.Config.builder()
.putAdditionalProperty("secretProperty", JsonValue.from("42"))
.build())
.build();These properties can be accessed on the nested built object later using the _additionalProperties() method.
To set a documented parameter or property to an undocumented or not yet supported value, pass a JsonValue object to its setter:
import com.x_twitter_scraper.api.models.x.tweets.TweetSearchParams;
TweetSearchParams params = TweetSearchParams.builder()
.q("from:elonmusk")
.limit(10L)
.build();The most straightforward way to create a JsonValue is using its from(...) method:
import com.x_twitter_scraper.api.core.JsonValue;
import java.util.List;
import java.util.Map;
// Create primitive JSON values
JsonValue nullValue = JsonValue.from(null);
JsonValue booleanValue = JsonValue.from(true);
JsonValue numberValue = JsonValue.from(42);
JsonValue stringValue = JsonValue.from("Hello World!");
// Create a JSON array value equivalent to `["Hello", "World"]`
JsonValue arrayValue = JsonValue.from(List.of(
"Hello", "World"
));
// Create a JSON object value equivalent to `{ "a": 1, "b": 2 }`
JsonValue objectValue = JsonValue.from(Map.of(
"a", 1,
"b", 2
));
// Create an arbitrarily nested JSON equivalent to:
// {
// "a": [1, 2],
// "b": [3, 4]
// }
JsonValue complexValue = JsonValue.from(Map.of(
"a", List.of(
1, 2
),
"b", List.of(
3, 4
)
));Normally a Builder class's build method will throw IllegalStateException if any required parameter or property is unset.
To forcibly omit a required parameter or property, pass JsonMissing:
import com.x_twitter_scraper.api.core.JsonMissing;
import com.x_twitter_scraper.api.models.x.tweets.TweetSearchParams;
TweetSearchParams params = TweetSearchParams.builder()
.q(JsonMissing.of())
.build();To access undocumented response properties, call the _additionalProperties() method:
import com.x_twitter_scraper.api.core.JsonValue;
import java.util.Map;
Map<String, JsonValue> additionalProperties = client.x().tweets().search(params)._additionalProperties();
JsonValue secretPropertyValue = additionalProperties.get("secretProperty");
String result = secretPropertyValue.accept(new JsonValue.Visitor<>() {
@Override
public String visitNull() {
return "It's null!";
}
@Override
public String visitBoolean(boolean value) {
return "It's a boolean!";
}
@Override
public String visitNumber(Number value) {
return "It's a number!";
}
// Other methods include `visitMissing`, `visitString`, `visitArray`, and `visitObject`
// The default implementation of each unimplemented method delegates to `visitDefault`, which throws by default, but can also be overridden
});To access a property's raw JSON value, which may be undocumented, call its _ prefixed method:
import com.x_twitter_scraper.api.core.JsonField;
import java.util.Optional;
JsonField<Object> field = client.x().tweets().search(params)._field();
if (field.isMissing()) {
// The property is absent from the JSON response
} else if (field.isNull()) {
// The property was set to literal null
} else {
// Check if value was provided as a string
// Other methods include `asNumber()`, `asBoolean()`, etc.
Optional<String> jsonString = field.asString();
// Try to deserialize into a custom type
MyClass myObject = field.asUnknown().orElseThrow().convert(MyClass.class);
}In rare cases, the API may return a response that doesn't match the expected type. For example, the SDK may expect a property to contain a String, but the API could return something else.
By default, the SDK will not throw an exception in this case. It will throw XTwitterScraperInvalidDataException only if you directly access the property.
If you would prefer to check that the response is completely well-typed upfront, then either call validate():
import com.x_twitter_scraper.api.models.x.tweets.TweetSearchResponse;
TweetSearchResponse response = client.x().tweets().search(params).validate();Or configure the method call to validate the response using the responseValidation method:
import com.x_twitter_scraper.api.models.x.tweets.TweetSearchResponse;
TweetSearchResponse response = client.x().tweets().search(
params, RequestOptions.builder().responseValidation(true).build()
);Or configure the default for all method calls at the client level:
import com.x_twitter_scraper.api.client.XTwitterScraperClient;
import com.x_twitter_scraper.api.client.okhttp.XTwitterScraperOkHttpClient;
XTwitterScraperClient client = XTwitterScraperOkHttpClient.builder()
.fromEnv()
.responseValidation(true)
.build();Java enum classes are not trivially forwards compatible. Using them in the SDK could cause runtime exceptions if the API is updated to respond with a new enum value.
Using JsonField<T> enables a few features:
- Allowing usage of undocumented API functionality
- Lazily validating the API response against the expected shape
- Representing absent vs explicitly null values
Why don't you use data classes?
It is not backwards compatible to add new fields to a data class and we don't want to introduce a breaking change every time we add a field to a class.
Checked exceptions are widely considered a mistake in the Java programming language. In fact, they were omitted from Kotlin for this reason.
Checked exceptions:
- Are verbose to handle
- Encourage error handling at the wrong level of abstraction, where nothing can be done about the error
- Are tedious to propagate due to the function coloring problem
- Don't play well with lambdas (also due to the function coloring problem)
This package generally follows SemVer conventions, though certain backwards-incompatible changes may be released as minor versions:
- Changes to library internals which are technically public but not intended or documented for external use. (Please open a GitHub issue to let us know if you are relying on such internals.)
- Changes that we do not expect to impact the vast majority of users in practice.
We take backwards-compatibility seriously and work hard to ensure you can rely on a smooth upgrade experience.
We are keen for your feedback; please open an issue with questions, bugs, or suggestions.