Dear Support team, I am trying to fetch millions of records from DB In 10k batch.
But after some successfully fetching some batches, It throws below Exception:
com.mongodb.MongoExecutionTimeoutException: Request timed out. Retries due to rate limiting: True.
Please help me with a Java Solution, on how can I read all records from DB in any way.
As I understand from the question, you are getting timeout exception when fetching records from Cosmos DB.
Could you please check if in your code you have something like db.Show() (listing od Databases) which might create timeout issue.
Please check and let us know if this helps. If not, kindly share the snippet of how you are fetching records from DB in code.
this code block I am using to Fetch Products from DB. Which have StoreId = 3.
int PER_BATCH_COUNT = 1000;
List<Product> dataList = new ArrayList<Product>();
int count = 77235; //no of records in DB
int iterations = BigDecimal.valueOf(count).divide(BigDecimal.valueOf(PER_BATCH_COUNT), RoundingMode.UP).intValue();
int iterationNo = 0;
/*this works as a page no, to calcualte the next start index*/
int rowCount = 1;
while(iterations != 0) {
iterationNo++;
log.info("Getting result for Iteration No: " + iterationNo);
int startIndex = (iterationNo-1)*PER_BATCH_COUNT+1;
Query query = Query.query(Criteria.where(SearchParameters.STORE_ID.getAttributeName()).is("3"));
query.skip(startIndex);
query.limit(PER_BATCH_COUNT);
dataList.addAll(mongoTemplate.find( query, Product.class));
iterations--;
return dataList;
getProductsByCriteriaWithPaginationForExport function is written below.
public Slice<Product> getProductsByCriteriaWithPaginationForExport( CriteriaDefinition criteriaDefinition, Pageable pageable) { Query query = new Query().addCriteria(criteriaDefinition).with(pageable); return new SliceImpl<Product>(mongoTemplate.find(query, Product.class), pageable, true);}
One more code I tried, which gives same error:
"Request timed out. Retries due to rate limiting: True.; nested exception is com.mongodb.MongoExecutionTimeoutException: Request timed out. Retries due to rate limiting: True."
int page = 0;
int partitionKey = Integer.parseInt("100");
int pageSize = 10000;
int count = 118;
List<List<Product>> products = new ArrayList<>();
while (count > 0){
Pageable pageable = PageRequest.of(page++, pageSize);
Slice<Product> productPages = productRepository.getProductsByCriteriaWithPaginationForExport( mongodbCriteriaBuilderService.getCriterionForGetExportProductsBySearchRequest(exportFileRequest), pageable );
count--;
log.info("Processed PageNo: " + page); products.add(productPages.getContent());
Slice<Product> productPages = null;
log.info("Products Size: " + products.size());
return productPages;
Hi
@Deepanshu Goyal
,
For Rate Limiting Timeout errors in Cosmos DB, it is recommended to increase Request Units.
Azure Cosmos DB for MongoDB operations may fail with rate-limiting (16500/429) errors if they exceed a collection's throughput limit (RUs).
You can enable the Server Side Retry (SSR) feature and let the server retry these operations automatically. The requests are retried after a short delay for all collections in your account. This feature is a convenient alternative to handling rate-limiting errors in the client application.
Refer to the detailed documentation for the same here:
Prevent rate-limiting errors for Azure Cosmos DB for MongoDB operations
Hope this helps. If this answers your query, do click
Accept Answer
and
Up-Vote
for the same. And, if you have any further query do let us know.
Dear ShaktiSingh,
Please guide me how do I know my configured RUs?
And does 'x' RU means that I can only fetch x number of records from Mongo DB at a time? If yes, please suggest a java solution to fetch all Mongo Records with current RU without increasing RUs.
Currently I am able to fetch upto 130k around records from Mongo DB out of 1190876 records.
As you can see in screenshot, after 8 pm, I tried to fetch all records, but it gives "springframework.data.mongodb.UncategorizedMongoDbException: Request timed out. Retries due to rate limiting: True.; nested exception is com.mongodb.MongoExecutionTimeoutException: Request timed out. Retries due to rate limiting: True." error.
I try with this code:
Query query = Query.query(Criteria.where(SearchParameters.STORE_ID.getAttributeName()).is("3"));query.skip(startIndex);query.limit(PER_BATCH_COUNT);List<Product> localList = mongoTemplate.find( query, Product.class);
Hi @Deepanshu Goyal ,
As I have mentioned, for the rate limiting error in Mongo DB, it is recommended to increase Request Units.
I would request you to go through the below documentations related to RUs and its consumption:
Request Units in Azure Cosmos DB
How to monitor throughput or request unit usage of an operation in Azure Cosmos DB
How to correctly calculate the Request Units in Azure Cosmos DB
Try to go for Autoscale so that this error could be dealt:
Let us know if this helps in your case.
