Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

I am trying to list items of a large bucket while using pagination. I have two separate commands that work but I am having trouble merging them into one such that I can page through responses.

Command 1:

aws s3api list-objects-v2 --bucket <my bucket> --max-items 100
This returns a json with keys "Contents" and "NextToken". Unfortunately I cannot query/filter.
Command 2:
aws s3api list-objects-v2 --bucket <my bucket> --query "<my query>"
This returns a json with a list of my results, which appears to return everything (as described by this question which says the page size is 1000 but the CLI automatically handles paging to return all results). 
How can I form a command that lets me --query while also specifying a --max-items size limit, and be provided with a NextToken? The following command returned an empty list:
aws s3api list-objects-v2 --bucket <my bucket> --max-items 100 --query "<my query>"
                What are you actually wanting to accomplish as an end-result? That is, not just getting a token, but what final result are you wanting to obtain from the bucket? Do you just want a list of all objects keys? If you can describe your end-goal, we can probably help you find the best way to achieve it.
– John Rotenstein
                May 15, 2020 at 22:35
The issue is that NextToken occurs once in the result set, while Contents is a list with multiple items.
This command will return both values:
aws s3api list-objects-v2 --bucket my-bucket --max-items 10 --query [NextToken,Contents[].Key]
The output is:
    "eyJDb250aW51YXRpb25Ub2tlbiI6IG51bGwsICJib3RvX3RydW5jYXRlX2Ftb3VudCI6IDEwfQ==",
        "foo1.docx",
        "foo2.jpg",
        "2019/06/02/foo3.txt",
        "2019/06/02/foo4.js",
        "2019/06/02/foo5.py",
        "2019/06/02/foo6.html",
        "foo7.pdf",
        "CreateThumbnail.zip",
        "Jbookmarks.html",
        "basepart/20191222_1114/foo9.csv"
The first part is the Token, plus a list of object keys.
                This doesn't seem to work when specifying KeyCount in place of NextToken. How do I add the other potential outputs besides Contents to a --query?
– ericOnline
                Apr 27, 2021 at 0:44
                @ericOnline The NextToken value provided by the previous call must be supplied in the subsequent call. KeyCount (whatever that is) is not relevant.
– John Rotenstein
                Apr 27, 2021 at 2:47
                I'm referring to the AWS docs here. There seem to be many other outputs to --query besides Contents and NextToken (including KeyCount, CommonPrefixes, etc.).
– ericOnline
                Apr 27, 2021 at 16:39
                Oh, are you saying that you want to output the value for KeyCount? Yes, the Output does not seem to match the documented outputs. Some of them only appear if relevant (eg NextToken), but I was unable to get KeyCount to appear. Looks like a documentation error.
– John Rotenstein
                Apr 27, 2021 at 22:20
                If you take out the --query section, you will see the full data returned. Many of the mentioned fields are missing. I have reported it via the AWS CLI Feedback page.
– John Rotenstein
                Apr 27, 2021 at 22:31
        
Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.