Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Learn more about Collectives

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

Learn more about Teams

I am trying to set up a file upload REST API via Spring Boot.

I currently have a list/GET method curl http://localhost:8080/api/aws/s3/list which returns a list of objects that currently exist in the bucket.

For upload, I have been trying:

curl -F "data=@test.txt" http://localhost:8080/api/aws/s3/upload -i

Which produces:

HTTP/1.1 100 Continue
HTTP/1.1 200 OK
Date: Sun, 25 Jun 2017 23:28:36 GMT
X-Application-Context: application
Content-Type: application/json;charset=utf-8
Transfer-Encoding: chunked

But when I look at the bucket, it hasn't been updated with the new file.

Would this be a permissions issue on AWS? Only my account has read and write access. The user and group I created have admin privileges. I didn't add a Bucket Policy. This is my CORS configuration:

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
    <AllowedOrigin>*</AllowedOrigin>
    <AllowedMethod>GET</AllowedMethod>
    <AllowedMethod>POST</AllowedMethod>
    <AllowedMethod>DELETE</AllowedMethod>
    <MaxAgeSeconds>3000</MaxAgeSeconds>
    <AllowedHeader>Authorization</AllowedHeader>
</CORSRule>
</CORSConfiguration>

Here is the upload section of my S3 Controller in spring:

@RequestMapping(value = "/upload", method = RequestMethod.POST)
public List<PutObjectResult> upload(@RequestParam("file") MultipartFile[] multipartFiles) {
    return s3Wrapper.upload(multipartFiles);

You should specify your using bucket name and some parameters for S3. And I guess it's PUT, not POST. There are several command line samples in the internet.

file=/path/to/file/to/upload.tar.gz
bucket=your-bucket
resource="/${bucket}/${file}"
contentType="application/x-compressed-tar"
dateValue=`date -R`
stringToSign="PUT\n\n${contentType}\n${dateValue}\n${resource}"
s3Key=xxxxxxxxxxxxxxxxxxxx
s3Secret=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
signature=`echo -en ${stringToSign} | openssl sha1 -hmac ${s3Secret} -binary | base64`
curl -X PUT -T "${file}" \
  -H "Host: ${bucket}.s3.amazonaws.com" \
  -H "Date: ${dateValue}" \
  -H "Content-Type: ${contentType}" \
  -H "Authorization: AWS ${s3Key}:${signature}" \
  https://${bucket}.s3.amazonaws.com/${file}
  • Uploading to S3 in 18 lines of Shell (used to upload builds for http://soltrader.net)
  • Uploading to S3 in Bash
  • Make sure you use /bin/echo instead of just echo. That one trapped me. See unix.stackexchange.com/questions/153660/… – Lukas Bradley Jul 17, 2019 at 16:40 I think this Authorization method is no longer supported, you have to use AWS4-HMAC-SHA256 – sale108 Aug 20, 2020 at 8:58 doesn't work The request signature we calculated does not match the signature you provided. Check your key and signing method. – Arsenius Aug 21, 2020 at 11:22
        #run this code on ec2 linux with s3 write role 
    TOKEN=`curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`
    xamztoken=`curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/security-credentials/ec2files3 | jq -r ".Token"`
    yyyymmdd=`date +%Y%m%d`
    s3Bucket="testfiles3bc"
    bucketLocation="us-east-1"
    s3SecretKey=`curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/security-credentials/ec2files3 | jq -r '.SecretAccessKey'`
    s3AccessKey=`curl -s -H "X-aws-ec2-metadata-token: $TOKEN"  http://169.254.169.254/latest/meta-data/iam/security-credentials/ec2files3 | jq -r ".AccessKeyId"`
    endpoint="testfiles3bc.s3.amazonaws.com"
    fileName="image.gif"
    contentLength=`cat ${fileName} | wc -c`
    contentHash=`openssl sha -sha256 -hex ${fileName} | sed 's/.* //'`
    contentType=`file -b --mime-type $fileName`
    b64=`openssl md5 -binary "$fileName" | openssl base64`
    acl="private"
    date=`date -u +%Y%m%dT%H%M%SZ`
    expdate_s="2022-12-30T12:00:00.000Z"
    region="us-east-1"
    p=$(cat <<POLICY | openssl base64 | tr -d \\n
    { "expiration": "${expdate_s}T12:00:00.000Z",
      "conditions": [
        {"acl": "$acl" },
        {"bucket": "$s3Bucket" },
        ["starts-with", "\$key", ""],
        ["starts-with", "$contentType", "image/"],
        {"x-amz-date": "$date" },
        {"content-md5": "$b64" },
        {"x-amz-credential": "${s3AccessKey}/${yyyymmdd}/${region}/s3/aws4_request" },
        {"x-amz-security-token": "${xamztoken}" },
        {"x-amz-algorithm": "AWS4-HMAC-SHA256" }
    POLICY
    stringToSign=$p
    echo "----------------- canonicalRequest --------------------"
    echo -e ${canonicalRequest}
    echo "----------------- stringToSign --------------------"
    echo -e ${stringToSign}
    echo "-------------------------------------------------------"
    # calculate the signing key
    DateKey=`echo -n "${yyyymmdd}" | openssl sha -sha256 -hex -hmac "AWS4${s3SecretKey}" | sed 's/.* //'`
    DateRegionKey=`echo -n "${bucketLocation}" | openssl sha -sha256 -hex -mac HMAC -macopt hexkey:${DateKey} | sed 's/.* //'`
    DateRegionServiceKey=`echo -n "s3" | openssl sha -sha256 -hex -mac HMAC -macopt hexkey:${DateRegionKey} | sed 's/.* //'`
    SigningKey=`echo -n "aws4_request" | openssl sha -sha256 -hex -mac HMAC -macopt hexkey:${DateRegionServiceKey} | sed 's/.* //'`
    # then, once more a HMAC for the signature
    signature=`echo -en ${p} | openssl sha -sha256 -hex -mac HMAC -macopt hexkey:${SigningKey} | sed 's/.* //'`
    key_and_sig_args="-F X-Amz-Credential=${s3AccessKey}/${yyyymmdd}/${region}/s3/aws4_request -F X-Amz-Algorithm=AWS4-HMAC-SHA256 -F X-Amz-Signature=$signature -F X-Amz-Date=${date}"
    curl -v   \
    -F key=$fileName \
    -F acl=$acl \
    $key_and_sig_args  \
    -F "content-md5= ${b64}" \
    -F "Policy=$p" \
    -F "X-Amz-Security-Token= ${xamztoken}" \
    -F "file=@$fileName" \
    https://${s3Bucket}.s3.amazonaws.com/
            

    Thanks for contributing an answer to Stack Overflow!

    • Please be sure to answer the question. Provide details and share your research!

    But avoid

    • Asking for help, clarification, or responding to other answers.
    • Making statements based on opinion; back them up with references or personal experience.

    To learn more, see our tips on writing great answers.