Aws S3 Cli Upload All Files in Directory
Introduction
Amazon Web Service, aka AWS, is a leading cloud infrastructure provider for storing your servers, applications, databases, networking, domain controllers, and active directories in a widespread cloud compages. AWS provides a Southwardimple Storage Service (S3) for storing your objects or information with (119's) of data durability. AWS S3 is compliant with PCI-DSS, HIPAA/HITECH, FedRAMP, EU Information Protection Directive, and FISMA that helps satisfy regulatory requirements.
When you log in to the AWS portal, navigate to the S3 bucket, choose your required saucepan, and download or upload the files. Doing it manually on the portal is quite a time-consuming task. Instead, you tin utilize the AWS Command Line Interface (CLI) that works best for bulk file operations with like shooting fish in a barrel-to-use scripts. Y'all tin schedule the execution of these scripts for an unattended object download/upload.
Configure AWS CLI
Download the AWS CLI and install AWS Command Line Interface V2 on Windows, macOS, or Linux operating systems.
Yous tin follow the installation magician for a quick setup.
Create an IAM user
To admission the AWS S3 bucket using the command line interface, we need to set an IAM user. In the AWS portal, navigate to Identity and Admission Management (IAM) and click Add User .
In the Add User page, enter the username and admission type as Programmatic admission.
Next, we provide permissions to the IAM user using existing policies. For this article, nosotros take chosen [AmazonS3FullAccess] from the AWS managed policies.
Review your IAM user configuration and click Create user .
Once the AWS IAM user is created, information technology gives the Access Key ID and Secret access key to connect using the AWS CLI.
Notation : You lot should copy and relieve these credentials. AWS does not allow you to call back them at a after stage.
Configure AWS Contour On Your Computer
To work with AWS CLI on Amazon web service resources, launch the PowerShell and run the following command.
>aws configure It requires the following user inputs:
- IAM user Access Key ID
- AWS Secret Admission central
- Default AWS region-name
- Default output format
Create S3 Saucepan Using AWS CLI
To store the files or objects, we need an S3 bucket. We can create information technology using both the AWS portal and AWS CLI.
The following CLI command creates a bucket named [mys3bucket-testupload1] in the us-east-1 region. The query returns the bucket name in the output, as shown below.
>aws s3api create-bucket --saucepan mys3bucket-testupload1 --region us-east-one 
Yous can verify the newly-created s3 saucepan using the AWS console. As shown below, the [mys3bucket-testupload1] is uploaded in the US East (N. Virginia).
To list the existing S3 bucket using AWS CLI, run the command – aws s3 ls
Uploading Objects in the S3 Bucket Using AWS CLI
We can upload a single file or multiple files together in the AWS S3 bucket using the AWS CLI command. Suppose we have a unmarried file to upload. The file is stored locally in the C:\S3Files with the proper noun script1.txt.
To upload the single file, apply the following CLI script.
>aws s3 cp C:\S3Files\Script1.txt s3://mys3bucket-testupload1/ It uploads the file and returns the source-destination file paths in the output:
Note: The fourth dimension to upload on the S3 saucepan depends on the file size and the network bandwidth. For the demo purpose, I used a pocket-size file of a few KBs.
You tin can refresh the s3 bucket [mys3bucket-testupload1] and view the file stored in it.
Similarly, nosotros tin use the same CLI script with a slight modification. Information technology uploads all files from the source to the destination S3 bucket. Hither, we employ the parameter –recursive for uploading multiple files together:
>aws s3 cp c:\s3files s3://mys3bucket-testupload1/ --recursive Every bit shown beneath, it uploads all files stored inside the local directory c:\S3Files to the S3 bucket. You go the progress of each upload in the console.
We can see all uploaded files using recursive parameters in the S3 bucket in the post-obit figure:
If you do non desire to go to the AWS portal to verify the uploaded list, run the CLI script, return all files, and upload timestamps.
>aws s3 ls s3://mys3bucket-testupload1 
Suppose we desire to upload but files with a specific extension into the separate folder of AWS S3. You tin can do the object filtering using the CLI script too. For this purpose, the script uses include and exclude keywords.
For example, the query beneath checks files in the source directory (c:\s3bucket), filters files with .sql extension, and uploads them into SQL/ folder of the S3 bucket. Here, we specified the extension using the include keyword:
>aws s3 cp C:\S3Files s3://mys3bucket-testupload1/ --recursive --exclude * --include *.sql In the script output, you can verify that files with the .sql extensions only were uploaded.
Similarly, the below script uploads files with the .csv extension into the S3 bucket.
>aws s3 cp C:\S3Files s3://mys3bucket-testupload1/ --recursive --exclude * --include *.csv 
Upload New or Modified Files from Source Folder to S3 Saucepan
Suppose you apply an S3 bucket to motility your database transaction log backups.
For this purpose, we employ the sync keyword. It recursively copies new, modified files from the source directory to the destination s3 bucket.
>aws s3 sync C:\S3Files s3://mys3bucket-testupload1/ --recursive --exclude * --include *.sql As shown beneath, it uploaded a file that was absent in the s3 saucepan. Similarly, if you modify any existing file in the source folder, the CLI script will pick it and upload information technology to the S3 bucket.
Summary
The AWS CLI script can make your work easier for storing files in the S3 bucket. You lot can apply it to upload or synchronize files betwixt local folders and the S3 bucket. Information technology is a quick way to deploy and piece of work with objects in the AWS deject.
(Visited viii,507 times, 29 visits today)
Tags: AWS, aws cli, aws s3, cloud platform Concluding modified: September 16, 2021
Source: https://codingsight.com/upload-files-to-aws-s3-with-the-aws-cli/
0 Response to "Aws S3 Cli Upload All Files in Directory"
Post a Comment