Instructions.txt

***File Transfer Module***

This module reads all files from a directory and its subdirectories, uploads images and media files to AWS S3, and documents to Google Cloud Storage and handles high volume of data.

-----Set Up Your Environment----
Clone the Repository:

git clone <repository-url>
cd <repository-name>

-----Create and Activate a Virtual Environment-----

python -m venv virtual_env
virtual_env\Scripts\activate

-----Install Dependencies------

pip install -r requirements.txt

----Configure AWS and Google Cloud Credentials----
AWS Credentials:

Ensure you have your AWS credentials configured. You can use the AWS CLI to configure them:

aws configure

export AWS_ACCESS_KEY_ID='your-access-key-id'
export AWS_SECRET_ACCESS_KEY='your-secret-access-key'
export AWS_DEFAULT_REGION='your-region'
Default output format [None]: json

-----Configure Google Cloud Credentials-----

Ensure you have your Google Cloud credentials JSON file. Set the environment variable to point to this file:
export GOOGLE_APPLICATION_CREDENTIALS='/path/to/your/credentials-file.json'


-----Configure the Project-----
Update Configuration:

Open FileTransfer/config.py and update the bucket names and file types if necessary.
FILE_TYPES = {
    'images': ['jpg', 'png', 'svg', 'webp'],
    'media': ['mp3', 'mp4', 'mpeg4', 'wmv', '3gp', 'webm'],
    'documents': ['doc', 'docx', 'csv', 'pdf']
}

AWS_S3_BUCKET = 'your-s3-bucket-name'
GCS_BUCKET = 'your-gcs-bucket-name'


-----Run the script with the directory containing your files-----

Terminal: 
python -m FileTransfer.file_transfer /path/to/your/files

Example:  python -m FileTransfer.file_transfer C:\Users\Maha\AWS_Test

---------------------------------------------------------------------------------------------------------------------------------------