-
Notifications
You must be signed in to change notification settings - Fork 230
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
C2C Changes into Mover/stage #2934
base: mover/stage
Are you sure you want to change the base?
Conversation
…thin mover to be utilized for dynamic credential fetching(C2C use case)
… certData for SPN login
…entation Mover/otendolkar/provider implementation
@otendolkar please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.
Contributor License AgreementContribution License AgreementThis Contribution License Agreement (“Agreement”) is agreed to by the party signing below (“You”),
|
This PR introduces three key enhancements to AzCopy to better support cloud-to-cloud migrations, particularly for moving data from S3 to Azure. These changes are designed to improve the robustness, security, and usability of AzCopy for long-duration migration tasks within Azure Storage Mover. The proposed changes are as follows:
1. Enable S3 Sync Capabilities:
We are adding support for S3 sync functionality within AzCopy. This enhancement is crucial for cloud-to-cloud consumers who need to efficiently synchronize data between S3 and Azure during migration tasks. Enabling this feature ensures seamless, continuous synchronization, reducing data migration overhead and improving overall migration efficiency.
2. Dynamic Fetching of S3 Credentials:
To ensure uninterrupted job execution, we propose enabling the dynamic fetching of S3 credentials. For long-running migration tasks, this allows credentials to be retrieved on the fly, preventing unnecessary pauses or interruptions.
A new field, "Provider", will be added within the s3credinfo structure. This field will interface with the MinIO library to enable dynamic credential retrieval. This change is backward compatible and will not impact the existing AzCopy functionality; the new field will only be utilized if the consumer of AzCopy specifically implements the provider interface.
3. Improved Service Principal Authentication:
Currently, AzCopy supports environment variable-based authentication for service principal login but requires a certPath to be provided as part of the authentication process. This method has limitations, especially in multi-tenant environments, where storing certificates as files could lead to security vulnerabilities, unauthorized access, or accidental leaks.
We propose the addition of certData as an option for service principal authentication. This enhancement allows for more secure, flexible, and manageable authentication, especially in containerized setups. By using certData instead of certPath, we reduce the need for storing sensitive certificates as files, mitigating potential security risks, and simplifying tenant-specific configurations. This also enhances the overall automation and portability of the system.
Impact:
These changes will directly benefit Azure Storage Mover by improving the tool’s ability to handle S3-to-Azure migrations with greater flexibility and security.