HOW TO

Full Migration from
Amazon S3 to Cloud Storage

Introduction

This page provides a comprehensive guide for users looking to migrate their data from Amazon S3 to Cloud Storage. It outlines the necessary steps for a complete migration, including adjustments to headers, access control configurations, and supported request methods. By following these guidelines, users can seamlessly transition their data and maintain access control while leveraging the features of Google Cloud Storage. Whether you are looking for a quick start or a detailed migration process, this guide is designed to help you navigate the transition efficiently.

Migrating from Amazon S3 to Cloud Storage

To completely migrate from Amazon S3 to Cloud Storage, you need to complete the following steps:

  1. Change existing x-amz-* headers to the corresponding x-goog-* headers.
  2. Convert AWS Access Control List (ACL) XML to the corresponding Cloud Storage ACL XML (see “Creating and Managing Access Control Lists”).
  3. Set the x-goog-project-id header in your requests.
  4. Complete the relevant configuration to use OAuth 2.0 authentication as described in your OAuth 2.0 authorization. The header will appear as follows:

Authorization: Bearer OAUTH2_TOKEN

OAuth 2.0 relies on SSL for security, making it easier to implement without requiring direct encryption signatures.

Access Control

This section provides examples of access control to assist you in migrating from Amazon S3 to Cloud Storage.

In Cloud Storage, you can evaluate the ACLs of buckets and objects in various ways. You can apply ACLs to specific scopes using the acl query string parameter or apply predefined ACLs using the x-goog-acl header.

Using ACL Query String Parameters

You can use the acl query string parameter in Cloud Storage requests, corresponding to similar functionality in Amazon S3.

Here’s an example showing how to send a PUT request to Amazon S3 using the acl query string parameter. The ACL will be defined in the XML document sent in the request body. The PUT request changes the ACL of the object europe/france/paris.jpg located in the bucket named my-travel-maps, granting jane@gmail.com  FULL_CONTROL permission.

PUT europe/france/paris.jpg?acl HTTP/1.1
Host: my-travel-maps.s3.amazonaws.com
Date: Wed, 06 Nov 2013 19:28:18 GMT
Content-Length: 598
Content-Type: application/xml
Authorization: AWS4-HMAC-SHA256 Credential=AWS-ACCESS-KEY/20131106/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-type;date;host, Signature=4c45f2567996256796aeb9f4cc91c33a1c45fg

<?xml version=’1.0′ encoding=’utf-8′?>
<AccessControlPolicy>
     <Owner>
         <ID>…</ID>
         <DisplayName>ownerEmail@example.com</DisplayName>
     </Owner>
     <AccessControlList>
         <Grant>
             <Grantee xsi:type=”CanonicalUser”>
                 <ID>…</ID>
                 <DisplayName>jane@gmail.com</DisplayName>
             </Grantee>
             <Permission>FULL_CONTROL</Permission>
         </Grant>
     </AccessControlList>
</AccessControlPolicy>

Here’s the same request sent to Cloud Storage:

PUT europe/france/paris.jpg?acl HTTP/1.1
Host: my-travel-maps.storage.googleapis.com
Date: Wed, 06 Nov 2013 19:37:33 GMT
Content-Length: 268
Content-Type: application/xml
Authorization: Bearer ya29.AHES6ZRVmB7fkLtd1XTmq6mo0S1wqZZi3-Lh_s-6Uw7p8vtgSwg

<?xml version=’1.0′ encoding=’utf-8′?>
<AccessControlList>
     <Entries>
         <Entry>
             <Permission>FULL_CONTROL</Permission>
             <Scope type=”UserByEmail”>
                 <EmailAddress>jane@gmail.com</EmailAddress>
             </Scope>
         </Entry>
     </Entries>
</AccessControlList>

Note that Cloud Storage does not require the <Owner/> element in the ACL XML file.

You can also combine the acl query string parameter with the GET method to retrieve bucket and object ACLs, which will be described in an XML document attached to the response body. You must have FULL_CONTROL permission to apply or retrieve ACLs on an object or bucket.

Applying ACLs via Extended Request Headers

In Cloud Storage requests, you can use the x-goog-acl header to apply predefined ACLs, which is identical to using the x-amz-acl header in Amazon S3. The predefined ACLs in Cloud Storage are similar to Amazon S3’s default ACLs, including private, public-read, public-read-write, and other permission settings.

The following example shows a PUT request that applies the public-read ACL to the object europe/france/paris.jpg.

PUT europe/france/paris.jpg HTTP/1.1
Host: my-travel-maps.storage.googleapis.com
Date: Wed, 06 Nov 2013 20:49:57 GMT
Content-Length: 888814
Content-Type: image/jpg
x-goog-acl: public-read
Authorization: Bearer ya29.AHES6ZRVmB7fkLtd1XTmq6mo0S1wqZZi3-Lh_s-6Uw7p8vtgSwg
<888814 bytes in entity body>

You can also apply predefined ACLs to existing buckets or objects using the x-goog-acl header. To do this, add the acl query string parameter to your request but do not include the XML document.

If you want to change from one predefined ACL to another or update a custom ACL to a predefined ACL, it is recommended to apply the predefined ACL to the existing object or bucket.

Request Methods for Migrating from Amazon S3 to Cloud Storage

Cloud Storage supports standard HTTP request methods that are equivalent to those supported by Amazon S3. Therefore, most tools and libraries currently used in Amazon S3 will also work with Cloud Storage.

Cloud Storage supports the following request methods:

  • GET service requests.
  • PUT, GET, DELETE bucket requests.
  • GET, POST, PUT, HEAD, and DELETE object requests.

Keep in mind that when you send requests to Cloud Storage, you may need to adjust the request body to use the appropriate Cloud Storage syntax.

Headers for Migrating from Amazon S3 to Cloud Storage

Cloud Storage supports multiple standard HTTP headers and several custom (extended) HTTP headers. If you are transitioning from Amazon S3 to Cloud Storage, you can convert custom Amazon S3 headers to equivalent Cloud Storage custom headers.

For most Amazon S3 headers, simply replace x-amz with x-goog.

 
Amazon S3 Header -> Cloud Storage Header

x-amz-storage-class -> x-goog-storage-class

x-amz-acl -> x-goog-acl

x–amz-date -> x-goog-date

x-amz-meta-* -> x-goog-meta-*

x-amz-copy-source -> x-goog-copy-source

x-amz-metadata-directive -> x-goog-metadata-directive

…

Note: While Amazon S3 does not have a corresponding x-goog-project-id header, Cloud Storage accepts x-amz-project-id for simple migrations.

Next Steps: Starting the Migration

  1. Plan Migration: From Amazon S3.
  2. Use Storage Transfer Service: To transfer data from external sources to Cloud Storage (such as Amazon S3 and Microsoft Azure Blob Storage).
  3. Establish Event-Driven Transfer Jobs: To keep Cloud Storage buckets in sync with Amazon S3 using Amazon S3 event notifications.

By following these steps, you can smoothly migrate your data from Amazon S3 to Google Cloud Storage, ensuring that your data is securely and reliably stored in the cloud.