Dell ECS clusters allow you to migrate your data to any S3 compatible store. Dell ECS calls this feature “Data Movement”, also called copy-to-cloud. It's a feature introduced in ECS 3.8.0.1 that allows you to copy objects from Dell ECS to MinIO which is rather popular with customers and prospects who are modernizing their storage stack to support their AI data infrastructure requirements. The Data Movement is built atop of the ECS Sync open-source tool which provides the capability to copy the data in parallel.
In this overview we’ll show you how to migrate data from Dell ECS to MinIO by specifically focusing on the following:
Before we can start creating the policy to migrate the data, let’s ensure the source and target buckets are configured for Data Movement.
Internally the Data Movement policy scans the source bucket to enumerate all objects for data movement using Metadata (MD) Search.
For Data Movement to actually move data, you need to ensure MD Search is enabled on the Dell ECS source bucket and include LastModified as an indexed field.
Next let's configure MinIO, the target bucket.
In order for the data to be transferred to MinIO, we need to create the following resource in MInIO beforehand:
When creating the bucket, be sure to enable bucket versioning, unless the target bucket in MinIO is dedicated just for the data movement policy. Be sure to make note of the above details after creating them as their values are needed later.
The IAM policy should allow the following APIs:
Follow this guide to learn how to create IAM policies, Access Keys and Buckets in MinIO console.
Object Browser: Buckets which have been created and data uploaded will be shown here.
Access Keys: AWS IAM Style Access Keys
Buckets: List all the buckets that are available.
Policies: IAM Policies
Identity: Create and Connect various IDPs such as OpenID and LDAP.
Monitoring: Monitor all aspects of the cluster and even send it to Prometheus.
Once the Source and Target buckets are configured, let's set up the Data Movement Policy.
A Data Movement Policy is a definition in Dell ECS that can be set either via UI or API that defines which objects in a Dell ECS source bucket should be copied to MinIO target bucket. The Data Movement policy scan jobs are automatically triggered but can be paused or resumed at any time. This is very similar to MinIO’s
We’ll show you two different Data Movement scenarios to give you an idea of how this could work, but the sky's the limit when it comes to how you would want to do the Migration.
In this configuration we’ll add the necessary MinIO bits and bobs for Dell ECS to communicate with. The first step, once MD Search is enabled, is to enable Data Mobility to ON
as shown below.
Once the Data Mobility is set to ON
, we can go ahead and configure the policy.
http://<minio_ip>:<minio_port>
There are other settings, I’ve not gone through all the settings, just the most important ones. Please note that if data is deleted from the Dell ECS source bucket, it won’t be deleted from MinIO target bucket after the migration has been completed.
Now let's take a look at how the migration would look like using an application that uses Dremio.
There are a few steps that take place during this migration
These are just two examples but you can use this methodology to migrate from any application using Dell ECS to MinIO.
During the data migration process, it's important to keep an eye on the overall migration process as it moves data to MinIO. The Dell ECS GUI provides overview dashboards with advanced monitoring that shows total objects copied, total bytes copied, watermark lag, total errors, objects copied, bytes copies, among others.
You can further drill down to show source/target specific information such as the object count and bucket size over a selected period during the migration process.
In the initial diagram at the start of this blog we’ve shown a log bucket we have on the ECS side, this is where all the operations from the Data Movement are logged. This is very helpful to debug any issues during the migration process especially when it takes a very long time due to hardware and physical constraints.
Here is an example of how the logs look like:
2024-08-31T11:40:51Z DM.COPY demo sourcebucket ASIAD708D0875B4F32F8 test.pdf 2022-08-31T09:30:52Z 1,951,137 5895c19c9e742a88d1bec75d40288e0f http://targetendpoint targetbucket AKIA7A04FF4B251997E0 288 SUCCESS
MinIO is a single Go binary that can be launched in many different types of cloud and on-prem environments. It's very lightweight, but is also feature packed with things like replication and encryption, and it provides integrations with various applications.
We’ve
Not only that but out of the box MinIO also includes:
Encryption: MinIO supports both encryption at
Secure Access ACLs and PBAC: Supports IAM S3-style policies with built in IDP, see
Not to mention, good software is nothing without good support. MinIO provides one of the best support out there using our
If you have any questions on how to migrate your data from Dell ECS to MinIO migration be sure to reach out to us on