Night 20 — Migration & Transfer: DMS + S3 staging for database moves

~2 hr. Very low AWS spend — S3 storage for export files only (pennies). No DMS replication instance unless you opt into the optional -DeployDmsEndpoints flag (creates endpoints only; still no replication instance). Builds on Night 18 DynamoDB (PITR required for export), Night 16 Aurora (DMS target in theory), and Night 19 backups (restore elsewhere, then sync delta).

Plain language first

Why migration is a separate exam domain from DR

Night 19 answered “how do we recover after disaster?” Tonight answers “how do we move data from A to B while the business keeps running?”

GSA story: On-prem resort_stats PostgreSQL → Aurora in Night 9 VPC. Wiki counters stay in DynamoDB. Bulk historical wiki rows might export to S3 for analytics (Athena Night 23) or import into a new table after schema change.

CDC — Change Data Capture (define this first)

CDC stands for Change Data Capture. Plain English: after the initial copy, keep reading every new insert, update, and delete from the source database and apply those changes to the target — so the target stays current while production still runs on the old system.

Analogy (GSA wiki): Imagine photocopying every wiki page on Monday (full load). Through the week, editors keep fixing typos and bumping view counts on the original wiki. CDC is the intern who watches the edit log every few seconds and updates the copy with only what changed — not re-photocopying the whole site every time.

Two phases — almost every DMS cutover uses both:

What DMS reads for CDC (exam awareness — source must expose logs):

CDC is not tonight’s lab. DynamoDB export to S3 is a one-time snapshot of table data at a point in time — it does not stream live writes afterward. CDC is the DMS concept you need when the exam says “minimal downtime migration” or “keep target in sync until cutover.”

Cutover checklist (when CDC is in play):

1. Full load finished.
2. CDC running — check replication lag (seconds behind source).
3. Stop writes to source (maintenance window).
4. Wait until lag ≈ 0.
5. Point apps at target and decommission source.

AWS DMS — the database migration workhorse

AWS Database Migration Service (DMS) moves relational and NoSQL data between sources and targets with minimal downtime.

┌──────────────────┐     full load + CDC      ┌──────────────────┐
│  Source DB       │ ───────────────────────► │  Target DB       │
│  (on-prem Oracle │   DMS replication      │  (Aurora Postgres│
│   or RDS MySQL)  │   instance or Serverless │   or DynamoDB)   │
└──────────────────┘                        └──────────────────┘
         │                                              ▲
         │  optional S3 staging (parquet/csv)           │
         └──────────────────────────────────────────────┘

Homogeneous vs heterogeneous:

SCT — AWS Schema Conversion Tool (schema only; DMS moves rows)

SCT = AWS Schema Conversion Tool. Plain English: a desktop app that reads your source database schema (tables, indexes, views, stored procedures) and generates equivalent DDL for the target engine — plus a report flagging what it could not auto-convert.

Analogy (GSA): DMS is the moving truck that hauls rows from the old resort_stats database to Aurora. SCT is the architect who redraws the floor plan when the old building was Oracle and the new one is PostgreSQL — VARCHAR2 → VARCHAR, Oracle sequences → Postgres sequences, stored procedure syntax rewritten or marked “manual fix needed.”

Who does what in a heterogeneous migration:

Typical order: SCT converts schema → apply DDL on Aurora → DMS full load + CDC from Oracle → cutover.

When you do not need SCT: Homogeneous moves — e.g. PostgreSQL → Aurora PostgreSQL (Night 16 stack). Same SQL dialect; DMS alone.

Exam trap: DMS replicates data — it does not automatically rewrite application code. SCT helps schema; your team still updates connection strings and hand-fixes SQL the report flagged.

S3 as migration staging (Tonight’s hands-on)

Not every migration uses DMS end-to-end. AWS native export/import often stages in S3:

Tonight’s lab runs DynamoDB export to S3 on saa-study-gsa-wiki-views (Night 18). That is the same staging bucket a DMS S3 target or Athena would read — tying “migration” to concrete storage.

Night 18 table (PITR on)
  saa-study-gsa-wiki-views
        │
        │ export-table-to-point-in-time
        ▼
S3  saa-study-gsa-migration-298043721974
      night-20/dynamodb/wiki-views/   ← Tonight
        │
        ├──► (exam) DMS S3 target / Glue / Athena
        └──► (exam) import-table → new DynamoDB table

PITR requirement: DynamoDB export uses continuous backup metadata — table must have PITR enabled (Night 18 lab turns this on).

Tie Night 19 backup → Night 20 migration

Classic lift-and-shift database cutover:

1. Night 19: On-demand backup / snapshot before change window.
2. Restore snapshot to new Aurora cluster in target VPC (or new Region).
3. DMS full load + CDC from production source to restored target — catches writes during migration window.
4. Cutover: Stop writes to source, wait for CDC lag ≈ 0, point apps to target.

AWS Backup recovery points and DMS solve different problems — backup = rewind; DMS = forward sync to a live target.

Migration & Transfer decision matrix (exam)

Bandwidth rule of thumb: If transfer time > ~1 week at your link speed, evaluate Snowball. DataSync and DMS assume online paths.

DMS Serverless vs replication instance

Study lab does not start a replication instance by default — cost. Optional flag documents endpoints only for console familiarity.

Exam traps

• DMS ≠ database upgrade service — does not patch engines or resize RDS automatically.
• CDC source requirements — source must expose logs (PostgreSQL WAL, MySQL binlog, Oracle redo). DynamoDB uses streams/other paths — know export/import for DynamoDB.
• DataSync ≠ DMS — DataSync is file/object replication; DMS is database replication.
• Snowball ≠ Snowmobile — Snowmobile is exabyte-scale container truck; Snowball Edge has local compute.
• Transfer Family — managed SFTP/FTP/FTPS into S3/EFS, not generic DB migration.
• S3 export is async — poll ExportDescription.ExportStatus; not instant like GetItem.

Block 1 (~30 min) — Migration scenarios for GSA

Work three scenarios on paper:

Read (15 min):

• AWS DMS — how it works — endpoints, tasks, CDC.
• DynamoDB export to S3 — PITR prerequisite.

Block 2 (~60 min) — Lab: S3 staging bucket + DynamoDB export

Prerequisites

If the table is missing, run Night 18 setup first.

What the lab creates

Same-account export uses your caller IAM permissions; current AWS CLI does not require a separate --export-role-arn parameter.

Run the lab

# PowerShell — from repo root
.\HTML\study-lab\night-20-lab-migration-setup.ps1
.\HTML\study-lab\night-20-lab-migration-setup.ps1 -ExportTable -VerifyExport

# Git Bash / WSL
bash HTML/study-lab/night-20-lab-migration-setup.sh
bash HTML/study-lab/night-20-lab-migration-setup.sh --export-table --verify-export

End-to-end validation

1. Bucket exists:
   aws s3api head-bucket --bucket saa-study-gsa-migration-298043721974

2. PITR enabled:
   aws dynamodb describe-continuous-backups --table-name saa-study-gsa-wiki-views --query 'ContinuousBackupsDescription.PointInTimeRecoveryDescription'

3. Export started (after -ExportTable):
   aws dynamodb list-exports --table-arn <table-arn>

4. Objects in S3 (after COMPLETED):
   aws s3 ls s3://saa-study-gsa-migration-298043721974/night-20/dynamodb/wiki-views/ --recursive

5. Result file: night-20-migration-result.json in study-lab/.

Troubleshoot in this order:

1. Does saa-study-gsa-wiki-views exist with PITR ENABLED? Night 18 setup enables it.
2. Export FAILED with access denied? Ensure your IAM user/role has dynamodb:ExportTableToPointInTime and S3 write on the migration bucket.
3. Export IN_PROGRESS > 15 min on tiny table — check aws dynamodb describe-export --export-arn <arn>.
4. Bucket name globally taken? Script uses account-scoped name; change suffix only if collision.

Optional: DMS endpoint sketch (console, no replication instance)

If you want console familiarity without replication-instance cost:

1. Open AWS DMS → Endpoints → Create endpoint.
2. Source type PostgreSQL, target Amazon Aurora — use Night 16 cluster endpoint (if still deployed).
3. Stop before creating a replication instance — note which SG rules DMS would need (5432 from replication instance SG to saa-study-gsa-aurora-sg).

Document in your notes: full load + CDC task would run after replication instance exists.

Compare migration paths (Nights 16–20)

Reference files

• night-20-migration-result.json — bucket, export ARN
• night-18-dynamodb-result.json — source table ARN

Teardown

.\HTML\study-lab\night-20-lab-migration-teardown.ps1

bash HTML/study-lab/night-20-lab-migration-teardown.sh

Empties and deletes the migration bucket. Does not delete Night 18 DynamoDB table, Night 16 Aurora, or Night 19 backup vault.

Block 3 (~30 min) — Quiz + DMS Serverless reading

File: night-20-quiz.json — 15 timed scenario questions (~24 min at 96 sec each).

Score on the study site: /aws-solutions-architect-study/quiz?night=20.

Reading (10 min, no deploy):

• DMS Serverless — when exam mentions “variable migration workload.”
• Skim AWS DataSync vs Snowball — online vs offline file transfer.

Database + Migration domain map (Week 3)

5 flashcards (write tonight)

1. CDC (Change Data Capture) — One sentence: what it captures, and what log source DMS reads for Postgres.
   (Ongoing inserts/updates/deletes from source transaction log → target; Postgres/Aurora uses WAL.)

2. Full load vs CDC — Which runs first in a DMS cutover, and why you need both.
   (Full load = initial bulk copy; CDC = catch live changes until lag is zero and you flip apps.)

3. Homogeneous vs heterogeneous — Oracle→Aurora example and when SCT is required.
   (Heterogeneous needs SCT for schema; homogeneous same engine family.)

4. DataSync vs DMS vs Snowball — One-line pick for files, database, offline petabytes.
   (DataSync online files; DMS databases; Snowball offline bulk.)

5. DynamoDB export vs PITR restore — When export to S3 vs restore to new table?
   (Export = portable files in S3; PITR restore = new live DynamoDB table same account.)

6. Night 19 backup + Night 20 DMS — How do snapshot and CDC combine in a migration window?
   (Snapshot/backup seeds target; DMS CDC catches delta until cutover.)

Night 21 preview

Hybrid networking — Site-to-Site VPN, Direct Connect, Transit Gateway, and PrivateLink vs VPC peering. Diagram three exam scenarios (on-prem to VPC, multi-VPC hub, SaaS via PrivateLink).