Why teams trip over S3’s pre-2020 consistency model
Did you assume S3 behaves like a single, instant filesystem? Many teams did, and then they hit a familiar class of bugs: stale reads after an overwrite, missing objects right after delete, and surprises in list results. Before Amazon announced strong consistency for all GET and LIST operations in late 2020, S3 offered read-after-write for new objects in many cases but eventual consistency for overwrite PUTs, DELETEs, and LISTs. That mismatch between intuition and reality created hard-to-debug race conditions in distributed systems, CI pipelines, and user-facing flows. For a broader perspective on optimizing your workflows and experiences, see The Traveler’s Guide to Making Every Journey Feel Lighter and More Rewarding.
What did that look like in practice? Imagine two services A and B both writing to the same object key. A writes a new version and immediately asks B to process it. B does a GET and either sees the old content or an error. Or you deploy a static website and remove an asset; clients still get 200 responses for minutes. Or a batch job lists a prefix to find files to process and skips a file that should have appeared. These are the symptoms people ran into.
How stale reads and delayed deletes cost engineers and businesses
Why should you care if an object read is delayed by a few seconds or a minute? Because distributed systems don’t fail in isolation. A single stale GET can cascade:
- Data corruption: an overwrite that appears to never have happened leads to inconsistent state across services.
- Duplicate work: failing to see a newly added object causes reprocessing or missed work once the list eventually shows it.
- Revenue impact: user-facing assets that should be removed continue to be served, creating compliance or billing problems.
- Operational pain: debugging these issues is expensive because symptoms are intermittent and timing-dependent.
Have you spent hours chasing phantom versions or “it worked yesterday” bugs? That urgency is real. Fixing the design is cheaper than firefighting the next outage.
3 architectural reasons S3 behaved like this before 2020
Understanding the why helps you design around it. What made S3 eventually consistent in those cases?
Massive distributed storage with partitioned metadata
S3 stores object metadata across partitions for scale. When you overwrite or delete an object, the change must propagate across partitions and index nodes. A read hitting a partition that hasn’t applied the change yet will return the old state. That propagation takes time and is non-deterministic.
Index updates and list operations are eventually consistent
Listing a prefix consults an index that lags under high write rates. S3 prioritized availability and performance for global scale, so list results were allowed to be stale until internal replication caught up.
At-least-once delivery and independent subsystems
Event notifications, replication, and client-facing GET endpoints are implemented across multiple systems. Each subsystem has its own failure modes and delivery guarantees. That creates windows where different clients observe different versions of reality.
How to design reliably against eventual consistency in S3 (pre-2020 era)
What should you do if you must run on the older model, or you need designs that are robust regardless of provider guarantees? The short answer: stop depending on instant global visibility. Instead, pick patterns that provide determinism or move coordination out of S3.
Prefer immutability: write-once keys
Why is immutability the simplest winning move? If every new version gets a new key – often by embedding a version id, timestamp, or UUID – you avoid overwrite and delete races entirely. Consumers look for the specific key you produced. No overwrite, no confusion.
Use a strongly consistent index for metadata
S3 is excellent as durable object storage, but it’s not a coordination service in the pre-2020 model. Put authoritative metadata (current key pointers, processing state) into a strongly consistent store such as DynamoDB (with conditional writes), a relational database, or an eventually consistent store with a leader for coordination. Your services read the index to find the canonical key. That turns S3 into a content store while the index governs order and visibility.
Employ write-then-publish patterns
Write the object to a temporary, unique key first. Validate the write with HEAD or by checking the returned ETag. Then publish by writing a small manifest or pointer (the canonical key) into your consistent index. Consumers check the index. This pattern prevents consumers from reading a key that could be overwritten later.
Guard with versioning
Enable S3 versioning for buckets that need recovery from overwrite mistakes. Versioning won’t fix immediate visibility but it gives you a safe way to recover deleted or overwritten objects and audit changes.
Design lists as eventually consistent explicitly
Assume LIST is a hint, not a source of truth. If your processing needs to find “all objects that arrived,” consider storing messages in a queue (SQS) or writing index entries into DynamoDB as objects are produced. Use the queue as the work order. Use S3 lists only for reconciliation and auditing.
6 steps to implement a robust S3 workflow that avoids race conditions
Here is a concrete, implementable plan you can adopt today. These steps assume you want deterministic processing and minimum surprises.
Make object keys unique and immutable
Append a UUID, monotonic sequence, or timestamp to keys. For example: /uploads/customer123/invoice-20210315T120501Z-.pdf. Stop overwriting the same key.
Write object, then write index entry to a strongly consistent store
On successful upload, write a single metadata row to DynamoDB with the final canonical pointer, ETag, and a status flag (READY). Use conditional writes to avoid races when multiple writers compete for the same logical object.
Use SQS or DynamoDB Streams to trigger downstream work
Emit a reliable message after the index entry is committed. Consumers read the index entry for the canonical key and process the object. The message queue becomes the ordering and delivery mechanism with explicit retries.
Validate reads with HEAD or ETag checks
Before processing an object, perform a HEAD to confirm the object exists and to read the ETag you recorded in the index. If the ETag doesn’t match, back off and retry. This guards against edge cases where the object write was not fully visible yet.
Implement exponential backoff and read-retry windows
For critical paths, implement exponential backoff with jitter when a read does not return the expected version. Combine this with a short read-retry window (several seconds) rather than immediate failure. This avoids transient errors becoming permanent incidents.
Reconcile with periodic audits
Run a scheduled reconciliation process that compares your index entries with S3 list results or S3 Inventory. Flag and repair inconsistencies by re-creating missing objects or updating index entries. Reconciliation is your safety net.
Example flow: publishing a processed file
Ask yourself: how would a document processing pipeline look? Here’s a short sequence:
- Worker uploads processed file to s3://bucket/tmp/
- Worker HEADs the tmp key, reads ETag
- Worker writes a DynamoDB row: logicalId, s3Key: tmp/, etag, status: READY
- Worker sends SQS message with logicalId
- Consumer receives SQS message, retrieves DynamoDB row, confirms ETag matches HEAD result
- Consumer copies or references the object as needed; if the design requires a stable public path, create a new immutable public key and update the index atomically
What to expect after you change your design: practical outcomes and timeline
If you adopt these patterns, what will change and when will you see benefits?
- Immediate reduction in race-related incidents: switching to immutable keys and an authoritative index removes most classes of overwrite and deletion races. You should see fewer tickets and retries within days.
- Predictable debugging: instead of chasing intermittent S3 visibility issues, you’ll trace problems via the index and message queue. That reduces time-to-resolution for new incidents.
- Some performance trade-offs: adding a DynamoDB write and an SQS message increases latency on the write path by tens to low hundreds of milliseconds. If you optimized for sub-50ms writes, expect to adjust SLAs.
- Operational overhead: you now have more moving parts to monitor: DynamoDB capacity, SQS queue depth, reconciliation job health. Build simple dashboards and alarms. The maintenance cost is low compared to repeated data inconsistencies.
Timeline: basic migration (unique keys + index writes) can be done in a sprint for a small team. Adding reconciliation and robust backoff patterns takes another sprint or two. Full organizational adoption, with audits and monitoring, typically completes in one to three months depending on scale.
Advanced techniques for high-scale systems
For systems with extreme throughput or strict ordering needs, the basic patterns may need augmentation. What options exist when you must process millions of files per hour or preserve strict ordering?
Sequence numbers and partitioned workloads
Can you partition by key to reduce cross-writer contention? Yes. Use a partition key (customer id, tenant id, time bucket) and keep strict ordering within each partition. Store per-partition sequence numbers in DynamoDB or use an append-only log like Kafka for ordering. This keeps coordination local and minimizes write hotspots.
Use of FIFO queues for ordered delivery
SQS FIFO queues preserve order per message group id. If you can map each logical stream to a group id, you get ordered delivery guarantees for downstream processors, which helps when order matters more than absolute low latency.
Idempotent consumers
Design consumers to be idempotent. If a message is delivered twice or a file is processed multiple times, the consumer should detect duplicates using the canonical index or ETag and skip reprocessing. Idempotency is one of the most practical defenses against the real-world messiness of distributed systems.
Event sourcing alternative
Are you building a system S3 bucket access control where events are the source of truth? Consider storing events in an ordered, strongly consistent store (Kafka, Kinesis, or a relational DB) and make S3 a derived materialization. Write files from the event stream instead of treating S3 as the primary log. That flips the model: S3 is cheap long-term storage, not the coordination plane.
Tools and resources
Which tools and docs should you bookmark?
- AWS S3 documentation on consistency (read archival notes and the post-2020 announcement for historical context)
- DynamoDB documentation on conditional writes and transactions
- SQS FIFO documentation and dead-letter queue best practices
- S3 Inventory and S3 Batch Operations for reconciliation at scale
- Open-source libraries for idempotent processing and retry policies
Final questions to ask before you redesign
Before you invest in changes, answer these questions to focus work where it matters:
- Which operations currently depend on immediate visibility from S3?
- Can we make those flows accept eventual visibility, or must we enforce strong ordering?
- Is adding a small latency to the write path acceptable in exchange for consistency?
- What is the cost of an occasional inconsistent read in business terms?
Relying on S3 to do coordination was always a gamble. The more honest approach is to treat S3 as reliable object storage and use a small, strongly consistent store for the control plane. That leads to predictable systems and fewer nights spent chasing timing-dependent bugs. Which pattern will you adopt first: immutable keys, an authoritative index, or event sourcing? Pick one, prototype it, and measure the difference in production errors – you’ll be surprised how fast the noisy incidents disappear.
