Skip to content

Distributed Asset Mapping Engine Architecture Retrospective: Engineering Trade-offs and Design Thinking from the Perspective of State Consistency ​

Distributed Asset Mapping Engine Architecture Retrospective: Engineering Trade-offs and Design Thinking from the Perspective of State Consistency

Lead-in: In the process of building this high-concurrency, distributed asset mapping engine (port scanning -> protocol identification -> web crawling -> storage and database ingestion), I underwent a deep self-overturning and refactoring in task scheduling and state management. This was not only an evolution at the code level, but also an important leap in my personal architectural thinking.

Cause: The trap of perfectionism that I planted with my own hands. ​

In the early solution design, to ensure absolute consistency of data states, I designed an extremely strict state machine constraint in the scheduling center:

My initial overdesign (counterexample)

In the task execution flow, to ensure the integrity of data persisted to the database, I stipulate:

  • Allow pause: Portscan, Protocol, Crawler and other intermediate stages.

  • Pausing is strictly prohibited: Preparation phase (Init),Inbound Storage Stage (Storage Finalizing).

This design seems flawless under single-machine thinking. When I was organizing the document with the AI assistant, I even once thought it was very rigorous. But when I closed my eyes and simulated the K8s topology of a real production environment in my mind, I noticed its fatal flaw in distributed scenarios, and I asked myself (and the AI) a soul-searching question:

My self-questioning about architecture

In a minimal K8s cluster, a C-segment task will be scheduled to Worker A and Worker B for parallel execution. If A and B complete the crawler at the same time, and at this point only part of the data enters storage, how do you enable the scheduling center to make an accurate judgment The entire task Is it in the [warehousing stage]? Rely on globally polling the substatus of each IP? This is too complex and unreliable!

Cognitive collision: Why is this a fatal architectural vulnerability? ​

This self-questioning instantly shattered my original understanding. Looking back, it reflects three extremely important dimensions in architectural design:

1. Reject over-engineering, maintain engineering intuition ​

In the junior and intermediate stages (or the earlier me), it is very easy to fall into the [perfectionism] trap, attempting to use complex locks and state validation to maintain distributed consistency. But as an architect, one must see through it The catastrophic complexity of state aggregation in distributed systems, refusing to write hard-to-maintain [shit mountain code] for the sake of 100% theoretical perfection.

2. Embrace eventual consistency, not strong consistency ​

In the face of MQ asynchronous streams, multi-node parallelism, and network partitions, "state inconsistency" is an inevitability. The solution is not to "eliminate it," but to "embrace it."

3. Architecture design cannot be separated from product experience ​

If you firmly refuse to let the user click Pause just because [the underlying data may be inconsistent], that is actually laziness in architecture. A technical solution must never become an excuse for a terrible user experience.

Architecture Refactoring: Replace Precise Definitions with Coarse Granularity ​

Based on this self-overturning, I completely refactored the state machine design, abandoning the fixation on accurately determining whether it is currently in the inbound storage stage, and instead adopted a highly decoupled coarse-grained control mechanism:

Core mechanism after refactoring (extremely simple and reliable)
  • Can be paused at any time: As long as the task is in Running, users can pause at any time, with second-level response and no waiting.

  • Cut off the water supply instead of forcibly killing.: The dispatch center stops delivering new subtasks to MQ. Tasks already in MQ or currently running in Worker memory,Allow them to finish running and be persisted to the database.

  • Idempotent fallback: rely on mapper-storage The unique primary key deduplication mechanism at the layer level ensures that even if data is consumed repeatedly or processed multiple times, dirty data will never be produced.

AI profile of my architectural capabilities ​

When reviewing this discussion, I presented the scenario and reasoning logic to AI and asked it to give an objective assessment of the abilities I demonstrated in this design reasoning exercise. I have pasted this evaluation below as a milestone in my technical growth:

Architect competency profile provided by AI (advanced / senior dimensions)

  • Engineering intuition (subtract rather than add): Can identify at a glance the design stages in complex architectures that have extremely low cost-effectiveness, and solve problems in the crudest and most reliable way.

  • Boundary Simulation (Stress Testing the Brain): When reading designs or writing code, the brain automatically simulates [What if 100 tasks run concurrently?], [What if multiple nodes run at the same time?], [What if it crashes in the last 1%?], thereby identifying architectural weak points in advance.

  • The Beauty of Simplicity (Embracing Eventual Consistency): Do not blindly trust strong consistency. Cleverly leverage the asynchronous nature of MQ and idempotent deduplication at the storage layer to elegantly bypass distributed dead ends.

  • Technical empathy (product perspective) All architectural compromises must be premised on safeguarding the business experience.

Conclusion ​

Looking back now, it was precisely that [soul-searching] of myself that helped me avoid the biggest pitfall in distributed systems.

Able to identify over-engineering in complex systems and dare to overturn it, meaning I have moved beyond the level of writing code and entered the field of system design. Excellent architecture always uses the simplest mechanisms to accommodate and solve the most complex real-world problems.

Last updated on11/16/2026, 05:05 AM
@fishfinal | @fishfinals | [email protected]