So there should be a way of capturing the changes done to the source data system and propagating them to derived data systems in a reliable and scalable manner.
If we go back to our original example, a change made in the database has to be reflected in the search index, the cache, and ultimately in the warehouse. They have to be synchronised to preserve consistency across the application. Source data and derived data should not be kept in silos. Source data and derived data need to be synchronised If you lose derived data, you can recreate it from the source. This data is called Derived Data - which is often redundant and in denormalised form. Other systems can take source data, apply transformations, and store with own representations to serve different purposes. For example, when a customer creates an order, it is first stored in the orders database. The very first time a user creates data, it is captured into Systems of Records. This version is often called Systems of Records data or source data. That means, when there’s a discrepancy across versions, the source of truth will be accepted as the correct one. When you have multiple versions of the same data set, you need to appoint one as the source of truth, or the authoritative version. That forces them to keep their data in multiple places, in a redundant and denormalised manner. Consequently, applications have to use different data storage technologies such as indexes, caches, and warehouses together in their architecture. Practically speaking, no one database can satisfy all those needs simultaneously. Application maintains its data in multiple places
0 kommentar(er)
