diff --git a/roadmaps/data-engineer/content/amazon-ec2--compute@AHLsBfPfBJOhLlJ-64GcK.md b/roadmaps/data-engineer/content/amazon-ec2--compute@AHLsBfPfBJOhLlJ-64GcK.md index 16f62978d..3f9a80255 100644 --- a/roadmaps/data-engineer/content/amazon-ec2--compute@AHLsBfPfBJOhLlJ-64GcK.md +++ b/roadmaps/data-engineer/content/amazon-ec2--compute@AHLsBfPfBJOhLlJ-64GcK.md @@ -1,5 +1,5 @@ # Amazon EC2 (Compute) - + Amazon EC2 (Elastic Compute Cloud) provides virtual servers in the AWS cloud. Users can choose instance types optimized for compute, memory, or storage, and pay only for what they run. EC2 is used for running data processing jobs, hosting databases, and building custom data infrastructure on AWS. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/amazon-rds-database@GtFk7phYGfXUhxanicYNQ.md b/roadmaps/data-engineer/content/amazon-rds-database@GtFk7phYGfXUhxanicYNQ.md index 17aa6d762..354b767b6 100644 --- a/roadmaps/data-engineer/content/amazon-rds-database@GtFk7phYGfXUhxanicYNQ.md +++ b/roadmaps/data-engineer/content/amazon-rds-database@GtFk7phYGfXUhxanicYNQ.md @@ -1,5 +1,5 @@ # Amazon RDS (Database) - + Amazon RDS (Relational Database Service) is a managed relational database service from AWS that supports MySQL, PostgreSQL, MariaDB, Oracle, and MS SQL Server. It handles provisioning, backups, patching, and replication automatically. RDS is used for transactional databases that require minimal database administration overhead. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/apache-spark@qHMtJFYcGmESiz_VwRwiI.md b/roadmaps/data-engineer/content/apache-spark@qHMtJFYcGmESiz_VwRwiI.md index ead9d2fdc..373c11c15 100644 --- a/roadmaps/data-engineer/content/apache-spark@qHMtJFYcGmESiz_VwRwiI.md +++ b/roadmaps/data-engineer/content/apache-spark@qHMtJFYcGmESiz_VwRwiI.md @@ -1,5 +1,5 @@ # Apache Spark - + Apache Spark is a distributed data processing engine for large-scale batch and streaming workloads. It processes data in memory across a cluster, making it significantly faster than MapReduce for many workloads. Spark supports Python, Scala, Java, and R, and provides APIs for SQL, streaming, machine learning, and graph processing. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/apis@cxTriSZvrmXP4axKynIZW.md b/roadmaps/data-engineer/content/apis@cxTriSZvrmXP4axKynIZW.md index e0a60a8a4..8c4225c5a 100644 --- a/roadmaps/data-engineer/content/apis@cxTriSZvrmXP4axKynIZW.md +++ b/roadmaps/data-engineer/content/apis@cxTriSZvrmXP4axKynIZW.md @@ -1,5 +1,5 @@ # APIs - + APIs (Application Programming Interfaces) expose data from external services in a structured format, typically JSON or XML over HTTP. Many data pipelines pull data from third-party APIs such as payment processors, marketing platforms, or social networks. Rate limits, authentication, and schema changes are common challenges when ingesting from APIs. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/argocd@PUzHbjwntTSj1REL_dAov.md b/roadmaps/data-engineer/content/argocd@PUzHbjwntTSj1REL_dAov.md index aec5b8b0b..09017f517 100644 --- a/roadmaps/data-engineer/content/argocd@PUzHbjwntTSj1REL_dAov.md +++ b/roadmaps/data-engineer/content/argocd@PUzHbjwntTSj1REL_dAov.md @@ -1,5 +1,5 @@ # ArgoCD - + Argo CD is a declarative GitOps continuous delivery tool for Kubernetes. It continuously monitors a Git repository and ensures that the state of the Kubernetes cluster matches the desired state defined in code. Argo CD is used in data engineering to manage Kubernetes-based pipeline deployments and infrastructure changes through Git. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/async-vs-sync-communication@VefHaP7rIOcZVFzglyn66.md b/roadmaps/data-engineer/content/async-vs-sync-communication@VefHaP7rIOcZVFzglyn66.md index 8e8b2bbc6..05a054fed 100644 --- a/roadmaps/data-engineer/content/async-vs-sync-communication@VefHaP7rIOcZVFzglyn66.md +++ b/roadmaps/data-engineer/content/async-vs-sync-communication@VefHaP7rIOcZVFzglyn66.md @@ -1,5 +1,5 @@ # Async vs Sync Communication - + Synchronous communication means the sender waits for a response before continuing. Asynchronous communication means the sender sends a message and continues without waiting. Messaging systems enable asynchronous communication, which is better suited for high-throughput pipelines where blocking would create bottlenecks. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/aws-eks@eVqcYI2Sy2Dldl3SfxB2C.md b/roadmaps/data-engineer/content/aws-eks@eVqcYI2Sy2Dldl3SfxB2C.md index 883ef00c4..aabc17c88 100644 --- a/roadmaps/data-engineer/content/aws-eks@eVqcYI2Sy2Dldl3SfxB2C.md +++ b/roadmaps/data-engineer/content/aws-eks@eVqcYI2Sy2Dldl3SfxB2C.md @@ -1,5 +1,5 @@ # AWS EKS - + Amazon EKS (Elastic Kubernetes Service) is AWS's managed Kubernetes service. It runs the Kubernetes control plane across multiple availability zones and integrates with AWS services like IAM, VPC, and ECR. EKS is used to run containerized data pipelines and services on AWS without managing the Kubernetes control plane directly. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/aws-sns@uFeiTRobSymkvCinhwmZV.md b/roadmaps/data-engineer/content/aws-sns@uFeiTRobSymkvCinhwmZV.md index c3b7789b3..d1300edb1 100644 --- a/roadmaps/data-engineer/content/aws-sns@uFeiTRobSymkvCinhwmZV.md +++ b/roadmaps/data-engineer/content/aws-sns@uFeiTRobSymkvCinhwmZV.md @@ -1,5 +1,5 @@ # AWS SNS - + Amazon SNS (Simple Notification Service) is a fully managed pub/sub messaging service from AWS. It allows a single message to be sent to multiple subscribers simultaneously through topics. SNS is commonly used alongside SQS to fan out messages to multiple queues or trigger downstream processing in Lambda functions and data pipelines. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/best-practices@yyJJGinOv3M21MFuqJs0j.md b/roadmaps/data-engineer/content/best-practices@yyJJGinOv3M21MFuqJs0j.md index a0fba7589..932fa4d66 100644 --- a/roadmaps/data-engineer/content/best-practices@yyJJGinOv3M21MFuqJs0j.md +++ b/roadmaps/data-engineer/content/best-practices@yyJJGinOv3M21MFuqJs0j.md @@ -1,5 +1,5 @@ # Best Practices - + Best practices for messaging systems include designing idempotent consumers to handle duplicate delivery, setting appropriate retention and replication policies, monitoring consumer lag, and planning for schema evolution. Dead-letter queues are used to handle messages that fail processing repeatedly without losing them. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/big-data-tools@03BHmPhYkZrJwRvQdmxxr.md b/roadmaps/data-engineer/content/big-data-tools@03BHmPhYkZrJwRvQdmxxr.md index 85910d27a..a7a1eb39c 100644 --- a/roadmaps/data-engineer/content/big-data-tools@03BHmPhYkZrJwRvQdmxxr.md +++ b/roadmaps/data-engineer/content/big-data-tools@03BHmPhYkZrJwRvQdmxxr.md @@ -1,5 +1,5 @@ # Big Data Tools - + Big data tools are designed to process and analyze datasets too large to handle with traditional single-machine tools. They distribute computation across clusters and are optimized for throughput at scale. The most widely used big data processing framework is Apache Spark, with Hadoop-based tools remaining common in legacy environments. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/business-intelligence@zA5QqqBMsqymdiPGFdUnt.md b/roadmaps/data-engineer/content/business-intelligence@zA5QqqBMsqymdiPGFdUnt.md index f4cab7a5e..acc60b5ec 100644 --- a/roadmaps/data-engineer/content/business-intelligence@zA5QqqBMsqymdiPGFdUnt.md +++ b/roadmaps/data-engineer/content/business-intelligence@zA5QqqBMsqymdiPGFdUnt.md @@ -1,5 +1,5 @@ # Business Intelligence - + Business intelligence (BI) refers to the tools and processes used to collect, analyze, and visualize business data to support decisions. BI platforms connect to data warehouses and allow business users to build reports and dashboards without writing code. Common BI tools include Tableau, Power BI, Looker, and Streamlit. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/cap-theorem@AslPFjoakcC44CmPB5nuw.md b/roadmaps/data-engineer/content/cap-theorem@AslPFjoakcC44CmPB5nuw.md index f038edbf1..3e666c466 100644 --- a/roadmaps/data-engineer/content/cap-theorem@AslPFjoakcC44CmPB5nuw.md +++ b/roadmaps/data-engineer/content/cap-theorem@AslPFjoakcC44CmPB5nuw.md @@ -1,5 +1,5 @@ # CAP Theorem - + The CAP theorem states that a distributed system can provide at most two of three guarantees: Consistency, Availability, and Partition Tolerance. In practice, network partitions are unavoidable, so systems must choose between consistency and availability when a partition occurs. This trade-off shapes the design of distributed databases like Cassandra, DynamoDB, and HBase. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/cassandra@QYR8ESN7xhi4ZxcoiZbgn.md b/roadmaps/data-engineer/content/cassandra@QYR8ESN7xhi4ZxcoiZbgn.md index c000d8ada..2b1813565 100644 --- a/roadmaps/data-engineer/content/cassandra@QYR8ESN7xhi4ZxcoiZbgn.md +++ b/roadmaps/data-engineer/content/cassandra@QYR8ESN7xhi4ZxcoiZbgn.md @@ -1,5 +1,5 @@ # Cassandra - + Apache Cassandra is an open-source distributed wide-column database designed for high availability and linear scalability. It has no single point of failure and is optimized for fast writes across multiple data centers. Cassandra is used for time-series data, IoT workloads, and applications requiring continuous uptime. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/choosing-the-right-technologies@_MpdVlvvkrsgzigYMZ_P8.md b/roadmaps/data-engineer/content/choosing-the-right-technologies@_MpdVlvvkrsgzigYMZ_P8.md index a1b5583da..ac2df09c9 100644 --- a/roadmaps/data-engineer/content/choosing-the-right-technologies@_MpdVlvvkrsgzigYMZ_P8.md +++ b/roadmaps/data-engineer/content/choosing-the-right-technologies@_MpdVlvvkrsgzigYMZ_P8.md @@ -1,5 +1,5 @@ # Choosing the Right Technologies - + Selecting the right technology stack depends on data volume, team size, latency requirements, and budget. There is no universal best choice; a small startup may do well with a simple Postgres setup, while a large enterprise may need distributed processing and a cloud data warehouse. The decision involves evaluating trade-offs between cost, complexity, scalability, and maintainability. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/cicd@k2SJ4ELGa4B2ZERDAk1uj.md b/roadmaps/data-engineer/content/cicd@k2SJ4ELGa4B2ZERDAk1uj.md index a4485e7f6..2a8e09b83 100644 --- a/roadmaps/data-engineer/content/cicd@k2SJ4ELGa4B2ZERDAk1uj.md +++ b/roadmaps/data-engineer/content/cicd@k2SJ4ELGa4B2ZERDAk1uj.md @@ -1,5 +1,5 @@ # CI/CD - + CI/CD (Continuous Integration and Continuous Delivery) is a set of practices and tools for automating the testing and deployment of code changes. In data engineering, CI/CD pipelines validate pipeline code, run tests, and deploy updates to production automatically. This reduces manual errors and accelerates the delivery of pipeline changes. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/cloud-architectures@YLfyb_ycgz1hu0yW8SPNE.md b/roadmaps/data-engineer/content/cloud-architectures@YLfyb_ycgz1hu0yW8SPNE.md index 019617f65..b1e025fbd 100644 --- a/roadmaps/data-engineer/content/cloud-architectures@YLfyb_ycgz1hu0yW8SPNE.md +++ b/roadmaps/data-engineer/content/cloud-architectures@YLfyb_ycgz1hu0yW8SPNE.md @@ -1,5 +1,5 @@ # Cloud Architectures - + Cloud architectures describe how systems are designed to run on cloud infrastructure. Common patterns include multi-tier architectures, microservices, event-driven designs, and serverless functions. Good cloud architecture balances cost, reliability, scalability, and security. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/cloud-computing@lDeSL9qvgQgyAMcWXF7Fr.md b/roadmaps/data-engineer/content/cloud-computing@lDeSL9qvgQgyAMcWXF7Fr.md index ee47ae068..5378727f5 100644 --- a/roadmaps/data-engineer/content/cloud-computing@lDeSL9qvgQgyAMcWXF7Fr.md +++ b/roadmaps/data-engineer/content/cloud-computing@lDeSL9qvgQgyAMcWXF7Fr.md @@ -1,5 +1,5 @@ # Cloud Computing - + Cloud computing refers to the delivery of computing resources, including servers, storage, databases, networking, and software, over the internet. Major cloud providers offer on-demand infrastructure that scales with usage and is billed per consumption. Cloud platforms are the dominant environment for modern data engineering work. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/cluster-computing-basics@hB0y8A2U3owpAbTUb7LN5.md b/roadmaps/data-engineer/content/cluster-computing-basics@hB0y8A2U3owpAbTUb7LN5.md index 511a1c0ff..712d37dc6 100644 --- a/roadmaps/data-engineer/content/cluster-computing-basics@hB0y8A2U3owpAbTUb7LN5.md +++ b/roadmaps/data-engineer/content/cluster-computing-basics@hB0y8A2U3owpAbTUb7LN5.md @@ -1,3 +1,3 @@ # Cluster Computing Basics - + Cluster computing refers to using a group of connected machines that work together as a single system to process data. It enables workloads that are too large or slow for a single machine by distributing computation across multiple nodes. Concepts like job scheduling, distributed file systems, and resource management are central to working with clusters. \ No newline at end of file diff --git a/roadmaps/data-engineer/content/column@fBD6ZQoMac8w4kMJw_Jrd.md b/roadmaps/data-engineer/content/column@fBD6ZQoMac8w4kMJw_Jrd.md index bba00b5d0..4136f9117 100644 --- a/roadmaps/data-engineer/content/column@fBD6ZQoMac8w4kMJw_Jrd.md +++ b/roadmaps/data-engineer/content/column@fBD6ZQoMac8w4kMJw_Jrd.md @@ -1,5 +1,5 @@ # Column - + Column-family databases (also called wide-column stores) organize data into rows and dynamic columns grouped into column families. They are optimized for read and write operations on large datasets spread across many machines. This model is well suited for time-series data, logging, and analytical workloads. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/compute-engine-compute@-cU86vJWJmlmPHXDCo31o.md b/roadmaps/data-engineer/content/compute-engine-compute@-cU86vJWJmlmPHXDCo31o.md index f04886a3a..288daaad3 100644 --- a/roadmaps/data-engineer/content/compute-engine-compute@-cU86vJWJmlmPHXDCo31o.md +++ b/roadmaps/data-engineer/content/compute-engine-compute@-cU86vJWJmlmPHXDCo31o.md @@ -1,5 +1,5 @@ # Compute Engine (Compute) - + Google Cloud Compute Engine provides virtual machine instances on Google's infrastructure. It supports custom machine types, preemptible VMs for cost savings, and integration with other Google Cloud services. Compute Engine is used for running custom workloads, data processing jobs, and services that require full control over the operating environment. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/containers--orchestration@eTHitN2erd6z8-MZiXE9s.md b/roadmaps/data-engineer/content/containers--orchestration@eTHitN2erd6z8-MZiXE9s.md index 7c551444b..7281435d0 100644 --- a/roadmaps/data-engineer/content/containers--orchestration@eTHitN2erd6z8-MZiXE9s.md +++ b/roadmaps/data-engineer/content/containers--orchestration@eTHitN2erd6z8-MZiXE9s.md @@ -1,5 +1,5 @@ # Containers & Orchestration - + Containers package an application and its dependencies into a portable, isolated unit that runs consistently across environments. Container orchestration automates the deployment, scaling, and management of these containers across a cluster. Together, containers and orchestration form the foundation for running modern data workloads in cloud and hybrid environments. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/couchdb@-IesOBWPSIlbgvTjBqHcb.md b/roadmaps/data-engineer/content/couchdb@-IesOBWPSIlbgvTjBqHcb.md index e68ba3d25..1fd76ae32 100644 --- a/roadmaps/data-engineer/content/couchdb@-IesOBWPSIlbgvTjBqHcb.md +++ b/roadmaps/data-engineer/content/couchdb@-IesOBWPSIlbgvTjBqHcb.md @@ -1,5 +1,5 @@ # CouchDB - + Apache CouchDB is an open-source document database that uses JSON for documents and HTTP as its API. It is designed for reliability and offline-first use cases, with a built-in replication protocol that syncs data between devices and servers. CouchDB is used in scenarios where data needs to be available and writable even without a network connection. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/data-analytics@V30v5RLQrWSMBUIsZQG1o.md b/roadmaps/data-engineer/content/data-analytics@V30v5RLQrWSMBUIsZQG1o.md index 76569169a..56fcf6384 100644 --- a/roadmaps/data-engineer/content/data-analytics@V30v5RLQrWSMBUIsZQG1o.md +++ b/roadmaps/data-engineer/content/data-analytics@V30v5RLQrWSMBUIsZQG1o.md @@ -1,5 +1,5 @@ # Data Analytics - + Data analytics is the process of examining datasets to draw conclusions and support decision-making. It covers a spectrum from descriptive analytics (what happened) to diagnostic (why it happened), predictive (what might happen), and prescriptive (what to do). Data engineers build the infrastructure that makes analytics possible by ensuring clean, accessible, and timely data. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/data-collection-considerations@wDDWQgMVBYK4WcmHq_d6l.md b/roadmaps/data-engineer/content/data-collection-considerations@wDDWQgMVBYK4WcmHq_d6l.md index da2b92a1e..5bbe0d6e3 100644 --- a/roadmaps/data-engineer/content/data-collection-considerations@wDDWQgMVBYK4WcmHq_d6l.md +++ b/roadmaps/data-engineer/content/data-collection-considerations@wDDWQgMVBYK4WcmHq_d6l.md @@ -1,5 +1,5 @@ # Data Collection Considerations - + When collecting data, engineers must account for reliability, latency, volume, and schema consistency. Other considerations include data privacy regulations, deduplication, and handling of missing or malformed records. Good collection design reduces problems downstream in the pipeline. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/data-engineering-lifecycle@Ouph2bHeLQsrHl45ar4Cs.md b/roadmaps/data-engineer/content/data-engineering-lifecycle@Ouph2bHeLQsrHl45ar4Cs.md index 2f1e6a273..de6a1cebb 100644 --- a/roadmaps/data-engineer/content/data-engineering-lifecycle@Ouph2bHeLQsrHl45ar4Cs.md +++ b/roadmaps/data-engineer/content/data-engineering-lifecycle@Ouph2bHeLQsrHl45ar4Cs.md @@ -1,5 +1,5 @@ # Data Engineering Lifecycle - + The data engineering lifecycle describes the stages data moves through from creation to consumption. These stages typically include generation, ingestion, storage, transformation, and serving. Each stage has its own tools, failure modes, and design considerations. Understanding the full lifecycle helps engineers make better decisions about architecture and tooling. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/data-engineering-lifecycle@w3cfuNC-IdUKA7CEXs0fT.md b/roadmaps/data-engineer/content/data-engineering-lifecycle@w3cfuNC-IdUKA7CEXs0fT.md index 2f1e6a273..de6a1cebb 100644 --- a/roadmaps/data-engineer/content/data-engineering-lifecycle@w3cfuNC-IdUKA7CEXs0fT.md +++ b/roadmaps/data-engineer/content/data-engineering-lifecycle@w3cfuNC-IdUKA7CEXs0fT.md @@ -1,5 +1,5 @@ # Data Engineering Lifecycle - + The data engineering lifecycle describes the stages data moves through from creation to consumption. These stages typically include generation, ingestion, storage, transformation, and serving. Each stage has its own tools, failure modes, and design considerations. Understanding the full lifecycle helps engineers make better decisions about architecture and tooling. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/data-fabric@-x3QLMYhC67VJQ6EW6BrJ.md b/roadmaps/data-engineer/content/data-fabric@-x3QLMYhC67VJQ6EW6BrJ.md index 33270d516..77e85b42d 100644 --- a/roadmaps/data-engineer/content/data-fabric@-x3QLMYhC67VJQ6EW6BrJ.md +++ b/roadmaps/data-engineer/content/data-fabric@-x3QLMYhC67VJQ6EW6BrJ.md @@ -1,5 +1,5 @@ # Data Fabric - + Data fabric is an architectural concept that aims to provide a unified layer for accessing and managing data across heterogeneous environments, including on-premises and multiple clouds. It uses metadata, automation, and integration patterns to connect disparate data sources. Data fabric focuses on making data discoverable and accessible without requiring it to be moved to a central location. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/data-generation@AWf1y87pd1JFW71cZ_iE1.md b/roadmaps/data-engineer/content/data-generation@AWf1y87pd1JFW71cZ_iE1.md index e48333cf9..8db74c94c 100644 --- a/roadmaps/data-engineer/content/data-generation@AWf1y87pd1JFW71cZ_iE1.md +++ b/roadmaps/data-engineer/content/data-generation@AWf1y87pd1JFW71cZ_iE1.md @@ -1,5 +1,5 @@ # Data Generation - + Data generation refers to how raw data is produced and originates in a system. Data can come from user interactions, application logs, IoT sensors, databases, APIs, and many other sources. Understanding where data comes from and how it is structured at the source is the starting point for any data pipeline design. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/data-hub@OiWleAdMbPtisrJpk2eSJ.md b/roadmaps/data-engineer/content/data-hub@OiWleAdMbPtisrJpk2eSJ.md index d1bf68893..f7e121a99 100644 --- a/roadmaps/data-engineer/content/data-hub@OiWleAdMbPtisrJpk2eSJ.md +++ b/roadmaps/data-engineer/content/data-hub@OiWleAdMbPtisrJpk2eSJ.md @@ -1,5 +1,5 @@ # Data Hub - + A data hub is a centralized platform that acts as an integration point for data flowing between multiple systems. Unlike a data warehouse, a data hub focuses on data movement and integration rather than storage for analytics. It often combines features of a message broker, metadata catalog, and integration layer. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/data-lake@y0Lxz_wVyQ6lr1hvCsufa.md b/roadmaps/data-engineer/content/data-lake@y0Lxz_wVyQ6lr1hvCsufa.md index 9d573d733..5b622512c 100644 --- a/roadmaps/data-engineer/content/data-lake@y0Lxz_wVyQ6lr1hvCsufa.md +++ b/roadmaps/data-engineer/content/data-lake@y0Lxz_wVyQ6lr1hvCsufa.md @@ -1,5 +1,5 @@ # Data Lake - + A data lake is a centralized storage repository that holds large amounts of raw data in its native format, including structured, semi-structured, and unstructured data. Unlike a data warehouse, a data lake does not enforce a schema on ingestion. Data is stored cheaply at scale and processed when needed, which enables flexibility for future analysis. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/data-modelling-techniques@SlQHO8n97F7-_fc6EUXlj.md b/roadmaps/data-engineer/content/data-modelling-techniques@SlQHO8n97F7-_fc6EUXlj.md index 834cbef18..be615775d 100644 --- a/roadmaps/data-engineer/content/data-modelling-techniques@SlQHO8n97F7-_fc6EUXlj.md +++ b/roadmaps/data-engineer/content/data-modelling-techniques@SlQHO8n97F7-_fc6EUXlj.md @@ -1,5 +1,5 @@ # Data Modelling Techniques - + Data modelling is the process of defining how data is structured and related within a storage system. Common techniques include entity-relationship (ER) modelling for transactional databases and dimensional modelling (star and snowflake schemas) for analytics. The choice of model affects query performance, flexibility, and how easy it is to evolve the schema over time. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/data-normalization@kVPEoUX-ZAGwstieD20Qa.md b/roadmaps/data-engineer/content/data-normalization@kVPEoUX-ZAGwstieD20Qa.md index 8db14169b..3986bd786 100644 --- a/roadmaps/data-engineer/content/data-normalization@kVPEoUX-ZAGwstieD20Qa.md +++ b/roadmaps/data-engineer/content/data-normalization@kVPEoUX-ZAGwstieD20Qa.md @@ -1,5 +1,5 @@ # Data Normalization - + Data normalization is the process of organizing a relational database to reduce redundancy and improve data integrity. It involves decomposing tables into smaller, related ones according to normal forms (1NF, 2NF, 3NF, etc.). Normalized schemas are easier to maintain but may require more joins when querying. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/data-quality@cStrYgFZA2NuYq8TdWWP_.md b/roadmaps/data-engineer/content/data-quality@cStrYgFZA2NuYq8TdWWP_.md index 80177c6d6..a3a761bc9 100644 --- a/roadmaps/data-engineer/content/data-quality@cStrYgFZA2NuYq8TdWWP_.md +++ b/roadmaps/data-engineer/content/data-quality@cStrYgFZA2NuYq8TdWWP_.md @@ -1,3 +1,3 @@ # Data Quality - + Data quality refers to how well data meets the requirements for its intended use in terms of accuracy, completeness, consistency, timeliness, and validity. Poor data quality leads to incorrect analysis and poor decisions. Data engineers implement quality checks at ingestion and transformation stages to catch and prevent data issues. \ No newline at end of file diff --git a/roadmaps/data-engineer/content/data-storage@wydtifF3ZhMWCbVt8Hd2t.md b/roadmaps/data-engineer/content/data-storage@wydtifF3ZhMWCbVt8Hd2t.md index 1c6a6f2ae..880346d48 100644 --- a/roadmaps/data-engineer/content/data-storage@wydtifF3ZhMWCbVt8Hd2t.md +++ b/roadmaps/data-engineer/content/data-storage@wydtifF3ZhMWCbVt8Hd2t.md @@ -1,5 +1,5 @@ # Data Storage - + Data storage in the engineering lifecycle refers to where and how data is persisted after it is generated or ingested. The choice of storage system depends on access patterns, data volume, latency requirements, and cost. Options range from relational databases to object storage, data lakes, and columnar warehouses. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/data-structures-and-algorithms@fqmn6DPOA5MH7UWYv6ayn.md b/roadmaps/data-engineer/content/data-structures-and-algorithms@fqmn6DPOA5MH7UWYv6ayn.md index 8cfa8651b..82ae67ca6 100644 --- a/roadmaps/data-engineer/content/data-structures-and-algorithms@fqmn6DPOA5MH7UWYv6ayn.md +++ b/roadmaps/data-engineer/content/data-structures-and-algorithms@fqmn6DPOA5MH7UWYv6ayn.md @@ -1,5 +1,5 @@ # Data Structures and Algorithms - + Data structures and algorithms form the foundation for writing efficient code. This knowledge is relevant for data engineers when optimizing queries, designing storage schemas, and building processing logic that scales. Common topics include arrays, hash maps, trees, sorting, and complexity analysis. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/data-warehouse@ArOoKuf9scAURs8NRjAru.md b/roadmaps/data-engineer/content/data-warehouse@ArOoKuf9scAURs8NRjAru.md index 012821a89..5ba1f22b6 100644 --- a/roadmaps/data-engineer/content/data-warehouse@ArOoKuf9scAURs8NRjAru.md +++ b/roadmaps/data-engineer/content/data-warehouse@ArOoKuf9scAURs8NRjAru.md @@ -1,5 +1,5 @@ # Data Warehouse - + A data warehouse stores structured, processed data from operational systems, optimized for analytical queries. It typically uses columnar storage and is populated through ETL or ELT processes. Common cloud data warehouses include Google BigQuery, Snowflake, and Amazon Redshift. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/data-warehousing-architectures@J854xPM1X0BWlhtJw7Hs_.md b/roadmaps/data-engineer/content/data-warehousing-architectures@J854xPM1X0BWlhtJw7Hs_.md index e076ff87c..73884212d 100644 --- a/roadmaps/data-engineer/content/data-warehousing-architectures@J854xPM1X0BWlhtJw7Hs_.md +++ b/roadmaps/data-engineer/content/data-warehousing-architectures@J854xPM1X0BWlhtJw7Hs_.md @@ -1,3 +1,3 @@ # Data Warehousing Architectures - + Data warehousing architectures describe how data is organized, stored, and accessed across a warehouse system. Common patterns include traditional ETL-based warehouses, cloud-native warehouses, data lakehouse architectures, and federated query systems. The choice of architecture affects cost, query performance, scalability, and how fresh the data available for analysis is. \ No newline at end of file diff --git a/roadmaps/data-engineer/content/database-fundamentals@g4UC0go7OPCJYJlac9w-i.md b/roadmaps/data-engineer/content/database-fundamentals@g4UC0go7OPCJYJlac9w-i.md index b939c5558..f26191776 100644 --- a/roadmaps/data-engineer/content/database-fundamentals@g4UC0go7OPCJYJlac9w-i.md +++ b/roadmaps/data-engineer/content/database-fundamentals@g4UC0go7OPCJYJlac9w-i.md @@ -1,5 +1,5 @@ # Database Fundamentals - + Database fundamentals cover the core concepts that apply across most relational database systems: how data is organized into tables, how queries are executed, and how the database ensures consistency and durability. Topics include normalization, indexing, transactions, and query optimization. These concepts apply whether using PostgreSQL, MySQL, or any other relational system. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/docker@OQ3RqVgWEMxpAtrrjOG5U.md b/roadmaps/data-engineer/content/docker@OQ3RqVgWEMxpAtrrjOG5U.md index ad1683d96..bef803b2e 100644 --- a/roadmaps/data-engineer/content/docker@OQ3RqVgWEMxpAtrrjOG5U.md +++ b/roadmaps/data-engineer/content/docker@OQ3RqVgWEMxpAtrrjOG5U.md @@ -1,5 +1,5 @@ # Docker - + Docker is the most widely used platform for building, shipping, and running containers. It packages code and its dependencies into a lightweight, portable image that runs the same in any environment. Data engineers use Docker to containerize pipeline code, ensure reproducible environments, and simplify deployment. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/dynamodb@BDfpCDOxXZ-Tp0Abj_CVW.md b/roadmaps/data-engineer/content/dynamodb@BDfpCDOxXZ-Tp0Abj_CVW.md index bdccba255..9d11ff799 100644 --- a/roadmaps/data-engineer/content/dynamodb@BDfpCDOxXZ-Tp0Abj_CVW.md +++ b/roadmaps/data-engineer/content/dynamodb@BDfpCDOxXZ-Tp0Abj_CVW.md @@ -1,5 +1,5 @@ # DynamoDB - + Amazon DynamoDB is a fully managed key-value and document database service on AWS. It provides single-digit millisecond performance at any scale and handles replication and scaling automatically. DynamoDB is commonly used for applications that require predictable performance and high availability without database administration. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/ecpa@g1VwuSupohuDAT2O4hTXx.md b/roadmaps/data-engineer/content/ecpa@g1VwuSupohuDAT2O4hTXx.md index e80da8643..2c62238a9 100644 --- a/roadmaps/data-engineer/content/ecpa@g1VwuSupohuDAT2O4hTXx.md +++ b/roadmaps/data-engineer/content/ecpa@g1VwuSupohuDAT2O4hTXx.md @@ -1,5 +1,5 @@ # ECPA - + The Electronic Communications Privacy Act (ECPA) is a US federal law that regulates government access to electronic communications and stored data. It sets rules for when law enforcement can intercept communications or compel disclosure of stored data from service providers. Data engineers working with communication data must be aware of ECPA requirements when designing storage and access controls. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/elasticsearch@_F53cV3ln2yu0ics5BFfx.md b/roadmaps/data-engineer/content/elasticsearch@_F53cV3ln2yu0ics5BFfx.md index c07f69e44..4a2a80bae 100644 --- a/roadmaps/data-engineer/content/elasticsearch@_F53cV3ln2yu0ics5BFfx.md +++ b/roadmaps/data-engineer/content/elasticsearch@_F53cV3ln2yu0ics5BFfx.md @@ -1,5 +1,5 @@ # Elasticsearch - + Elasticsearch is a distributed search and analytics engine built on Apache Lucene. It is designed for full-text search, log analysis, and real-time data exploration. Elasticsearch is commonly used as the backend for search features in applications and as a centralized store for log and event data, often alongside Kibana. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/etl-vs-reverse-etl@LMFREK9dH_7qzx_s2xCjI.md b/roadmaps/data-engineer/content/etl-vs-reverse-etl@LMFREK9dH_7qzx_s2xCjI.md index 2a0ae2f62..85617f789 100644 --- a/roadmaps/data-engineer/content/etl-vs-reverse-etl@LMFREK9dH_7qzx_s2xCjI.md +++ b/roadmaps/data-engineer/content/etl-vs-reverse-etl@LMFREK9dH_7qzx_s2xCjI.md @@ -1,5 +1,5 @@ # ETL vs Reverse ETL - + ETL (Extract, Transform, Load) moves data from operational systems into a data warehouse for analysis. Reverse ETL goes in the opposite direction, syncing processed data from the warehouse back into operational tools like Salesforce, HubSpot, or Intercom. The two patterns are complementary and together form a complete data activation workflow. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/gdpr@MuPHohc7mJzcH5QdJ-K46.md b/roadmaps/data-engineer/content/gdpr@MuPHohc7mJzcH5QdJ-K46.md index 2f85c6286..5f278f590 100644 --- a/roadmaps/data-engineer/content/gdpr@MuPHohc7mJzcH5QdJ-K46.md +++ b/roadmaps/data-engineer/content/gdpr@MuPHohc7mJzcH5QdJ-K46.md @@ -1,5 +1,5 @@ # GDPR - + GDPR (General Data Protection Regulation) is a data privacy law enacted by the European Union that governs how personal data of EU residents is collected, stored, processed, and shared. It grants individuals rights over their data, including the right to access, correct, and delete it. Data engineers must design systems that support these rights and comply with GDPR requirements such as data minimization and purpose limitation. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/git-and-github@02TADW_PPVtTU_rWV3jf1.md b/roadmaps/data-engineer/content/git-and-github@02TADW_PPVtTU_rWV3jf1.md index 3e6f530fa..ccdeb149c 100644 --- a/roadmaps/data-engineer/content/git-and-github@02TADW_PPVtTU_rWV3jf1.md +++ b/roadmaps/data-engineer/content/git-and-github@02TADW_PPVtTU_rWV3jf1.md @@ -1,10 +1,10 @@ # Git and GitHub - + Git is a distributed version control system that tracks changes to code over time. GitHub is a platform built on top of Git that adds collaboration features like pull requests, code review, and CI/CD integrations. Data engineers use Git to manage pipeline code, infrastructure configurations, and shared scripts across teams. Visit the following resources to learn more: - [@roadmap@Visit Dedicated Git & GitHub Roadmap](https://roadmap.sh/git-github) +- [@course@Why use Git? (Interactive Lesson)](https://inter-git.com/lessons/introduction) - [@article@Git by Example - Learn Version Control with Bite-sized Lessons](https://antonz.org/git-by-example/) -- [@video@Git & GitHub Crash Course For Beginners](https://www.youtube.com/watch?v=SWYqp7iY_Tc) -- [@course@Why use Git? (Interactive Lesson)](https://inter-git.com/lessons/introduction) \ No newline at end of file +- [@video@Git & GitHub Crash Course For Beginners](https://www.youtube.com/watch?v=SWYqp7iY_Tc) \ No newline at end of file diff --git a/roadmaps/data-engineer/content/github-actions@N8vpCfSdZCADwO_qceWBK.md b/roadmaps/data-engineer/content/github-actions@N8vpCfSdZCADwO_qceWBK.md index d463b162b..d806f84c1 100644 --- a/roadmaps/data-engineer/content/github-actions@N8vpCfSdZCADwO_qceWBK.md +++ b/roadmaps/data-engineer/content/github-actions@N8vpCfSdZCADwO_qceWBK.md @@ -1,5 +1,5 @@ # GitHub Actions - + GitHub Actions is a CI/CD platform built into GitHub. It allows developers to define automated workflows as YAML files that trigger on code events like pushes and pull requests. GitHub Actions is widely used to run tests, lint code, build Docker images, and deploy data pipelines. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/glue-etl@nD36-PXHzOXePM7j9u_O_.md b/roadmaps/data-engineer/content/glue-etl@nD36-PXHzOXePM7j9u_O_.md index 481d10580..97b55d59b 100644 --- a/roadmaps/data-engineer/content/glue-etl@nD36-PXHzOXePM7j9u_O_.md +++ b/roadmaps/data-engineer/content/glue-etl@nD36-PXHzOXePM7j9u_O_.md @@ -1,5 +1,5 @@ # Glue (ETL) - + AWS Glue is a fully managed ETL service that automates the discovery, cataloging, and transformation of data. It includes a data catalog for storing metadata, a job scheduler, and a serverless Spark environment for running transformations. Glue is commonly used to move and transform data between S3, Redshift, and other AWS services. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/go@4z2i5NXTo9h3YY0kJvRrz.md b/roadmaps/data-engineer/content/go@4z2i5NXTo9h3YY0kJvRrz.md index 328d3496a..4b6e4d633 100644 --- a/roadmaps/data-engineer/content/go@4z2i5NXTo9h3YY0kJvRrz.md +++ b/roadmaps/data-engineer/content/go@4z2i5NXTo9h3YY0kJvRrz.md @@ -1,5 +1,5 @@ # Go - + Go is a compiled language developed by Google, known for its simplicity, fast execution, and strong concurrency model. In data engineering, it is used to build lightweight, high-throughput services and tools. Its performance characteristics make it a good fit for data pipeline components where latency and resource efficiency matter. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/google-cloud-gke@8qEgXYZEbDWC73SQSflDY.md b/roadmaps/data-engineer/content/google-cloud-gke@8qEgXYZEbDWC73SQSflDY.md index 37a678eb2..7a6641dcd 100644 --- a/roadmaps/data-engineer/content/google-cloud-gke@8qEgXYZEbDWC73SQSflDY.md +++ b/roadmaps/data-engineer/content/google-cloud-gke@8qEgXYZEbDWC73SQSflDY.md @@ -1,5 +1,5 @@ # Google Cloud GKE - + Google Kubernetes Engine (GKE) is Google Cloud's managed Kubernetes service. It handles cluster provisioning, upgrades, and scaling automatically, reducing the operational burden of running Kubernetes. GKE is used to run containerized data workloads on Google Cloud infrastructure. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/graph@W6RnhoD7fW2xzVwnyJEDr.md b/roadmaps/data-engineer/content/graph@W6RnhoD7fW2xzVwnyJEDr.md index 190e84423..088d5e035 100644 --- a/roadmaps/data-engineer/content/graph@W6RnhoD7fW2xzVwnyJEDr.md +++ b/roadmaps/data-engineer/content/graph@W6RnhoD7fW2xzVwnyJEDr.md @@ -1,5 +1,5 @@ # Graph - + Graph databases store data as nodes and edges, representing entities and the relationships between them. They are optimized for queries that traverse relationships, such as finding connections between users or mapping dependencies. Graph databases are used in social networks, fraud detection, recommendation engines, and knowledge graphs. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/indexing@ilbFKqhfYyykjJ7cOngwx.md b/roadmaps/data-engineer/content/indexing@ilbFKqhfYyykjJ7cOngwx.md index c6ce2029a..728989ce9 100644 --- a/roadmaps/data-engineer/content/indexing@ilbFKqhfYyykjJ7cOngwx.md +++ b/roadmaps/data-engineer/content/indexing@ilbFKqhfYyykjJ7cOngwx.md @@ -1,3 +1,3 @@ # Indexing - + Indexes are data structures that speed up query performance by allowing the database to find rows without scanning the entire table. They are created on one or more columns and come in various types, including B-tree, hash, and full-text indexes. Indexes improve read performance but add overhead to writes and storage. \ No newline at end of file diff --git a/roadmaps/data-engineer/content/infrastructure-as-code---iac@jgz7L8OSuqRNcf9buuMTj.md b/roadmaps/data-engineer/content/infrastructure-as-code---iac@jgz7L8OSuqRNcf9buuMTj.md index 0b9e16c74..67ad3f052 100644 --- a/roadmaps/data-engineer/content/infrastructure-as-code---iac@jgz7L8OSuqRNcf9buuMTj.md +++ b/roadmaps/data-engineer/content/infrastructure-as-code---iac@jgz7L8OSuqRNcf9buuMTj.md @@ -1,5 +1,5 @@ # Infrastructure as Code - IaC - + Infrastructure as Code (IaC) is the practice of managing and provisioning infrastructure through machine-readable configuration files rather than manual processes. Tools like Terraform, AWS CloudFormation, and Pulumi allow teams to define infrastructure declaratively and version it in Git. IaC makes infrastructure reproducible, auditable, and easier to manage at scale. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/introduction@WSYIFni7G2C9Jr0pwuami.md b/roadmaps/data-engineer/content/introduction@WSYIFni7G2C9Jr0pwuami.md index 8f37db337..f378503d2 100644 --- a/roadmaps/data-engineer/content/introduction@WSYIFni7G2C9Jr0pwuami.md +++ b/roadmaps/data-engineer/content/introduction@WSYIFni7G2C9Jr0pwuami.md @@ -1,5 +1,5 @@ # Introduction - + Data engineering is the discipline of designing, building, and maintaining systems that collect, store, and process data at scale. It sits between raw data sources and the analysts, scientists, and applications that consume that data. The work involves building pipelines, managing storage infrastructure, and ensuring data is reliable and accessible. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/kubernetes@I_IueX1DFp-LmBwr1-suX.md b/roadmaps/data-engineer/content/kubernetes@I_IueX1DFp-LmBwr1-suX.md index a995e256a..53ee1f044 100644 --- a/roadmaps/data-engineer/content/kubernetes@I_IueX1DFp-LmBwr1-suX.md +++ b/roadmaps/data-engineer/content/kubernetes@I_IueX1DFp-LmBwr1-suX.md @@ -1,5 +1,5 @@ # Kubernetes - + Kubernetes is an open-source container orchestration system that automates the deployment, scaling, and management of containerized applications. In data engineering, it is used to run pipeline workers, schedule jobs, and manage microservices. Kubernetes has become the standard infrastructure layer for modern data platforms. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/kubernetes@kcgDW6AFW7WXzXMTPE6J-.md b/roadmaps/data-engineer/content/kubernetes@kcgDW6AFW7WXzXMTPE6J-.md index 9df88656d..7647817bd 100644 --- a/roadmaps/data-engineer/content/kubernetes@kcgDW6AFW7WXzXMTPE6J-.md +++ b/roadmaps/data-engineer/content/kubernetes@kcgDW6AFW7WXzXMTPE6J-.md @@ -1,5 +1,5 @@ # Kubernetes - + Kubernetes is an open-source system for automating the deployment, scaling, and operation of containerized applications. It manages clusters of containers across multiple nodes and handles load balancing, self-healing, and rolling updates. Data engineers use Kubernetes to run distributed processing jobs, schedule pipeline workers, and manage infrastructure at scale. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/memcached@KYUh29Ok1aeOviboGDS_i.md b/roadmaps/data-engineer/content/memcached@KYUh29Ok1aeOviboGDS_i.md index a105e7a9b..58a6d63d4 100644 --- a/roadmaps/data-engineer/content/memcached@KYUh29Ok1aeOviboGDS_i.md +++ b/roadmaps/data-engineer/content/memcached@KYUh29Ok1aeOviboGDS_i.md @@ -1,5 +1,5 @@ # Memcached - + Memcached is a high-performance, distributed in-memory caching system. It is simpler than Redis, supporting only key-value string storage, but is very fast and horizontally scalable. Memcached is commonly used to cache database query results and reduce load on backend systems. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/messages-vs-streams@IZvL-1Xi0R9IuwJ30FDm4.md b/roadmaps/data-engineer/content/messages-vs-streams@IZvL-1Xi0R9IuwJ30FDm4.md index 6fa4ff90b..58f39c153 100644 --- a/roadmaps/data-engineer/content/messages-vs-streams@IZvL-1Xi0R9IuwJ30FDm4.md +++ b/roadmaps/data-engineer/content/messages-vs-streams@IZvL-1Xi0R9IuwJ30FDm4.md @@ -1,3 +1,3 @@ # Messages vs Streams - + Messages are discrete, individual units of data sent from a producer to a consumer. Streams are continuous, ordered sequences of data that consumers process in real time or replay from a position. Systems like RabbitMQ focus on message delivery, while Apache Kafka is designed around the stream abstraction and supports log retention and replay. \ No newline at end of file diff --git a/roadmaps/data-engineer/content/metadata-first-architecture@14CycunRC1p2qTRn-ncoy.md b/roadmaps/data-engineer/content/metadata-first-architecture@14CycunRC1p2qTRn-ncoy.md index 820b76419..43abe3058 100644 --- a/roadmaps/data-engineer/content/metadata-first-architecture@14CycunRC1p2qTRn-ncoy.md +++ b/roadmaps/data-engineer/content/metadata-first-architecture@14CycunRC1p2qTRn-ncoy.md @@ -1,3 +1,3 @@ # Metadata-first Architecture - + A metadata-first architecture treats metadata as a first-class citizen in data system design. Rather than just documenting data after the fact, metadata is captured and used actively to govern, discover, and lineage-track data across the organization. This approach supports better data quality, compliance, and self-service analytics. \ No newline at end of file diff --git a/roadmaps/data-engineer/content/metadata-management@a5gzM8msXibxD58eVDkM-.md b/roadmaps/data-engineer/content/metadata-management@a5gzM8msXibxD58eVDkM-.md index 2ebbdc5ad..817230cbf 100644 --- a/roadmaps/data-engineer/content/metadata-management@a5gzM8msXibxD58eVDkM-.md +++ b/roadmaps/data-engineer/content/metadata-management@a5gzM8msXibxD58eVDkM-.md @@ -1,3 +1,3 @@ # Metadata Management - + Metadata management involves capturing, storing, and governing information about data assets, including their structure, origin, ownership, and usage. A metadata catalog makes data discoverable and helps teams understand what data is available and how it is defined. Tools like Apache Atlas, DataHub, and Alation are used for metadata management. \ No newline at end of file diff --git a/roadmaps/data-engineer/content/microsoft-power-bi@6Nr5FAGT_oOPZwZWdv7hl.md b/roadmaps/data-engineer/content/microsoft-power-bi@6Nr5FAGT_oOPZwZWdv7hl.md index d0382c67c..e2a53c3e2 100644 --- a/roadmaps/data-engineer/content/microsoft-power-bi@6Nr5FAGT_oOPZwZWdv7hl.md +++ b/roadmaps/data-engineer/content/microsoft-power-bi@6Nr5FAGT_oOPZwZWdv7hl.md @@ -1,5 +1,5 @@ # Microsoft Power BI - + Microsoft Power BI is a business intelligence and data visualization platform from Microsoft. It connects to a wide range of data sources and allows users to build interactive dashboards and reports. Power BI integrates tightly with the Microsoft ecosystem including Azure and Excel. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/monitoring@dk5FQl7Pk3-O5eF7dKwmp.md b/roadmaps/data-engineer/content/monitoring@dk5FQl7Pk3-O5eF7dKwmp.md index ae13e3a00..05d79c57b 100644 --- a/roadmaps/data-engineer/content/monitoring@dk5FQl7Pk3-O5eF7dKwmp.md +++ b/roadmaps/data-engineer/content/monitoring@dk5FQl7Pk3-O5eF7dKwmp.md @@ -1,5 +1,5 @@ # Monitoring - + Monitoring is the practice of collecting and analyzing metrics, logs, and events from running systems to track their health and performance. For data pipelines, monitoring covers job success rates, data freshness, latency, and resource usage. Good monitoring enables fast detection and diagnosis of failures before they affect downstream consumers. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/ms-sql@YxnIQh6Y5ic795-YsajB8.md b/roadmaps/data-engineer/content/ms-sql@YxnIQh6Y5ic795-YsajB8.md index 8465a0089..9a8a6e8f7 100644 --- a/roadmaps/data-engineer/content/ms-sql@YxnIQh6Y5ic795-YsajB8.md +++ b/roadmaps/data-engineer/content/ms-sql@YxnIQh6Y5ic795-YsajB8.md @@ -1,5 +1,5 @@ # MS SQL - + Microsoft SQL Server (MS SQL) is a relational database developed by Microsoft, commonly used in enterprise and Windows-based environments. It integrates tightly with the Microsoft ecosystem, including Azure, Power BI, and .NET. MS SQL supports T-SQL, Microsoft's extension of SQL with additional procedural capabilities. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/neo4j@TG63YRbSKL1F9vlUVF1VY.md b/roadmaps/data-engineer/content/neo4j@TG63YRbSKL1F9vlUVF1VY.md index c8e75d688..92a7b9acc 100644 --- a/roadmaps/data-engineer/content/neo4j@TG63YRbSKL1F9vlUVF1VY.md +++ b/roadmaps/data-engineer/content/neo4j@TG63YRbSKL1F9vlUVF1VY.md @@ -1,5 +1,5 @@ # Neo4j - + Neo4j is the most widely used graph database. It stores data natively as nodes and relationships and uses Cypher, a declarative query language designed for graph traversal. Neo4j is used for applications where relationship-heavy queries are central, such as recommendation systems and network analysis. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/neptune@atAK4zGXIbxZvfBTzFEIe.md b/roadmaps/data-engineer/content/neptune@atAK4zGXIbxZvfBTzFEIe.md index 2943bf086..0ccca4473 100644 --- a/roadmaps/data-engineer/content/neptune@atAK4zGXIbxZvfBTzFEIe.md +++ b/roadmaps/data-engineer/content/neptune@atAK4zGXIbxZvfBTzFEIe.md @@ -1,5 +1,5 @@ # Neptune - + Amazon Neptune is a managed graph database service on AWS that supports both the Property Graph model (with Gremlin) and RDF (with SPARQL). It is designed for highly connected datasets and scales to billions of relationships. Neptune is used for knowledge graphs, fraud detection, and identity resolution. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/networking-fundamentals@cgkzFMmQils2sYj4NW8VW.md b/roadmaps/data-engineer/content/networking-fundamentals@cgkzFMmQils2sYj4NW8VW.md index ed2a3c82e..d48d3f0b7 100644 --- a/roadmaps/data-engineer/content/networking-fundamentals@cgkzFMmQils2sYj4NW8VW.md +++ b/roadmaps/data-engineer/content/networking-fundamentals@cgkzFMmQils2sYj4NW8VW.md @@ -1,5 +1,7 @@ +# undefined + Networking Fundamentals - + Networking fundamentals cover how data moves between systems: IP addressing, DNS, HTTP, TCP/UDP, and firewalls. Data engineers encounter networking when configuring cloud resources, troubleshooting pipeline failures, or setting up secure connections between services. A basic understanding of how networks operate helps diagnose connectivity issues and design reliable architectures. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/nosql-databases@uZYQ8tqTriXt_JIOjcM9_.md b/roadmaps/data-engineer/content/nosql-databases@uZYQ8tqTriXt_JIOjcM9_.md index c739c5120..e79f161ea 100644 --- a/roadmaps/data-engineer/content/nosql-databases@uZYQ8tqTriXt_JIOjcM9_.md +++ b/roadmaps/data-engineer/content/nosql-databases@uZYQ8tqTriXt_JIOjcM9_.md @@ -1,9 +1,17 @@ -# NoSQL Databases - -NoSQL databases are a category of database systems that do not use the traditional relational table model. They are designed for specific data models and access patterns, including document storage, key-value pairs, wide-column stores, and graphs. NoSQL databases are often chosen for their scalability, flexibility, and performance in high-volume or unstructured data scenarios. +# NoSQL databases + +NoSQL databases are a category of database management systems designed for handling unstructured, semi-structured, or rapidly changing data. Unlike traditional relational databases, which use fixed schemas and SQL for querying, NoSQL databases offer flexible data models and can be classified into several types: + +1. **Document Stores**: Store data in JSON, BSON, or XML formats, allowing for flexible and hierarchical data structures (e.g., MongoDB, CouchDB). +2. **Key-Value Stores**: Store data as key-value pairs, suitable for high-speed read and write operations (e.g., Redis, Riak). +3. **Column-Family Stores**: Store data in columns rather than rows, which is useful for handling large volumes of data and wide columnar tables (e.g., Apache Cassandra, HBase). +4. **Graph Databases**: Optimize the storage and querying of data with complex relationships using graph structures (e.g., Neo4j, Amazon Neptune). + +NoSQL databases are often used for applications requiring high scalability, flexibility, and performance, such as real-time analytics, content management systems, and distributed data storage. Visit the following resources to learn more: +- [@article@Types of NoSQL Databases: How to Choose the Right One](https://roadmap.sh/backend/types-of-nosql-databases) - [@article@NoSQL Explained](https://www.mongodb.com/nosql-explained) - [@video@How do NoSQL Databases work](https://www.youtube.com/watch?v=0buKQHokLK8) - [@video@SQL vs NoSQL Explained](https://www.youtube.com/watch?v=ruz-vK8IesE) \ No newline at end of file diff --git a/roadmaps/data-engineer/content/oltp-vs-olap@-VQQmIUGesnrT1N6kH5et.md b/roadmaps/data-engineer/content/oltp-vs-olap@-VQQmIUGesnrT1N6kH5et.md index 33e6a9d92..0db0a87e7 100644 --- a/roadmaps/data-engineer/content/oltp-vs-olap@-VQQmIUGesnrT1N6kH5et.md +++ b/roadmaps/data-engineer/content/oltp-vs-olap@-VQQmIUGesnrT1N6kH5et.md @@ -1,5 +1,5 @@ # OLTP vs OLAP - + OLTP (Online Transaction Processing) systems are optimized for fast, frequent read and write operations, typically backing operational applications. OLAP (Online Analytical Processing) systems are designed for complex queries over large datasets, used in reporting and analytics. The two have different storage formats, indexing strategies, and performance characteristics. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/oracle@PJcxM60h85Po0AAkSj7nr.md b/roadmaps/data-engineer/content/oracle@PJcxM60h85Po0AAkSj7nr.md index b19077d38..de37d41ac 100644 --- a/roadmaps/data-engineer/content/oracle@PJcxM60h85Po0AAkSj7nr.md +++ b/roadmaps/data-engineer/content/oracle@PJcxM60h85Po0AAkSj7nr.md @@ -1,5 +1,5 @@ # Oracle - + Oracle Database is a commercial relational database system widely used in enterprise environments. It is known for its robustness, advanced features, and support for very large-scale deployments. Oracle is common in financial services, healthcare, and government sectors where long-term vendor support and mature tooling are priorities. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/postgresql@__JFgwxeDLvz8p7DAJnsc.md b/roadmaps/data-engineer/content/postgresql@__JFgwxeDLvz8p7DAJnsc.md index 1bd8cbcff..8925b80ac 100644 --- a/roadmaps/data-engineer/content/postgresql@__JFgwxeDLvz8p7DAJnsc.md +++ b/roadmaps/data-engineer/content/postgresql@__JFgwxeDLvz8p7DAJnsc.md @@ -1,5 +1,5 @@ # PostgreSQL - + PostgreSQL is an open-source relational database known for its standards compliance, extensibility, and advanced feature set. It supports complex queries, JSON storage, full-text search, and custom data types. PostgreSQL is widely used in both transactional and analytical workloads. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/rabbitmq@ERcgPTACqYo9BXoRdLjbd.md b/roadmaps/data-engineer/content/rabbitmq@ERcgPTACqYo9BXoRdLjbd.md index 2494c750f..22eb94c49 100644 --- a/roadmaps/data-engineer/content/rabbitmq@ERcgPTACqYo9BXoRdLjbd.md +++ b/roadmaps/data-engineer/content/rabbitmq@ERcgPTACqYo9BXoRdLjbd.md @@ -1,5 +1,5 @@ # RabbitMQ - + RabbitMQ is an open-source message broker that implements the AMQP protocol. It routes messages between producers and consumers using exchanges and queues, supporting patterns like publish/subscribe, work queues, and routing. RabbitMQ is used for task queues, service-to-service communication, and event notification systems. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/redis@dW_eC4vR8BrvKG9wxmEBc.md b/roadmaps/data-engineer/content/redis@dW_eC4vR8BrvKG9wxmEBc.md index 672fbe3eb..872753f1c 100644 --- a/roadmaps/data-engineer/content/redis@dW_eC4vR8BrvKG9wxmEBc.md +++ b/roadmaps/data-engineer/content/redis@dW_eC4vR8BrvKG9wxmEBc.md @@ -1,5 +1,5 @@ # Redis - + Redis is an in-memory key-value store known for its extremely low latency. It supports a variety of data structures including strings, lists, sets, sorted sets, and hashes. Redis is widely used for caching, real-time leaderboards, pub/sub messaging, and session storage. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/relational-databases@cslVSSKBMO7I6CpO7vG1H.md b/roadmaps/data-engineer/content/relational-databases@cslVSSKBMO7I6CpO7vG1H.md index c7d60a66d..060aed12d 100644 --- a/roadmaps/data-engineer/content/relational-databases@cslVSSKBMO7I6CpO7vG1H.md +++ b/roadmaps/data-engineer/content/relational-databases@cslVSSKBMO7I6CpO7vG1H.md @@ -1,5 +1,5 @@ # Relational Databases - + Relational databases store data in tables with rows and columns, and use SQL for querying. Relationships between tables are defined through foreign keys. They are the most widely used type of database for transactional applications and form the backbone of most operational systems. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/reverse-etl-usecases@mBOGrJIUaatBe2PnJM2NK.md b/roadmaps/data-engineer/content/reverse-etl-usecases@mBOGrJIUaatBe2PnJM2NK.md index c38f7776c..7abd11a8b 100644 --- a/roadmaps/data-engineer/content/reverse-etl-usecases@mBOGrJIUaatBe2PnJM2NK.md +++ b/roadmaps/data-engineer/content/reverse-etl-usecases@mBOGrJIUaatBe2PnJM2NK.md @@ -1,3 +1,3 @@ # Reverse ETL Usecases - + Common use cases for reverse ETL include syncing customer health scores to a CRM, pushing segmented user lists to a marketing automation platform, and sending product usage data to customer success tools. It enables business teams to act on insights derived in the data warehouse without needing access to it directly. \ No newline at end of file diff --git a/roadmaps/data-engineer/content/skills-and-responsibilities@3BxbkrBp8veZj38zdwN8s.md b/roadmaps/data-engineer/content/skills-and-responsibilities@3BxbkrBp8veZj38zdwN8s.md index 3fc33cc50..c329765b4 100644 --- a/roadmaps/data-engineer/content/skills-and-responsibilities@3BxbkrBp8veZj38zdwN8s.md +++ b/roadmaps/data-engineer/content/skills-and-responsibilities@3BxbkrBp8veZj38zdwN8s.md @@ -1,5 +1,5 @@ # Skills and Responsibilities - + A data engineer works across a broad set of tools and systems: programming languages, databases, cloud platforms, pipeline orchestration, and distributed computing. Core responsibilities include building and maintaining data pipelines, managing database schemas, optimizing query performance, and ensuring data quality. Collaboration with data scientists, analysts, and software engineers is also a regular part of the role. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/star-vs-snowflake-schema@OfH_UXnxvGQgwlNQwOEfS.md b/roadmaps/data-engineer/content/star-vs-snowflake-schema@OfH_UXnxvGQgwlNQwOEfS.md index a5b80cfad..2773b2483 100644 --- a/roadmaps/data-engineer/content/star-vs-snowflake-schema@OfH_UXnxvGQgwlNQwOEfS.md +++ b/roadmaps/data-engineer/content/star-vs-snowflake-schema@OfH_UXnxvGQgwlNQwOEfS.md @@ -1,3 +1,3 @@ # Star vs Snowflake Schema - + Star and snowflake schemas are two approaches to organizing data in a data warehouse. A star schema has a central fact table connected directly to dimension tables, making queries simple and fast. A snowflake schema normalizes dimension tables into multiple related tables, reducing redundancy but requiring more joins. Star schemas are more common in analytical systems due to their query performance. \ No newline at end of file diff --git a/roadmaps/data-engineer/content/testing@DZoxLu-j1vq5leoXLRZqt.md b/roadmaps/data-engineer/content/testing@DZoxLu-j1vq5leoXLRZqt.md index c1a459663..347570b01 100644 --- a/roadmaps/data-engineer/content/testing@DZoxLu-j1vq5leoXLRZqt.md +++ b/roadmaps/data-engineer/content/testing@DZoxLu-j1vq5leoXLRZqt.md @@ -1,5 +1,5 @@ # Testing - + Testing in data engineering involves verifying that pipelines, transformations, and data outputs behave correctly. This includes unit tests for individual functions, integration tests for pipeline components, and data quality tests that validate the output data itself. A well-tested pipeline catches regressions early and builds confidence in the reliability of data delivered to consumers. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/tokenization@ZAKo9Svb8TQ6KkmOnfB5x.md b/roadmaps/data-engineer/content/tokenization@ZAKo9Svb8TQ6KkmOnfB5x.md index aa7c6254b..c92dd1b93 100644 --- a/roadmaps/data-engineer/content/tokenization@ZAKo9Svb8TQ6KkmOnfB5x.md +++ b/roadmaps/data-engineer/content/tokenization@ZAKo9Svb8TQ6KkmOnfB5x.md @@ -1,5 +1,5 @@ # Tokenization - + Tokenization replaces sensitive data values with non-sensitive placeholders called tokens. The original value is stored securely in a token vault, and the token can be used in systems that do not need the actual data. Tokenization is used to protect payment card numbers, personal identifiers, and other sensitive field Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/what-and-why-use-them@1qju7UlcMo2Ebp4a3BGxH.md b/roadmaps/data-engineer/content/what-and-why-use-them@1qju7UlcMo2Ebp4a3BGxH.md index de695fda1..e77d70d76 100644 --- a/roadmaps/data-engineer/content/what-and-why-use-them@1qju7UlcMo2Ebp4a3BGxH.md +++ b/roadmaps/data-engineer/content/what-and-why-use-them@1qju7UlcMo2Ebp4a3BGxH.md @@ -1,3 +1,3 @@ # What and why use them? - + Messaging systems solve the problem of tight coupling between systems. Instead of one service directly calling another, it sends a message to a broker, and the consumer reads it when ready. This improves reliability, scalability, and flexibility, especially when producers and consumers operate at different speeds or scales. \ No newline at end of file diff --git a/roadmaps/data-engineer/content/what-is-cluster-computing@Ad10evrGQuYRl5GaMhQwu.md b/roadmaps/data-engineer/content/what-is-cluster-computing@Ad10evrGQuYRl5GaMhQwu.md index 65c916715..e60659545 100644 --- a/roadmaps/data-engineer/content/what-is-cluster-computing@Ad10evrGQuYRl5GaMhQwu.md +++ b/roadmaps/data-engineer/content/what-is-cluster-computing@Ad10evrGQuYRl5GaMhQwu.md @@ -1,5 +1,5 @@ # What is Cluster Computing - + Cluster computing is a model where multiple computers are networked together to act as a unified processing system. Tasks are split across nodes in the cluster and executed in parallel. This approach is used in big data processing, scientific computing, and any workload that exceeds single-machine capacity. Visit the following resources to learn more: diff --git a/roadmaps/data-engineer/content/what-is-data-warehouse@dc3lJI27hJ3zZ45UCVqM1.md b/roadmaps/data-engineer/content/what-is-data-warehouse@dc3lJI27hJ3zZ45UCVqM1.md index 79d655f6d..3548ef327 100644 --- a/roadmaps/data-engineer/content/what-is-data-warehouse@dc3lJI27hJ3zZ45UCVqM1.md +++ b/roadmaps/data-engineer/content/what-is-data-warehouse@dc3lJI27hJ3zZ45UCVqM1.md @@ -1,5 +1,5 @@ # What is Data Warehouse? - + A data warehouse is a centralized repository for storing large volumes of structured, historical data from multiple sources. It is optimized for analytical queries rather than transactional operations. Data warehouses power business intelligence, reporting, and data analysis, providing a single source of truth across an organization. Visit the following resources to learn more: