(4) Analysis and Visualization (Data Warehouse and QuickInsight) — Amazon’s data services can be divided into five categories: data ingestion, storage, processing, analysis and visualization, and security. This article is part of a series, each dealing with each of the five topics above. 1. Data Ingestion 2. Data Storage 3. Data Processing 4. Analysis and Visualization Analysis: Data Warehouse - Redshift …

(3) Data Processing (AWS EMR and AWS ETL) — Amazon’s data services can be divided into five categories: data ingestion, storage, processing, analysis and visualization, and security. This article is part of a series, each dealing with each of the five topics above. 1. Data Ingestion 2. Data Storage 3. Data Processing - Glue - EMR - Lambda - AWS Data Pipeline - Sage…

(2) Data Storage (EBS, S3…) — Amazon’s data services can be divided into five categories: data ingestion, storage, processing, analysis and visualization, and security. This article is part of a series, each dealing with each of the five topics above. 1. Data Ingestion 2. Data Storage - Data Format - EBS, S3, DynamoDB, RDS - Architecture - Other services…

(1) Data Ingestion (AWS Kinesis and other related services) — Amazon’s data services can be divided into five categories: data ingestion, storage, processing, analysis and visualization, and security. This article is part of a series, each dealing with each of the five topics above. 1. Data Ingestion - Kinesis Tuning - Amazon Managed Streaming for Apache Kafka Security - Kinesis…

Hello 👋 I’m a Junior Data Engineer in Asia. In my opinion, many Asian companies are struggling to bring data architecture into their business. So, as a junior data engineer, I have some trouble figuring our how to develop my career. I tried to find out what kind of problem…

핵심 개념과 물리적인 구조, 드루이드와 다른 시스템(ES, Spark, Key-Value Storage)과의 차이점 — 목차 1. 드루이드 2. 장점 3. 주요 특징 4. 물리적 구조 5. 다른 빅데이터 솔루션과의 차이점 1. 드루이드 (Druid) 공식문서: https://druid.apache.org/ Github: https://github.com/apache/druid Druid is a high performance real-time analytics …

고민과 계획.. 앞으로 어떻게 살아야하지..? — 클라우드 덕후 삶의 시작 내가 클라우드 덕후가 된 이유 클라우드 조!아!soniacomp.medium.com 2020년 좋은 기회로, “클라우드 아키텍트” 직무로 인턴을 시작하게 됐고, 유연하고 다양한 솔루션에 반해서, “클라우드 아키텍트”를 목표로 삼게 되었다. 하고 싶었던 게 있었기 때문에, 이전 회사에서 프론트엔드를 시작으로 백엔드 서버 개발을 할 때에도, 틈틈이 클라우드 공부를 하면서, 자격증도 따고, 사내 클 …

with “Google Colaboratory” and “Word Count Example” — You can check complete code at the link below Google Colaboratory — WordCountExample colab.research.google.com code description 1. Spark Environment Setup - Install Java, Spark, and Findspark - Set Environment Variables - Start a SparkSession 2. Loading data into Spark - Create your own RDD - Import data from outside 3. …

// 데이터 엔지니어 병아리 탈출기 // 대표적인 데이터 프레임워크 (2) — 오늘도 공부.. 🥲 1. 스파크 분산시스템 아키텍처 2. spark 의 2가지 주요 개념 - RDD - DAG 3. 하둡 맵-리듀스 단점 & 스파크 장점 4. 스파크 API - RDD - Spark SQL 과 DataFrame 스파크 분산시스템 아키텍처 Spark Application 은 Spark Context obje …