PinnedGetting started with Pyspark (rdd, spark SQL)— A 10 minute tutorialwith “Google Colaboratory” and “Word Count Example” — You can check complete code at the link below Google Colaboratory — WordCountExample colab.research.google.com code description 1. Spark Environment Setup - Install Java, Spark, and Findspark - Set Environment Variables - Start a SparkSession 2. Loading data into Spark - Create your own RDD - Import data from outside 3. …Pyspark3 min readPyspark3 min read
Jun 11개춘기… 개발자 사춘기 극복 일지 🥸개발자 4년차에 접어들며… — 프론트 엔지니어로 시작해, 풀스택 엔지니어를 거쳐 데이터 엔지니어로 먹고 산지 벌써 3년을 꽉 채우게 되었다. 그동안 연봉도 높아지고, 이제 더 이상 귀여움 받으며 배우고 성장하는데 급급한 주니어가 아니게 되었다. 나도 이제 회사에서 밥값을 하기 위해서는 시스템의 개발과 개선을 주도해야하는 연차가 되었고, 내가 …3 min read3 min read
Oct 17, 2022장고 ORM 과 최적화 기법ORM, 장고 ORM, N+1 문제, Eager Loading — ORM 데이터베이스 시스템을 직접 다루지 않고도 데이터베이스를 활용할 수 있도록 하는 편리하고 강력한 인터페이스 간단한 것을 쉽게, 어려운 것을 가능하게 해줌 => ORM은 객체와 관계형 데이터베이스의 데이터를 매핑해주는 것 => 데이터베이스와 객체지향 프로그래밍 언어간의 호환되지 않는 데이터를 변환하는 프로그래밍 기법 객체와 관계형 데이터베이스의 데이터를 자동으로 매핑해주는 …3 min read3 min read
Jun 26, 2022AWS Certified Data Analytics(DAS-C01) — Certification Summary(5) Security — Amazon’s data services can be divided into five categories: data ingestion, storage, processing, analysis and visualization, and security. This article is part of a series, each dealing with each of the five topics above. 1. Data Ingestion 2. Data Storage 3. Data Processing 4. Analysis and Visualization 5. Security Security Cognito Using…AWS6 min readAWS6 min read
Jun 26, 2022AWS Certified Data Analytics(DAS-C01) — Certification Summary(4) Analysis and Visualization (Data Warehouse and QuickInsight) — Amazon’s data services can be divided into five categories: data ingestion, storage, processing, analysis and visualization, and security. This article is part of a series, each dealing with each of the five topics above. 1. Data Ingestion 2. Data Storage 3. Data Processing 4. Analysis and Visualization Data Lake - Lake Formation Analysis…AWS10 min readAWS10 min read
Jun 25, 2022AWS Certified Data Analytics(DAS-C01) — Certification Summary(3) Data Processing (AWS EMR and AWS ETL) — Amazon’s data services can be divided into five categories: data ingestion, storage, processing, analysis and visualization, and security. This article is part of a series, each dealing with each of the five topics above. 1. Data Ingestion 2. Data Storage 3. Data Processing - Glue - EMR - Lambda - AWS Data Pipeline - Sage…AWS8 min readAWS8 min read
Jun 22, 2022AWS Certified Data Analytics(DAS-C01) — Certification Summary(2) Data Storage (EBS, S3…) — Amazon’s data services can be divided into five categories: data ingestion, storage, processing, analysis and visualization, and security. This article is part of a series, each dealing with each of the five topics above. 1. Data Ingestion 2. Data Storage - Data Format - EBS, S3, DynamoDB, RDS - Architecture - Other services…AWS6 min readAWS6 min read
Jun 20, 2022AWS Certified Data Analytics(DAS-C01) — Certification Summary(1) Data Ingestion (AWS Kinesis and other related services) — Amazon’s data services can be divided into five categories: data ingestion, storage, processing, analysis and visualization, and security. This article is part of a series, each dealing with each of the five topics above. 1. Data Ingestion - Kinesis Tuning - Amazon Managed Streaming for Apache Kafka Security - Kinesis…AWS6 min readAWS6 min read
May 5, 2022AWS, Google— Data Engineer Interview QuestionsHello 👋 I’m a Junior Data Engineer in Asia. In my opinion, many Asian companies are struggling to bring data architecture into their business. So, as a junior data engineer, I have some trouble figuring our how to develop my career. I tried to find out what kind of problem…Interview Questions4 min readInterview Questions4 min read
Apr 10, 2022드루이드 (Druid)핵심 개념과 물리적인 구조, 드루이드와 다른 시스템(ES, Spark, Key-Value Storage)과의 차이점 — 목차 1. 드루이드 2. 장점 3. 주요 특징 4. 물리적 구조 5. 다른 빅데이터 솔루션과의 차이점 1. 드루이드 (Druid) 공식문서: https://druid.apache.org/ Github: https://github.com/apache/druid Druid is a high performance real-time analytics …Apache Druid9 min readApache Druid9 min read
