Published in Dev Genius·Mar 8Celery executor vs Kubernetes executor on AirflowAs we mentioned in previous articles, Airflow is an open-source platform that allows us to create, schedule and monitor workflows. And it has a variety of executors like Celery executor and Kubernetes executor. In this article, I will summarize a little about each of the executors and their benefits as…Data Engineering3 min readData Engineering3 min read
Published in Dev Genius·Feb 26Celery Executor on AirflowContinuing our series of articles talking about Airflow, in this one, we will cover a little more about Celery executor, I hope you like it. let’s go The Celery executor allows Airflow to offload task execution to a Celery worker cluster, which can provide scalable and distributed processing of tasks…Airflow2 min readAirflow2 min read
Published in Dev Genius·Feb 24Let’s talk about FlinkIn this article, I want to present a little more about Flink, one of the tools that I have been increasingly passionate about working with. Let's go! Apache Flink is an open-source distributed computing system that is designed to process large-scale data sets and streams. …Data Engineering3 min readData Engineering3 min read
Published in Dev Genius·Feb 22Kubernetes Executor on AirflowIn this new series, we’re going to talk a little more about Airflow, its benefits, and use cases. In this first article, I will discuss Kubernetes Executor. Let’s go 🚀 Airflow is an open-source platform to programmatically author, schedule, and monitor workflows. It allows developers to define their workflows as…Data Engineering3 min readData Engineering3 min read
Published in Dev Genius·Dec 19, 2021Start using Airflow on Heroku(Part II)In the previous article, we configured the environment to run Airflowlocally in this article, we are going to configure your Herokuand make the first deployment of our application. For this article, we will deal with some topics: Configuring fernet key Making the local environment more similar to production Setting up…Data Engineer3 min readData Engineer3 min read
Published in Dev Genius·Dec 14, 2021Create a simple data app with StreamlitHave you ever thought about creating a data app in less than 15 minutes? With the Steamlit lib, it’s possible! In this article, I will teach you how to create a data visualization app using financial market data quickly and with only 9 lines of code! 🚀 What do we…Data Science2 min readData Science2 min read
Published in Dev Genius·Dec 6, 2021Start using Airflow on Heroku (Part I)Whenever we start to develop something new in the data engineering area, we want to place our project in an environment that reproduces the production environment, thus being able to understand all the thresholds of our architecture. So I’m going to cover in a series of three articles how to…Airflow3 min readAirflow3 min read
Mar 28, 2021The first week in CTO positionI believe that the dream of 70% of software engineers, developers and architects is to take a leadership one. Maybe even each architects is to take a leadership role. Maybe even reach the position of CTO or even build your Startup from scratch to take on that role from there…Inspiration2 min readInspiration2 min read
Mar 25, 2021Como medir a produtividade em um time de desenvolvimento de softwareTudo bem com vocês? Venho trazer nesse artigo algumas experiências ao longo da minha carreira, assim como a ciência, esse simples artigo é de todo refutável. A vantagem competitiva de uma empresa depende da qualidade do seu produto e da velocidade com que você pode entregá-lo, trazendo assim um dos…Desenvolvimento Software3 min readDesenvolvimento Software3 min read
