A Bit everyday A Byte every week
Designated parameter changes at the desk. All other variables, not our department.
The requesting team wants to tweak a parameter whenever they need to, without going through an engineer every time.
Giving them a database table to edit is a maintenance burden, and Airflow Variables can’t be opened up one at a time.
I don’t want to grant overly broad permissions, and I don’t want to be on call to change a value either. So what now?
As long as the file shows up, I'll do anything
Waiting on an externally, asynchronously generated file before continuing is a common scenario in Airflow.
But “I’ll wait as long as it takes” is a heavy promise to make on your infrastructure’s behalf.
Unlocking parallel execution in Airflow with Dynamic Task Mapping
When tasks depend on each other yet remain independent events, what is the most elegant way to handle them?
Should we use a loop? Add another DAG? Or is there a better solution?
Use operator to easily trigger downstream DAGs in Airflow
When building dependencies between DAGs, you don’t always have to rely on RESTful APIs.
Airflow also provides a built-in Operator for this purpose.
End the mess of switching Python versions.
Pyenv is a tool for managing different Python versions.
Whether you’re upgrading a project to a newer Python release or switching between projects with different requirements, pyenv makes the process simple and hassle-free.
An introduction to two approaches for testing class instantiation when using the Factory Pattern, with practical examples using pytest-mock.
Introducing the factory pattern in your code can significantly improve its structure—but it also makes unit testing a bit trickier.
We’ll walk through two different mocking strategies, compare their pros and cons, and help you write more robust tests!
Free your unit tests from database dependencies using mocks.
With mock objects, we can simulate database operations without actually connecting to a real database.
This means we don’t need to prepare a dedicated test database or worry about cleaning it up after tests.
