Senior Data Engineer

Навыки

• Airflow
• Ansible
• BigQuery
• CI/CD
• Data Governance
• Data Lake
• Data Quality

О компании и продукте

• We are not satisfied until we are achieving industry-leading results in the market. We are looking for a Data Engineer to design, develop, and optimize our data infrastructure on Databricks

Задачи

• Data Architecture & Strategy – Define and implement the overall data architecture on GCP, including data warehousing in BigQuery/Databricks, data lake patterns in Google Cloud Storage, and Data Mart (Data Mach) solutions
• Integrate Terraform for Infrastructure as Code to provision and manage cloud resources efficiently
• Establish both batch and real-time data processing frameworks to ensure reliability, scalability, and cost efficiency
• Pipeline Development & Orchestration – Design, build, and optimize ETL/ELT pipelines using Apache Airflow for workflow orchestration
• Implement dbt (Data Build Tool) transformations to maintain version-controlled data models in BigQuery, ensuring consistency and reliability across the data pipeline
• Use Google Dataflow (based on Apache Beam) and Pub/Sub for large-scale streaming/batch data processing and ingestion
• Automate job scheduling and data transformations to deliver timely insights for analytics, machine learning, and reporting
• Event-Driven & Microservices Architecture – Implement event-driven or asynchronous data workflows between microservices
• Employ Docker and Kubernetes (K8s) for containerization and orchestration, enabling flexible and efficient microservices-based data workflows
• Implement CI/CD pipelines for streamlined development, testing, and deployment of data engineering components
• Data Quality, Governance & Security – Enforce data quality standards using Great Expectations or similar frameworks, defining and validating expectations for critical datasets
• Define and uphold metadata management, data lineage, and auditing standards to ensure trustworthy datasets
• Implement security best practices, including encryption at rest and in transit, Identity and Access Management (IAM), and compliance with GDPR or CCPA where applicable
• Scientists & Analytics Enablement – Collaborate with Data Science, Analytics, and Product teams to ensure the data infrastructure supports advanced analytics, including machine learning initiatives
• Maintain Data Mart (Data Mach) environments that cater to specific business domains, optimizing access and performance for key stakeholders
• Senior Data Engineer
• Development
• Fireart is an experience design & technology partner to clients large and small
• We believe that great ideas are only as good as how well they perform
• Our passion is for pushing the boundaries of our ideas within the limits of what’s possible
• You will architect scalable pipelines using BigQuery, Google Cloud Storage, Apache Airflow, dbt, Dataflow, and Pub/Sub, ensuring high availability and performance across our ETL/ELT processes
• You will leverage Great Expectations to enforce data quality standards
• The role also involves building our Data Mart (Data Mach) environment and implementing CI/CD best practices

Требования

• Experience – 3+ years of professional experience in data engineering, with at least 1 year in mobile data
• Programming & Containerization – Strong coding capabilities in Python, Java, or Scala, plus scripting for automation. – Experience with Docker and Kubernetes (K8s) for containerizing data-related services. – Hands-on with CI/CD pipelines and DevOps tools (e.g., Terraform, Ansible, Jenkins, GitLab CI) to manage infrastructure and deployments
• Data Quality & Governance – Proficiency in Great Expectations (or similar) to define and enforce data quality standards
• Expertise in designing systems for data lineage, metadata management, and compliance (GDPR, CCPA). – Strong understanding of OLTP (Online Transaction Processing) and OLAP (Online Analytical Processing) systems
• Communication – Excellent communication skills for both technical and non-technical audiences. – High level of organization, self-motivation, and problem-solving aptitude
• A successful candidate has extensive knowledge of cloud-native data solutions, strong proficiency with ETL/ELT frameworks (including dbt), and a passion for building robust, cost-effective pipelines

Условия

• Competitive compensation based on your experience and expertise
• Long-term contract-based collaboration
• A collaborative and open team environment
• Access to learning resources
• Flexible collaboration arrangements

Паспорт вакансии

История публикации

Откуда что взялось

Отмечено то, что вывели мы. Без пометки — значение назвал работодатель.

Почему на этом месте в выдаче

Порядок выдачи объявлен контрактом: свежесть решает между днями, полнота и зарплата — внутри дня.