Data Engineering Manager – (GCP, AI/ML & GenAI)

Position Overview

We are looking for an experienced Engineering Manager with strong hands-on expertise in AWS and GCP Data Engineering to lead a large-scale AWS-to-GCP data platform migration.

The ideal candidate will have strong experience designing enterprise data platforms on AWS and migrating them to Google Cloud Platform (GCP). The role requires a combination of technical architecture, hands-on engineering, migration leadership, team management, stakeholder management, and delivery ownership.

In addition to AWS-to-GCP data migration, this role will lead the design and delivery of AI/ML, Generative AI, and MLOps capabilities on GCP. The candidate will work with Data Science, Product, Business, Analytics, BI, and Engineering teams to build secure, scalable, governed, and production-ready data and AI solutions using GCP-native services.

All target-state data engineering, AI/ML, Generative AI, and MLOps architecture and implementation experience must be on GCP.

Key Responsibilities

1. AWS to GCP Migration Leadership

• Lead the end-to-end migration of enterprise data platforms from AWS to GCP.
• Assess existing AWS architecture, data pipelines, workloads, dependencies, data models, and operational processes.
• Define target-state GCP architecture, migration roadmap, technical dependencies, risks, rollback strategies, and delivery milestones.
• Develop migration strategies for:
• Amazon S3 to Google Cloud Storage
• Amazon Redshift to BigQuery
• AWS Glue to Dataflow, Dataproc, or BigQuery
• AWS Step Functions to Cloud Composer or Workflows
• AWS DMS to GCP-native CDC solutions
• Amazon Athena to BigQuery
• Identify opportunities to modernize AWS workloads rather than performing a simple lift-and-shift migration.
• Lead architecture reviews, technical design discussions, migration planning, and implementation governance.

2. GCP Data Platform Architecture

• Architect and implement scalable enterprise data platforms on GCP.
• Design Data Lake and Lakehouse architectures using Google Cloud Storage and BigQuery.
• Define Bronze, Silver, and Gold/Atomic data layers.
• Design scalable batch, real-time, and event-driven ingestion, transformation, and data-consumption frameworks.
• Establish standards for data modeling, partitioning, clustering, storage, metadata, lineage, and data access.
• Design multi-tenant and multi-location data architectures.
• Define schema-on-read and schema-on-write strategies.
• Build scalable data platforms that support analytics, BI, real-time reporting, machine learning, and Generative AI use cases.

3. AWS Data Platform Expertise

• Analyze and optimize existing AWS data platforms before migration.
• Work with Amazon S3, AWS Glue, AWS Glue Data Quality, Amazon Redshift, Redshift Serverless, Amazon Athena, AWS Step Functions, AWS DMS, and AWS Lake Formation.
• Understand existing AWS ETL/ELT pipelines, data models, workloads, security controls, and platform dependencies.
• Identify equivalent or improved GCP services for AWS data workloads.
• Prepare technical mapping, modernization recommendations, migration plans, and implementation roadmaps between AWS and GCP services.

4. GCP Streaming & Real-Time Data Engineering

• Architect real-time data pipelines using Google Pub/Sub, Dataflow, Apache Beam, BigQuery, and Cloud Storage.
• Design high-volume event ingestion, enrichment, transformation, and delivery pipelines.
• Implement event-driven architectures with appropriate delivery guarantees.
• Optimize streaming pipelines for latency, throughput, scalability, reliability, and cost.
• Design BigQuery streaming-ingestion patterns.
• Implement monitoring, logging, alerting, and operational support processes for real-time workloads.

5. ETL/ELT & Data Processing

• Design and implement scalable batch and real-time ETL/ELT pipelines on GCP.
• Migrate AWS Glue-based pipelines to appropriate GCP-native services.
• Develop transformation frameworks using Python, PySpark, SQL, Dataflow, Apache Beam, BigQuery, and dbt.
• Design CDC pipelines and real-time ingestion patterns.
• Build orchestration workflows using Cloud Composer and Apache Airflow.
• Optimize data-processing jobs, Spark workloads, pipeline execution, and query performance.
• Establish coding, testing, documentation, deployment, and operational standards for data engineering workloads.

6. Data Modeling & BigQuery

• Design enterprise data models for analytics, reporting, operational intelligence, and AI/ML workloads.
• Define dimensional, normalized, denormalized, and multi-tenant data models.
• Design BigQuery partitioning, clustering, storage, and query-optimization strategies.
• Optimize BigQuery SQL and query execution for performance and cost.
• Design data models that support real-time and batch workloads.
• Work closely with BI and Analytics teams to build scalable self-service consumption models.

7. AI/ML, Generative AI & MLOps

• Design and implement AI/ML and Generative AI solutions on GCP using Vertex AI, BigQuery, Cloud Storage, Dataflow, Pub/Sub, Cloud Run, and related GCP-native services.
• Build production-grade machine learning pipelines for data preparation, model training, validation, evaluation, deployment, monitoring, retraining, and lifecycle management.
• Develop Generative AI and Retrieval-Augmented Generation (RAG) solutions, including enterprise search, document intelligence, AI assistants, summarization, semantic search, embeddings, vector search, and knowledge-management applications.
• Design scalable ingestion, transformation, chunking, embedding, indexing, and retrieval pipelines for structured and unstructured enterprise data.
• Implement MLOps practices using Vertex AI Pipelines, Model Registry, model endpoints, Terraform, GitHub, Cloud Build, and CI/CD pipelines.
• Establish standards for model versioning, experiment tracking, data and feature validation, automated testing, deployment approvals, rollback, and environment promotion.
• Implement monitoring for model performance, data drift, latency, reliability, inference cost, response quality, retrieval accuracy, and GenAI risks such as hallucination and prompt injection.
• Ensure responsible AI, data privacy, security, governance, access control, auditability, and human-review processes are incorporated into AI/ML and GenAI solutions.
• Partner with Data Science, Analytics, Product, BI, Security, and US-based stakeholders to identify, prioritize, and deliver high-value AI/ML and GenAI use cases.

8. Data Governance, Security & Quality

• Establish data governance, data-quality, metadata, lineage, and ownership standards across the GCP data platform.
• Implement automated data-quality checks, validation frameworks, reconciliation processes, and monitoring.
• Establish governance and quality standards for AI/ML datasets, features, models, prompts, embeddings, vector stores, and Generative AI applications.
• Ensure appropriate security controls across all GCP data layers, including IAM, least-privilege access, encryption, service accounts, network security, secrets management, and data access policies.
• Ensure secure handling of confidential, sensitive, regulated, and personally identifiable information used in data, AI/ML, and GenAI workloads.
• Partner with governance, security, legal, and compliance teams to meet enterprise and regulatory requirements.
• Experience with Dataplex, Data Catalog, data lineage, and responsible AI governance is preferred.

9. DevOps, Infrastructure & Automation

• Lead infrastructure automation using Terraform.
• Build repeatable, scalable, secure, and compliant GCP infrastructure deployments.
• Implement CI/CD pipelines for data engineering, AI/ML models, Vertex AI pipelines, Generative AI applications, and infrastructure deployments.
• Work with Terraform, Git, GitHub, Cloud Build, Artifact Registry, and CI/CD pipelines.
• Automate data-pipeline deployment, testing, validation, security scanning, approvals, and rollback mechanisms.
• Establish Development, QA, UAT, and Production deployment standards for GCP data, AI/ML, and GenAI workloads.

10. Performance & Cost Optimization

• Lead performance-optimization initiatives across GCP data workloads.
• Optimize BigQuery query performance, partitioning, clustering, Dataflow pipelines, Spark workloads, Cloud Storage, and streaming workloads.
• Analyze AWS workloads and define the most cost-effective target-state GCP architecture.
• Develop cloud FinOps, capacity-planning, budget-monitoring, and cost-optimization strategies.
• Optimize AI/ML and Generative AI workloads for training cost, inference cost, latency, throughput, model selection, storage, and compute utilization.
• Establish performance benchmarks, SLAs, SLOs, and cost controls for critical data and AI services.

11. Engineering Management & Team Leadership

• Lead and mentor a team of Data Engineers, Senior Data Engineers, ML Engineers, Technical Leads, and other engineering resources.
• Provide technical direction and establish engineering standards.
• Conduct architecture reviews, code reviews, design reviews, and technical planning sessions.
• Define technical roadmaps, engineering priorities, delivery plans, and modernization strategies.
• Break complex migration and AI/ML requirements into actionable deliverables.
• Track engineering progress, risks, dependencies, delivery milestones, and quality metrics.
• Promote best practices around coding, testing, CI/CD, data quality, security, governance, documentation, and operational excellence.
• Mentor engineers on GCP, data architecture, modern data engineering, AI/ML, Generative AI, and MLOps practices.

12. Stakeholder & Client Management

• Act as the primary technical point of contact for US-based stakeholders.
• Work closely with Business, Product, Data Science, BI, Analytics, DevOps, Security, and Architecture teams.
• Translate business requirements into scalable data, cloud, AI/ML, and Generative AI solutions.
• Present architecture decisions, migration strategies, technical roadmaps, implementation plans, risks, timelines, and trade-offs.
• Communicate technical dependencies, delivery status, operational risks, and mitigation plans clearly to technical and non-technical stakeholders.
• Collaborate with business teams to define operational, analytical, data-quality, platform, and AI/ML KPIs.

Required Qualifications

• 15+ years of experience in Data Engineering, Data Architecture, Cloud Engineering, AI/ML Engineering, or related technology leadership roles.
• Strong hands-on experience with AWS Data Engineering and Data Architecture.
• 5+ years of strong hands-on GCP Data Engineering experience.
• Proven experience delivering AWS-to-GCP migration projects.
• Strong experience designing enterprise Data Lake and Lakehouse platforms on GCP.
• Strong hands-on experience with BigQuery, Google Cloud Storage, Dataflow, Pub/Sub, Cloud Composer, Dataproc, IAM, and Terraform.
• Experience migrating AWS data workloads, pipelines, and platforms to GCP.
• Strong knowledge of AWS and GCP service mapping, migration patterns, modernization strategies, and cloud architecture best practices.
• Experience designing, building, and deploying AI/ML solutions on GCP using Vertex AI.
• Hands-on experience with Generative AI, LLM-based applications, RAG architectures, embeddings, vector search, prompt engineering, and enterprise AI assistants.
• Strong understanding of MLOps, including model training, model registry, CI/CD/CT, model deployment, monitoring, retraining, governance, and rollback strategies.
• Experience implementing secure and responsible AI solutions, including data privacy, model evaluation, access controls, auditability, and governance.
• Expert-level SQL and strong Python and PySpark skills.
• Strong data modeling, data warehousing, batch processing, and real-time data engineering experience.
• Experience with Terraform, Git, GitHub, Cloud Build, CI/CD pipelines, and infrastructure automation.
• Experience managing and mentoring data engineering and cross-functional technical teams.
• Strong communication skills with experience working with US-based stakeholders.

Preferred Qualifications

• Google Cloud Professional Data Engineer certification.
• Google Cloud Professional Machine Learning Engineer certification.
• Experience with Vertex AI Agent Builder, Vertex AI Search, Gemini models on Vertex AI, or enterprise Generative AI platforms.
• Experience with dbt, Apache Airflow, Kafka, Apache Spark, Kubernetes, Cloud Run, and API-driven architectures.
• Experience with Dataplex, Data Catalog, data lineage, metadata management, data governance, master data management, and data-quality frameworks.
• Experience supporting enterprise or regulated environments with strong data privacy, security, compliance, audit, and governance requirements.