Leads the design, development, deployment, and support of complex data pipelines and data products across cloud, warehouse, and lakehouse platforms. Builds scalable batch and streaming solutions using Snowflake, Databricks, SQL, Python, dbt, Spark, and related technologies. Ensures data quality, security, observability, reliability, and cost efficiency while troubleshooting production issues. Provides technical leadership, mentors engineers, establishes reusable standards, and collaborates with architecture, governance, cybersecurity, product, and platform teams.
Job Description Overview:
Primary Responsibilities:
Education and Experience Required:
Education and Experience Preferred:
- Designs, develops, implements and operates highly complex data pipelines, data products and data management solutions across modern cloud, warehouse and lakehouse platforms. Applies strong software-engineering, data-modeling and distributed-processing practices to deliver solutions that are scalable, secure, resilient, observable and cost efficient. Independently resolves ambiguous engineering problems, contributes to technical direction and reusable standards, and mentors other engineers while partnering across product, architecture, governance, cybersecurity and platform teams.
Primary Responsibilities:
- Lead the detailed engineering design, development, testing, deployment and production support of complex data pipelines and reusable data products.
- Develop maintainable data-processing and transformation solutions using advanced SQL and programming languages such as Python.
- Engineer solutions using Snowflake, Databricks, dbt and comparable modern data platforms; select appropriate warehouse, lakehouse or hybrid patterns based on workload needs.
- Design analytical storage solutions using relational databases, cloud object storage, columnar formats and open table formats such as Apache Iceberg.
- Implement batch, micro-batch, change data capture (CDC), event-driven and streaming patterns using technologies such as Apache Kafka or comparable platforms.
- Integrate data from relational and mainframe systems, APIs, event streams, SaaS platforms, files and object storage with appropriate security and operational controls.
- Apply normalized, dimensional, denormalized and data-product-oriented modeling approaches based on access, performance and lifecycle requirements.
- Optimize Snowflake, Databricks/Spark and SQL workloads for performance, scalability, reliability and cost, including partitioning, parallelism, data movement and compute utilization.
- Build data quality, reconciliation, metadata, lineage, observability, alerting and service-level controls into production solutions.
- Design for restartability, idempotency, retry, replay, recovery and failure isolation; lead diagnosis and root-cause analysis for complex production incidents.
- Apply source control, peer review, automated testing, CI/CD, environment promotion and infrastructure/configuration automation to data engineering delivery.
- Create reusable libraries, templates, implementation patterns and reference solutions that improve consistency and engineering velocity.
- Evaluate design alternatives and clearly document tradeoffs involving security, privacy, resiliency, maintainability, performance, cost and operational complexity.
- Contribute to technical standards, identify technical debt and recommend pragmatic modernization or remediation approaches.
- Mentor engineers through design reviews, code reviews, pairing and technical problem solving; influence engineering practices across the team and adjacent teams.
- Partner with product owners, architects, governance, cybersecurity, platform and consuming teams to translate business and data requirements into scalable solutions.
- Develop technical epics and stories, estimate complex work, surface dependencies and risks, and support iterative delivery through Agile practices.
- Provide technical leadership and mentorship without direct people-management responsibility.
- Understand and adhere to the Company’s risk and regulatory standards, policies and controls in accordance with the Company’s Risk Appetite.
- Design, implement, maintain and enhance internal controls to mitigate risk on an ongoing basis. Identify risk-related issues requiring escalation to management.
- Maintain internal control standards, including timely implementation of internal and external audit points and issues raised by external regulators, as applicable.
- Promote an environment that supports belonging and reflects the Company brand.
- Complete other related duties as assigned.
Education and Experience Required:
- Associate’s degree and a minimum of 7 years’ systems analysis and/or application development work experience or Bachelor's degree and a minimum of 5 years' systems analysis and/or application development work experience. In lieu of a degree, a combined minimum of 9 year’s education and/or relevant work experience, including a minimum of 5 years’ system analysis and/or application development work experience.
- Expert proficiency in a minimum of 1 relevant programming language and professional proficiency in a minimum of 1 additional relevant programming language.
- Advanced SQL skills, including complex transformations, analytical queries and workload optimization.
- Experience with Python or another general-purpose programming language used for production data engineering.
- Hands-on engineering experience with Snowflake and/or Databricks.
- Experience with modern ELT/ETL and transformation practices, including dbt or a comparable framework.
- Experience with data modeling for warehouse, lakehouse and data-product use cases.
- Experience with distributed data processing using Apache Spark/PySpark or a comparable framework.
- Experience with Git-based development, automated testing, code review and CI/CD.
- Experience with batch and streaming architectures, orchestration and production operations.
- Knowledge of cloud computing and object-storage concepts, preferably in Microsoft Azure.
- Experience working with structured and semi-structured data using formats such as Parquet, JSON, Avro, CSV and XML.
- Demonstrated ability to independently design and deliver complex data solutions spanning ingestion, processing, persistence, quality and consumption.
- Demonstrated ability to troubleshoot complex distributed-processing, data-quality and production reliability issues.
- Demonstrated ability to balance performance, cost, scalability, resiliency, security and maintainability in technical decisions.
- Demonstrated ability to review and improve technical designs and code developed by other engineers.
- Demonstrated ability to communicate complex designs and tradeoffs to engineering, architecture, product and business audiences.
Education and Experience Preferred:
- Advanced production experience with Snowflake, Databricks, Apache Spark/PySpark and dbt.
- Experience implementing Apache Iceberg or another open table format, including schema evolution, partitioning, snapshots and table maintenance.
- Experience with Apache Kafka, CDC and event-driven architectures.
- Experience with orchestration technologies such as Apache Airflow, Azure Data Factory, Databricks Workflows or dbt orchestration.
- Experience with Infrastructure as Code and configuration automation, including Terraform or comparable tooling.
- Experience with data catalogs, metadata management, lineage, data observability and automated data-quality frameworks.
- Experience building governed enterprise data products or domain-oriented data solutions.
- Experience working in a highly regulated environment such as financial services.
- Master’s degree in Computer Science, Engineering, Data Engineering, Information Systems or a related technical field.
M&T Bank New York, New York, USA Office
New York, United States, 0
M&T Bank Wayne, New Jersey, USA Office
Wayne, United States
Similar Jobs
Financial Services
Lead the design, development, and support of scalable enterprise data pipelines and platforms. Build Snowflake-based solutions, Python components, and Airflow workflows while ensuring data quality, governance, security, monitoring, performance, and reliability. Partner with business and technology stakeholders on architecture and modernization, troubleshoot production issues, conduct code reviews, and provide technical guidance to data engineering teams in an Agile enterprise environment.
Top Skills:
Apache AirflowSparkAWSAzure Data FactoryDatabricksGCPAzurePythonSnowflakeSQL
Aerospace • Security • Cybersecurity • Industrial
Leads the design, development, governance, and productization of enterprise data models, pipelines, data products, and ontologies. Builds machine-learning-ready datasets, implements security and lineage controls, optimizes Databricks and Delta Lake environments, establishes data-product standards, collaborates with architects and data scientists, and mentors engineering talent. The role also defines product roadmaps, service levels, usage metrics, and continuous-improvement processes for enterprise data assets supporting AI agents, analytics, and applications.
Top Skills:
Apache AirflowApache NifiSparkCloud IamDatabricksDelta LakeFeature StoresGenerative AiMachine Learning PipelinesPysparkPythonScalaSQLUnity Catalog
Software
Lead hands-on data engineering delivery across analytics and data platform initiatives. Responsibilities include designing Medallion architecture transformations, building Gold-layer data models, implementing complex PySpark and Databricks transformations, improving pipeline reliability, coordinating contractors, planning sprint execution, reviewing pull requests, and communicating risks and dependencies. The role also supports GDPR-first, multi-instance data strategies and external analytics consumption.
Top Skills:
SparkAWSAzureData WarehousingDatabricksGCPMedallion ArchitecturePysparkSpark Sql
What you need to know about the NYC Tech Scene
As the undisputed financial capital of the world, New York City is an epicenter of startup funding activity. The city has a thriving fintech scene and is a major player in verticals ranging from AI to biotech, cybersecurity and digital media. It also has universities like NYU, Columbia and Cornell Tech attracting students and researchers from across the globe, providing the ecosystem with a constant influx of world-class talent. And its East Coast location and three international airports make it a perfect spot for European companies establishing a foothold in the United States.
Key Facts About NYC Tech
- Number of Tech Workers: 549,200; 6% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Capgemini, Bloomberg, IBM, Spotify
- Key Industries: Artificial intelligence, Fintech
- Funding Landscape: $25.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Greycroft, Thrive Capital, Union Square Ventures, FirstMark Capital, Tiger Global Management, Tribeca Venture Partners, Insight Partners, Two Sigma Ventures
- Research Centers and Universities: Columbia University, New York University, Fordham University, CUNY, AI Now Institute, Flatiron Institute, C.N. Yang Institute for Theoretical Physics, NASA Space Radiation Laboratory
