We are seeking a Director, Data Engineering – AI & Data Platforms to lead the strategy, architecture, development, and evolution of our data and AI infrastructure. This is a hands-on leadership role responsible for building a scalable, reliable, and AI-ready data platform that powers analytics, machine learning, automation, and emerging generative AI applications.
The Director will lead the design and implementation of modern data architecture while partnering closely with engineering, analytics, product, and business stakeholders. There will be a a focus on AI initiatives, including developing the data foundations required for machine learning and generative AI, identifying opportunities for AI-driven automation, and helping translate emerging AI capabilities into practical business applications.
This role is ideal for a technical leader who enjoys operating at both the strategic AND hands-on levels and is comfortable building systems, establishing engineering standards, mentoring engineers, and driving cross-functional initiatives in a fast-moving digital media environment.
This position will be on-site in our New York, NY or Los Angeles, CA office.
What You'll Do
Own the strategy, architecture, and roadmap for the company's data engineering and analytics platform.
Design and oversee scalable, secure, and cost-effective data architectures and pipelines supporting analytics, reporting, machine learning, and AI applications.
Establish engineering standards for data modeling, pipeline development, testing, deployment, observability, documentation, and operational excellence.
Lead the development and optimization of batch and near-real-time data pipelines using SQL and Python.
Oversee data modeling and warehouse architecture in Snowflake, ensuring scalability, performance, reliability, and efficient use of resources.
Drive the evolution of our cloud-based data infrastructure using AWS, including S3, EC2, Lambda, and related services.
Establish robust frameworks for data quality, testing, monitoring, lineage, observability, and alerting.
Evaluate and introduce technologies that improve the scalability, reliability, and efficiency of the data platform.
Balance hands-on technical contribution with architectural oversight and engineering leadership.
AI & Machine LearningLead the data engineering strategy supporting machine learning, generative AI, and AI-powered applications.
Partner with data scientists, engineers, analysts, and business leaders to identify and prioritize high-value AI opportunities.
Design and oversee data pipelines supporting model training, feature engineering, inference, evaluation, and monitoring.
Develop the data foundations required for LLM and generative AI applications, including data preparation, embeddings, vector data, retrieval pipelines, and RAG architectures where appropriate.
Establish processes for AI data quality, model evaluation, experimentation, and performance monitoring.
Identify opportunities to use AI to improve internal workflows, analytics, data operations, content-related processes, and engineering productivity.
Evaluate emerging AI technologies and determine where they can provide practical business value.
Establish responsible and scalable approaches to incorporating AI into the company's data and technology ecosystem.
Partner with leadership to develop an AI roadmap aligned with business priorities and measurable outcomes.
Engineering Leadership & Team DevelopmentProvide technical leadership and mentorship to data engineers and other technical contributors.
Establish engineering best practices for code quality, version control, CI/CD, testing, documentation, security, and operational reliability.
Define technical objectives, priorities, and development standards for the data engineering function.
Participate in hiring, onboarding, coaching, performance development, and career growth for data engineering team members.
Build a culture of technical ownership, experimentation, continuous improvement, and knowledge sharing.
Determine when to build, buy, or integrate third-party technologies and services.
Promote reusable frameworks, tooling, and engineering practices that improve team productivity.
Analytics & Business EnablementPartner with analysts and business stakeholders to ensure the data platform supports reliable and accessible business intelligence and analytics.
Improve data accessibility, discoverability, documentation, and usability across the organization.
Work with stakeholders to translate business requirements into scalable technical solutions.
Support data experimentation and statistical analysis by ensuring analysts and data scientists have high-quality, appropriately structured datasets.
Help establish data definitions, governance practices, and standards that improve trust in company data.
Communicate complex technical concepts and architectural decisions clearly to both technical and non-technical audiences.
Cloud & Platform OperationsOwn the reliability, scalability, and cost management of the organization's cloud-based data infrastructure.
Oversee deployment and management of data applications and services using AWS.
Establish appropriate practices for infrastructure automation, CI/CD, security, access controls, and operational monitoring.
Identify opportunities to optimize cloud costs and platform performance.
Develop disaster recovery, resiliency, and operational processes appropriate for the company's data infrastructure.
Work closely with engineering and technology leadership on broader cloud infrastructure initiatives.
Cross-Functional LeadershipServe as a strategic technical partner to executive leadership, product, engineering, analytics, and business teams.
Lead cross-functional initiatives involving data, AI, analytics, automation, and technology modernization.
Translate technical capabilities and limitations into clear business implications and recommendations.
Establish priorities across competing data and AI initiatives based on business value, technical feasibility, and available resources.
Represent the data engineering function in broader technology and organizational planning.
Who You Are
- 8+ years of experience in data engineering, software engineering, analytics engineering, or a related technical discipline.
- 3+ years of experience leading data engineering teams, technical initiatives, or data platform architecture.
- 5+ years of experience with SQL, including complex data exploration, optimization, and relational/data warehouse modeling.
- 5+ years of experience with Python for data engineering, automation, application development, or machine learning workflows.
- 3+ years of experience with Snowflake, including data modeling, architecture, performance optimization, and large-scale data processing.
- 3+ years of experience with AWS, including services such as S3, EC2, Lambda, and related cloud technologies.
- 3+ years of experience with Apache Airflow or comparable orchestration technologies such as Dagster or Prefect.
- Demonstrated experience designing and implementing scalable data platforms and production-grade data pipelines.
- Strong experience with data quality, testing, observability, monitoring, and operational reliability.
- Demonstrated experience supporting machine learning and/or AI initiatives through data engineering, feature engineering, model data pipelines, or AI infrastructure.
- Strong understanding of machine learning concepts, statistical analysis, experimentation, and data science workflows.
- Working knowledge of generative AI, LLMs, embeddings, vector search, RAG, and AI application architectures.
- Strong analytical and critical-thinking skills with the ability to solve complex technical and business problems.
- Demonstrated ability to communicate technical concepts clearly to technical and non-technical audiences.
- Experience working effectively with executives, stakeholders, engineers, analysts, and data scientists.
- Must be willing to work in our NY or LA office.
- Experience leading AI transformation or AI platform initiatives.
- Experience designing production systems for LLM or generative AI applications.
- Experience with vector databases, embeddings, RAG architectures, model evaluation, and AI observability.
- Experience implementing AI-powered automation within data or business workflows.
- Experience with infrastructure-as-code technologies such as Terraform.
- Experience with CI/CD, Docker, Kubernetes, or other modern DevOps practices.
- Experience with distributed data processing technologies such as Spark.
- Experience with data governance, metadata management, lineage, security, and privacy.
- Experience working in a digital media, advertising, publishing, entertainment, or consumer technology environment.
- Experience managing data platforms in a high-growth or resource-constrained organization.
- Experience evaluating third-party data, analytics, and AI platforms and negotiating build-versus-buy decisions.
Preferred Qualifications:
What We Offer
- Best in class health, dental, and vision insurance
- Healthcare FSA
- Dependent Care FSA
- Commuter Benefits FSA
- Short-term/long-term disability and life insurance
- Paid Parental leave
- 401k with 4% match
- Pet Insurance
- Legal and Identity Theft Plans
- Flexible PTO
Complex NTWRK New York, New York, USA Office
New York, New York, United States
Similar Jobs
What you need to know about the NYC Tech Scene
Key Facts About NYC Tech
- Number of Tech Workers: 549,200; 6% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Capgemini, Bloomberg, IBM, Spotify
- Key Industries: Artificial intelligence, Fintech
- Funding Landscape: $25.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Greycroft, Thrive Capital, Union Square Ventures, FirstMark Capital, Tiger Global Management, Tribeca Venture Partners, Insight Partners, Two Sigma Ventures
- Research Centers and Universities: Columbia University, New York University, Fordham University, CUNY, AI Now Institute, Flatiron Institute, C.N. Yang Institute for Theoretical Physics, NASA Space Radiation Laboratory
