Data Engineer (Gen AI)

This is Hybrid role (3 days in office /2 days remote)

About your Team:

The Enterprise Architecture organization is looking for a Data Engineer to join the team and help build the next generation of data infrastructure and AI-enabled workflows. In this role, you will be responsible for designing, building, and maintaining scalable data pipelines, data lake platforms, and analytics solutions that support enterprise-wide AI initiatives and advanced analytics capabilities. You will partner closely with internal development teams and IT leadership to architect data solutions that meet diverse use-cases across the organization. This role offers the opportunity to work with cutting-edge data technologies, AI knowledge bases, and cloud-native data platforms as we scale our data operations. Your work will focus on delivering robust, well-documented data solutions and establishing best practices for data engineering across the enterprise.

 What will be your responsibilities within IBKR: 

· Design, build, and maintain scalable data crawlers and ETL/ELT pipelines to ingest data from various sources including enterprise collaboration tools, web applications, and internal databases

· Develop and manage data lake platform infrastructure including S3-based storage, Iceberg tables, AWS Glue data catalog, and cloud data warehouse solutions for analytics workloads

· Build and optimize real-time streaming data pipelines using Kafka for event-driven analytics and data processing

· Create and maintain data transformation pipelines to clean, curate, and prepare data for analytics and AI/ML applications, supporting multiple data formats including structured, semi-structured, and unstructured data (text, images, audio, video)

· Develop Python-based applications for data ingestion, processing, and integration supporting Gen AI RAG workflows and knowledge base systems

· Collaborate with internal development teams, data scientists, and stakeholders to understand requirements, architect appropriate data solutions, and create comprehensive technical documentation

· Monitor, troubleshoot, and optimize data pipelines for performance, reliability, and data quality

· Write clean, maintainable, well-tested code following software engineering and data engineering best practices

· Create comprehensive technical documentation for data pipelines, architectures, and platform capabilities

 What required skill’s you need:

· 6+ years of hands-on data engineering experience with modern data stack technologies

· Strong experience with AWS cloud services, particularly: S3, AWS Glue, Athena, EMR, Lambda

· Proficiency in Python for data processing, ETL, and application development

· Experience with PySpark on EMR for large-scale data processing

· Strong SQL skills for data analysis and transformation

· Experience building and maintaining ETL/ELT pipelines at scale

· Experience with Kafka for streaming data pipelines and real-time data processing

· Knowledge of data lake architectures and modern table formats (e.g., Iceberg)

· Experience with CI/CD practices using Git, version control systems, and containerization (Docker)

· Understanding of data modeling, data warehousing concepts, and analytics best practices

· Exceptional problem-solving and analytical skills

· Excellent collaboration and communication (verbal and written) skills

· Self-motivated with ability to work independently and manage multiple priorities

· Willingness and enthusiasm to learn AI/ML technologies, stay current with emerging data engineering trends

 To be successful in this position, you will have the following:

• Self-motivated and able to handle tasks with minimal supervision.
• Superb analytical and problem-solving skills.
• Excellent collaboration and communication (Verbal and written) skills.
• Outstanding organizational and time management skills.