Design, develop, and optimize scalable batch and real-time data pipelines on GCP. Implement Medallion Architecture, build Python and PySpark transformations, optimize BigQuery and SQL workloads, deploy Dataflow pipelines, and orchestrate workflows with Cloud Composer and Airflow. Manage GCS storage, CI/CD, version control, IAM, data governance, security, reliability, scalability, and cost efficiency across enterprise data platforms.
This is a remote position.
KANINI is seeking a highly skilled Senior Data Engineer with deep expertise in Google Cloud Platform (GCP) and modern data architecture. The ideal candidate will have hands-on experience designing scalable data pipelines, implementing Medallion Architecture, and building robust enterprise-grade data solutions.
This role requires strong technical proficiency in BigQuery, PySpark, Dataflow, and Airflow, along with a solid understanding of cloud data governance, performance optimization, and CI/CD practices.
Key Responsibilities
- Design, develop, and maintain scalable batch and real-time data pipelines on GCP
- Implement and manage Medallion Architecture (Bronze, Silver, Gold layers) for data processing
- Build high-performance data transformations using Python and PySpark
- Develop and optimize complex SQL queries for analytical workloads
- Work extensively with BigQuery for large-scale data processing and performance tuning
- Develop and deploy pipelines using Cloud Dataflow
- Orchestrate workflows using Cloud Composer (Apache Airflow)
- Manage data storage and lifecycle using Google Cloud Storage (GCS)
- Implement version control and CI/CD pipelines using Git-based tools
- Ensure data security, governance, and access control using GCP IAM
- Optimize data solutions for performance, scalability, reliability, and cost-efficiency
Required Skills & Experience
- Strong hands-on experience with Google Cloud Platform (GCP)
- Expertise in BigQuery (partitioning, clustering, query optimization)
- Proven experience implementing Medallion Data Architecture
- Strong programming skills in Python and PySpark
- Hands-on exposure on Java
- Advanced proficiency in SQL (complex joins, window functions, performance tuning)
- Hands-on experience with Cloud Dataflow
- Experience with Cloud Composer (Airflow) for orchestration
- Experience working with Google Cloud Storage (GCS)
- Knowledge of version control systems (Git) and CI/CD practices
- Strong understanding of GCP IAM and cloud security best practices
Preferred Qualifications
- Experience working with large-scale enterprise data platforms
- Knowledge of data warehousing and data lake concepts
- Familiarity with real-time streaming frameworks
- Experience in data governance and data quality frameworks
- Exposure to
Similar Jobs
Automotive
Design, develop, customize, and support secure, scalable Salesforce solutions on the Force.com platform. Build Apex, Lightning, Visualforce, Flow, OmniStudio, and integration solutions; create automated tests; oversee interface development; document systems; and support deployment processes. Collaborate with product and business stakeholders, provide Salesforce architecture leadership, identify technical risks, and mentor engineers while delivering CRM and workflow capabilities.
Top Skills:
ApexAWSAzureChatterCopadoFlexcardsForce.ComJavascript Es6Lightning ComponentsLightning Web ComponentsOmniscriptReactRestSalesforceSalesforce FlowsSalesforce Sales CloudSalesforce Service CloudSoapSOQLSoslSQLVisualforce
_0.png)
Aerospace • Hardware • Software • Virtual Reality • Defense
Owns the end-to-end capture lifecycle for Department of Defense pursuits, including competitive assessments, win strategies, teaming, bid/no-bid decisions, customer engagement, proposal development, and post-award transition. The role coordinates engineering, contracts, finance, security, and proposal teams while shaping R&D and prototype opportunities into programs of record. It requires expertise in Shipley methodology, DoD acquisition pathways, training and simulation markets, and emerging AR/XR/LVC technologies.
Top Skills:
Augmented Reality (Ar)Bloomberg GovernmentExtended Reality (Xr)GovwinLive Virtual Constructive (Lvc)Microsoft DynamicsSalesforceSam.Gov
Artificial Intelligence • Marketing Tech • Software • Generative AI • Automation
Own monthly close activities, general ledger operations, balance sheet reconciliations, corporate card and accounts payable workflows, subscription billing reconciliations, equity accounting, and variable compensation calculations. Ensure compliance with US GAAP and internal controls while supporting audits, tax filings, financial reporting, process improvements, and cross-functional issue resolution.
Top Skills:
CartaLlmsMaxioNetSuiteNumericRagRampSaas Billing Platforms
What you need to know about the NYC Tech Scene
As the undisputed financial capital of the world, New York City is an epicenter of startup funding activity. The city has a thriving fintech scene and is a major player in verticals ranging from AI to biotech, cybersecurity and digital media. It also has universities like NYU, Columbia and Cornell Tech attracting students and researchers from across the globe, providing the ecosystem with a constant influx of world-class talent. And its East Coast location and three international airports make it a perfect spot for European companies establishing a foothold in the United States.
Key Facts About NYC Tech
- Number of Tech Workers: 549,200; 6% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Capgemini, Bloomberg, IBM, Spotify
- Key Industries: Artificial intelligence, Fintech
- Funding Landscape: $25.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Greycroft, Thrive Capital, Union Square Ventures, FirstMark Capital, Tiger Global Management, Tribeca Venture Partners, Insight Partners, Two Sigma Ventures
- Research Centers and Universities: Columbia University, New York University, Fordham University, CUNY, AI Now Institute, Flatiron Institute, C.N. Yang Institute for Theoretical Physics, NASA Space Radiation Laboratory
