Phyton with Spark Developer

Infotel UK

Job Overview

Location

Chennai, Tamil Nadu, India

Employment Type

Full-time

Work Arrangement

On-site

Sector

Information Technology & Software

Experience Level

Mid-level (3-5 years)

Application Deadline

June 27, 2026

About the Company

Infotel UK Consulting, a part of the broader Infotel Group, is a dynamic and innovative consulting firm dedicated to enhancing business performance through strategic IT systems and applications. We partner with companies globally, leveraging our expertise to drive significant improvements in their operational output.

We foster a challenging yet rewarding environment where IT professionals can thrive and develop their analytical and technical skills. Our culture encourages continuous learning and the adoption of cutting-edge technologies. Joining Infotel UK offers a fantastic opportunity to work alongside passionate IT experts and contribute to impactful projects that shape the future of technology solutions.

Job Description

We are seeking a skilled Python with Spark Developer to join our dynamic team in Chennai. This role is crucial for designing and implementing robust data pipelines that drive business intelligence and strategic decision-making.

You will be responsible for developing and optimizing ingestion, transformation, and enrichment pipelines using Python, PySpark, and SQL. This includes writing and optimizing complex SQL queries, collaborating with data architects and analysts, and ensuring code quality through unit testing and integration testing.

Key responsibilities also involve maintaining CI/CD pipelines for automated deployment, monitoring production workloads, troubleshooting performance issues, and documenting pipeline designs. You will also contribute to innovation, suggest new technical practices, and participate in Agile ceremonies.

The ideal candidate will possess strong hands-on experience with Python frameworks, PySpark DataFrames, Spark SQL, and performance tuning. Advanced SQL knowledge, familiarity with data wrangling and visualization using Python packages, and experience in a Linux/Unix environment are essential. Experience with build and DevOps tools like Maven, Bitbucket, and Jenkins, along with knowledge of Agile and Scrum methodologies, is highly desirable.

To apply for this role, click the Apply button on this page and follow the instructions.

Required Skills

PythonPySparkSQLETL

Key Responsibilities

Design and develop robust ingestion, transformation, and enrichment pipelines with Python, PySpark, and SQL.
Write and optimize complex SQL queries, analytical UDFs, and window functions for data aggregation and reporting.
Collaborate with data architects, data scientists, and business analysts to translate functional requirements into technical specifications.
Perform unit testing, integration testing, and code reviews.
Maintain CI/CD pipelines (Git, Jenkins, Docker) for automated build, test, and deployment of jobs.
Monitor production workloads and troubleshoot performance bottlenecks, memory issues, and job failures.
Document data lineage, pipeline design, and operational run-books.
Stay updated with the latest technologies and trends, providing input and recommendations.
Contribute towards innovation (e.g., AI/ML) and suggest new technical practices for efficiency improvement.
Participate in Agile ceremonies (sprint planning, daily standups, retrospectives) and help groom backlogs.
Mentor junior engineers and champion best practices in Python coding, Spark optimization, and data-engineering patterns.
Evaluate emerging technologies and deliver proof-of-concepts.

Qualifications

Bachelor’s degree or equivalent.
Good knowledge of front-end technologies, preferably Flask.
Resourceful to quickly understand complexities and provide solutions.
Good experience in technical analysis of n-tier applications with multiple integrations using object-oriented, APIs & Microservices approaches.
Strong knowledge of design patterns and development principles.
Inclination and prior experience working across SQL, Python, and ETL.
Strong hands-on experience with Python (NumPy, pandas, Python Frameworks, Restful APIs, MS-SQL or Oracle).
Experience with PySpark - DataFrames, Spark SQL, Structured Streaming, performance tuning (partitioning, caching, broadcast joins).
Advanced SQL – complex queries, stored procedures, query optimization.
Good knowledge and experience using Python packages such as Pandas, NumPy, etc., for data cleaning, wrangling, analysis, and visualization.
Good experience in development and maintenance of code/scripts in both functional and technical specifications of all application components, bug fixing, and production support.
Good knowledge on Linux/Unix environment (basic commands, shell scripting, etc.), testing phases, documentation, and new frameworks.
Some experience working with build tools like Maven & DevOps tools like Bitbucket, Jenkins.
Knowledge of Agile, Scrum, DevOps.
Development experience in a Data Engineering environment.
Ability & willingness to learn & work on diverse technologies (languages, frameworks, and tools).
Self-motivated, good interpersonal skills, and inclination to constantly upgrade new technologies and frameworks.
Good communication and co-ordination skills.

How to Apply

To apply for this role, click the Apply button on this page and follow the instructions.

Join Our Communities

WhatsApp Channels

Global Jobs Gulf Jobs Nigerian Jobs Indian Jobs

Telegram Groups

Global Jobs Nigerian Jobs Gulf Jobs

The data engineering landscape is rapidly evolving, with a significant surge in demand for skilled professionals capable of managing complex data pipelines. This role is pivotal in shaping how data is processed and utilized, directly impacting business intelligence and strategic decision-making. You will leverage cutting-edge technologies such as Python, PySpark, and advanced SQL to build and optimize robust data ingestion, transformation, and enrichment pipelines. Expertise in data warehousing, ETL processes, and cloud-based data solutions will be crucial for success. Your contributions will drive efficiency and unlock valuable insights from vast datasets, supporting the company's growth objectives.