Job Search
India || Lead Data Engineer IRC233609
| Job: | IRC233609 |
| Location: | India - Noida |
| Designation: | Consultant |
| Experience: | 5-10 years |
| Function: | Engineering |
| Skills: | Azure, Databricks, ETL, Python, Scala, SQL |
| Work Model: | Hybrid |
Description:
NA
Requirements:
Required Qualifications
Bachelor’s degree in Computer Science, Engineering or related field
7+ years of overall work experience working on Data first systems
4+ years of experience on Data Lake/Data Platform projects on Azure
Strong knowledge of SQL for handling relational databases and familiarity with NoSQL databases to manage structured and unstructured data effectively.
Understanding of data warehousing concepts, including data storage, data marts, and ETL processes.
Skilled in using Microsoft Azure services such as Azure Data Lake, Azure Data Factory, Azure Synapse Analytics, and Azure Databricks. These tools are essential for data ingestion, storage, processing, and analytics.
Knowledge of cloud storage solutions provided by Azure, such as Blob Storage and Table Storage, which are integral for data lakes.
Familiarity with ETL tools and frameworks for data extraction, transformation, and loading. Skills in Azure Data Factory or similar tools are particularly valuable.
Ability to perform data cleaning, transformation, and enrichment to ensure data quality and usability.
Highly proficient in programming languages such as Python or Scala, which are widely used in data engineering for scripting and automation.
Skills in scripting to automate routine data operations and processes, improving efficiency and reducing manual errors.
Understanding of how to develop and maintain APIs, particularly JDBC/ODBC APIs for data querying. Knowledge of RESTful API principles is also beneficial.
Awareness of data security best practices, including data encryption, secure data transfer, and access control within Azure.
Understanding of compliance requirements relevant to data security and privacy, such as GDPR.
Experience with data testing frameworks to ensure the integrity and accuracy of data through unit tests and integration tests.
Proficiency with version control tools like Git to manage changes in data scripts and data models.
Advanced skills in data analysis to derive insights and identify data trends, which can help in troubleshooting and improving data processes.
Experience in implementing security measures to protect data at rest and in transit.
Skills in performance tuning and optimization of data processes.
Preferred Qualifications
Knowledge of data privacy laws and compliance requirements.
Experience with automated testing frameworks for data validation.
Familiarity with DevOps practices, including CI/CD pipelines, particularly within Azure environments.
Job Responsibilities:
Responsibilities
Data Ingestion & Integration:
Build and maintain robust ETL (Extract, Transform, Load) processes using Azure Data Factory or similar tools to ingest and transform data from various business systems into the data lake.
Ensure data consistency and integrity across different systems, managing data flows that support real-time and batch processing.
Data Management:
Help manage the organization of data within the data lake, applying techniques for efficient data storage and retrieval.
Support the maintenance of the data catalog, ensuring metadata is accurate and up-to-date, which facilitates easy data discovery and governance.
Develop processes to cleanse and standardize data, ensuring high data quality and usability for analytical purposes.
Enhance data with additional context or insights, potentially integrating external data sources where applicable.
Schema & Metadata Management
Assist in designing and maintaining the schema of the data lake to support data consistency, ease of data access, and performance.
Support the creation and maintenance of a metadata catalog that facilitates data discovery and compliance with data governance standards.
API Support & Data Access:
Develop and maintain JDBC/ODBC based SQL query APIs that allow users to access and query the data platform efficiently.
Monitor and optimize API performance to handle varying load and ensure responsiveness.
Create and maintain detailed documentation for data engineering processes, data models, and API interfaces.
Quality Assurance & Testing:
Conduct routine data quality checks as part of the data ingestion and transformation processes to ensure the integrity and accuracy of data in the data lake.
Participate in testing of the data ingestion and API interfaces, identifying bugs and issues for resolution to ensure robustness of the data platform.
Collaboration and Teamwork:
Work closely with the Principal Data Architect and Principal Data Engineer, assisting in various tasks and learning advanced skills and techniques in data management and engineering.
Provide technical support for business analysts and other users running analytical queries on the data platform, optimizing query performance and data accessibility.
Assist in integrating with BI tools and data visualization platforms, facilitating easy access to data insights.
Platform Monitoring & Maintenance:
Help monitor the performance of data processes and the data lake infrastructure, assisting in troubleshooting and resolving issues that may arise.
Learning & Development:
Continuously learn and upgrade skills in data engineering tools and practices, especially those related to Azure cloud services and big data technologies.
Contribute ideas for process improvements and innovations based on day-to-day work experiences and challenges encountered.
What We Offer
Exciting Projects: We focus on industries like High-Tech, communication, media, healthcare, retail and telecom. Our customer list is full of fantastic global brands and leaders who love what we build for them.
Collaborative Environment: You Can expand your skills by collaborating with a diverse team of highly talented people in an open, laidback environment — or even abroad in one of our global centers or client facilities!
Work-Life Balance: GlobalLogic prioritizes work-life balance, which is why we offer flexible work schedules, opportunities to work from home, and paid time off and holidays.
Professional Development: Our dedicated Learning & Development team regularly organizes Communication skills training(GL Vantage, Toast Master),Stress Management program, professional certifications, and technical and soft skill trainings.
Excellent Benefits: We provide our employees with competitive salaries, family medical insurance, Group Term Life Insurance, Group Personal Accident Insurance , NPS(National Pension Scheme ), Periodic health awareness program, extended maternity leave, annual performance bonuses, and referral bonuses.
Fun Perks: We want you to love where you work, which is why we host sports events, cultural activities, offer food on subsidies rates, Corporate parties. Our vibrant offices also include dedicated GL Zones, rooftop decks and GL Club where you can drink coffee or tea with your colleagues over a game of table and offer discounts for popular stores and restaurants!