Шлях до BigData. Частина друга: досвід GlobalLogic

Author: Oleksandr Fedirko, Solution Architect, Trainer, GlobalLogic Ukraine

Last time, we looked at the ways that exist for those who want to join the BigData community and become a big data enthusiast. In this column, we will delve deeper into the GlobalLogic team’s experience: we will talk about our projects and GL BigData ProCamp courses .

How is GL BigData ProCamp different from online courses?

The most valuable thing about GL ProCamp is, of course, the opportunity to communicate with expert practitioners, get a lot of life hacks, and learn about best practices used in real projects.

GlobalLogic has clients in various areas: marketing, sales, robotics, financial institutions, automotive, IoT. On projects, we build platforms for data analysis and collaborate with other technology areas (DevOps, Embedded, Data Science, QA) to create quality solutions.

I will give a few examples:

• One of the marketing projects solves the problem of MTA (Multi Touch Attribution), which combines online advertising with offline purchases in stores. This allows large companies producing products for supermarkets to better plan their advertising campaigns and understand which advertising has the greatest impact on the consumer’s final decision.
• Another example: every time you visit any website, an ad appears on it. How does this happen? How does the website know which ad to show? Immediately after you arrive on the website, within a very short period of time (10–20 ms), bidding takes place for the right to display an ad unit. Bidding takes place on Ad Exchange platforms. One such platform is being developed by GlobalLogic engineers.
• Another example. The project focused on data processing in the context of security. The client has a developed infrastructure, approximately 50,000 devices (personal computers, routers , servers, etc.). The task was to develop a system that would be able to detect cyber threats in real time and notify operators about them. One of the main challenges was that the system had to process up to 1,000,000 events per second. The entire infrastructure was deployed in Google Cloud and had auto-scaling capabilities.
• Perhaps the most innovative projects in the context of big data are those related to the Internet of Things: smart devices produce a large amount of data. For example, in one project, GlobalLogic engineers support an analytical platform that collects information from 750,000 devices across the United States for more than 1,000,000 customers. The information is collected, aggregated, and based on this analysis, end users are provided with recommendations for improving their health.

What kind of background do you need to start studying BigData? Is it possible to join the team after completing GL BigData ProCamp?

The entry requirements for this course are quite low. If you know Python or any other programming language, that’s enough. However, remember that after each course, it’s worth consolidating your skills. So find yourself a pet project. Any opportunity to apply the skills and knowledge from the courses to practical problems will help you master them.

If you have completed our course and have a good rating and feedback from GL BigData ProCamp trainers , you will receive an offer to try yourself in the position and project that best suits your profile.

What are the requirements for BigData engineers at GlobalLogic?

Knowledge of SQL is critical as one of the main tools for analyzing and manipulating data. Spark is used in most projects, so knowledge of this framework is very relevant.

Competence in the following areas is also required:

• Spark job performance optimization.
• Distributed computing in general and data warehouse design.
• Quality software implementation, such as CI/CD : what it is and how an engineer deals with it.
• Orchestration of ETL processes.
• Basic questions about stream data processing.
• Working with different types of data storage, such as columnar storage/document oriented/key-value/graph.

In conclusion, the demand for BigData engineers is constantly growing. So, no matter what you do — Java developer, QA engineer, or DevOps — the ability to work with big data will beneficially expand your profile as a specialist and significantly increase your competitiveness in the market.

For example, if you are an Embedded engineer, then BigData is one of the promising ways to expand your specialization, because IoT is closely related to the collection, processing and storage of data from devices. So don’t hesitate, discover the world of BigData for yourself today – register for GL BigData ProCamp!