Archives

Accelerating Data Platform Adoption & Frictionless Use Across the Enterprise

Introduction

A data platform is one of many parts of an enterprise city map. Even though it's not the only platform, it's a significant piece of an enterprise city map that helps teams meet different business objectives and overcome challenges.

When dealing with a data platform, finding the hidden meaning, relationships, and embedded knowledge can still be challenging when attempting to realize the data's value.

Handling big data or real-time unstructured data presents challenges across collection, scalability, processing, management, data fragmentation, and data quality.

A data platform helps enterprises move information up the value chain by helping lay the foundation for powerful insights. Not only does a data platform pull data from external and internal sources, but it also helps to process, store, and curate the data so that teams can leverage the knowledge to make decisions.

The central aspect of leveraging a data platform is to consider it as a horizontal enterprise capability. Teams across the organization can use the data platform as a centralized location to aggregate data and find insights for specific use cases.

On its own, a data platform cannot realize its full potential. Are you setting it up for maximum impact?

While the goal of a data platform is to remove silos in an organization, it is difficult to do so until the organization enables a complete data platform. Then different units can leverage the platform functions so departments will have easy data sharing capabilities.

In this post, we discuss the principles that help ensure teams can optimize their data platform for use across the enterprise.

At GlobalLogic, we refer to these principles as the ‘Synthesize and Syncretize Paradigm’ for implementing data platforms.

These principles help weave together composability aspects into the data platform and lakehouse architectures. Additionally, it utilizes data mesh and data fabric principles with appropriate governance. This paradigm allows the implementation of a 360-degree data platform with enablers for easier adoption and uses across the enterprise as it facilitates the synthesis of platform components for syncretic use.

Principles

Enterprise Data Platform as the Core Foundation

The core data platform will form the foundation and own all the capabilities and technology stack to enable the following:

  • Data storage
  • Data ingestion interfaces for ingesting data into the storage layer
  • Data processing during the ingestion and post-ingestion phases to transform and enrich the data
  • Data access interfaces
  • Endpoints for data ingress and data egress
  • Orchestration and scheduling
  • Data governance and data cataloging
  • Control pane, monitoring, and security
  • Data querying and data analytics

Teams will need to enable continuous delivery of new data platform features with centralized governance.

The Interplay of Domains & Data Products

Domains must be first-class concepts in the entire setup.

Teams can link domains to business aspects, data origin, use cases, source data, or consumption. Additionally, teams can enable particular feature sets within domain systems depending on the need.

Domains will vary from organization to organization since businesses closely tie domains to their organization's structure and design.

The core data platform foundation must be compatible with data products and domains. Teams can build their own data products for a domain on top of the core data platform foundation. Teams can also deliver data products in an agile fashion for incremental business value realization.

Microservices Based Architecture

The core data platform foundation will have a decentralized microservice architecture. This architecture provides API, messaging, microservices, and containerization capabilities for operationalizing data platform features.

The decentralized microservice architecture will enable the enterprise data platform so teams can use it as a central base with a decoupled architecture.

A team can leverage these capabilities to ensure the platform is resilient, elastic, loosely coupled, flexible, and scalable.

This will allow different domain teams to operationalize the data and features across the enterprise for their feature sets.

They also enable data and decision products in a domain on top of the unified data platform to access reliable data ubiquitously and securely.

Composability

The ability for teams to select the tools and services in a frictionless manner for their data products within a domain is crucial since it allows teams to assemble the required components. In addition, a composable architecture will enable teams to fabricate the necessary elements to deliver data and decision products.

This architecture paradigm will utilize both the infrastructure aspects as well as microservices.

A microservices-powered composable architecture for infrastructure, services, and CI/CD processes will allow separate teams and domains to utilize the same data platform infrastructure stack. The key to delivering a composable architecture is when the team focuses on DevOps and automation practices.

This will also enable dynamic provisioning with the definition of scalability parameters during the provisioning process itself.

Self Serve Data Platform Infrastructure

Teams should be able to use the data platform technology stack, features, and infrastructure. Teams can use a “No Code” or a “Low Code” approach with portals and self-service capabilities to enable these functions.

This principle will help teams reduce difficulties and friction when using and provisioning their environment. This will also help teams leverage the data platform to become a first-class asset across the enterprise and become the source of accurate data.

Discoverability & Data Sharing

Discovering and utilizing the platform and data assets elements is crucial to enable ease of synthesizing the right set of necessary components.

Data management is essential to catalog and manage data assets and datasets. Another important component is automation. It’s crucial to use automation for auto-discovering, tagging, cataloging and profiling data, and data classification with relationship inferences. This will enable teams to discover and utilize data assets efficiently.

Similarly, another key to discovering the capabilities is a catalog of available platform elements and features. This can cover the data connectors, existing data pipelines, services, interfaces, and usage guides.

The data platform also needs to have mechanisms for data exchange to ensure teams can effortlessly share data with appropriate access controls applied.

Centralized Governance

Centralized governance is a pillar to enable interoperability between various domains and teams and their data products. It will also ensure proper controls on new data platform features development and operationalization based on the actual needs of the teams so that they can quickly realize business value. This will act in conjunction with the data governance processes, data stewardship, and data management to ensure teams can access and share datasets in a controlled manner.

360-Degree Data Platform to power business with GlobalLogic

A data platform that leverages the above principles enables frictionless platform use and thereby accelerates utilization of the platform capabilities across an organization and value realization.

At GlobalLogic, we help our partners implement end-to-end modern data platforms with our big data and analytics services. Reach out to the Big Data and Analytics team at practice-bigdataanalytics-org@globallogic.com – let’s explore your data platform implementation options and how to drive the adoption of data platforms across your organization.

The stakes are high! Place your bets on GlobalLogic.

Sports betting and online gaming companies are racing to offer the best customer experience. The stakes are high, and they have to build a secure, compliant, and reliable platform and mobile apps soon! And working with a matured software engineering partner like GlobalLogic determines who wins the race! With its robust reference architecture and leveraging the power of digital technologies, GlobalLogic can deliver engaging and data-driven experiences. Download the ebook to learn more.

Cloud – A Great Refactor for the Financial Services Industry

The banking, financial services and insurance (BFSI) sectors are customer-service driven, document-reliant, and compliance-focused. You know the ongoing challenges. Time-consuming, repetitive data entry tasks across multiple platforms can lead to human error, processing delays, and lost opportunities to personalize marketing and cross-sell products. 

Digital transformation fueled by cloud-based technology is changing the game. Artificial intelligence (AI), natural language processing (NLP), machine learning, optical character recognition (OCR), and intelligent automation are reshaping the future of the financial services industry. Here’s how.

Advantages of Digital Transformation in BFSI

A study by Allied Market Research determined the global digital transformation in BFSI market was valued at $52.44 billion in 2019 and is projected to reach $164.08 billion by 2027. Among the factors driving the transformation have been the widespread use of mobile devices, developments in the Internet of Things (IoT), and cloud technology. 

Intelligent automation including AI, NLP, machine learning and OCR backed by cloud technology can:

  • identify new revenue streams through technology
  • attract (and retain) customers through seamless omnichannel experiences
  • improve decision-making through powerful data analytics
  • mitigate risks through fraud detection and regulatory compliance solutions.

Increased Data Handling Capacity in the Cloud

One of the challenges BFSI encounters is the documentation required in day-to-day financial operations. Much of the required information is on paper, in emails or faxes, or on photocopies or even carbon copies that deteriorate over time. In addition, documentation takes a great deal of storage, is not easily searchable, and can lead to delays, errors, and missed opportunities for cross-selling and personalized customer experiences.

Enter intelligent automation. OCR can digitize data from a variety of sources, including faxes, paper, email and notes, making it accessible and searchable. Machine learning and artificial intelligence can “learn” a financial institution’s systems, identifying and flagging areas of weakness or areas of concern. Documentation stored in the cloud is quickly retrievable, yet takes a fraction of physical storage space.

AI and machine learning can scan, analyze, sort, distribute and file documentation, it can flag discrepancies or missing information, send notifications, follow up for information or escalate, and perform the repetitious data entry, which frees the employees to do higher-value work, such as customer retention or investigating more complex issues or problems. 

AI can scan for customer profiles across omnichannel quickly, and flag potential duplicates or fraudulent accounts. OCR and machine learning can detect anomalies in photo identification and flag for investigation in real-time and can research multiple accounts simultaneously. This level of compliance can provide additional security and protection.

Augmented Customer Experience & Support

Robotic process automation (RPA) can employ intelligent automation and natural language processing to provide an enhanced customer experience. For example, Odigo is a world-leading Contact Center as Service (CCaaS) provider that handles 3 billion customer interactions per year. They have partnered with Global Logics to expand their product’s capabilities. 

One advantage of CCaaS is the ability for companies to only purchase the technology they require, to handle customer service inquiries, chat, email and social media, and other messaging using intelligent chatbots and natural language processing. AI with NLP can escalate to an employee at any point during the interaction, and machine learning means the bots “learn” through interactions, providing more complete and robust information to inquiries based on previous interactions. 

AI can input a customer profile, search for other customer accounts across multiple systems, request a welcome letter or package, confirm identification based on compliance protocols, complete Know-Your-Client (KYC) information, and begin to search for personalized recommendations based on information. 

AI and NLP can provide customer service in the customer’s language of choice, in multiple time zones simultaneously, and can scale quickly to meet increased demand or need. AI can operate 24/7/365, providing an enhanced customer service experience with access to financial services on the customer’s schedule, rather than during traditional banking hours. 

Security & Blockchain Applications in BFSI

Cloud technology provides enhanced business continuity, mitigation of risk and cybersecurity measures. More transactions are being conducted digitally using the IoT – for example, insurance packages can now be customized using a vehicle’s telemetry data. As more of these transactions and processing happen at the edge, the need for more secure hardware and data transmission increases. 

Security access protocols such as multifactor authentication, robust identity access management protocols, continuous monitoring, and encryption can allow for secure transmission between data warehouse/analytics in the cloud and processing at the edge. AI can retrieve information from cloud technology in a fraction of the time it takes a human employee to cross-reference and search information, providing enhanced fraud detection and cybersecurity measures.

Security is only as strong as its weakest component, so it is essential for BFSI to invest in secure hardware and employ multiple encryption and security protocols. Cybersecurity in BFSI is becoming more challenging as cyber-attacks become more sophisticated. One of the ways that the financial sector can protect cloud transactions is to combine AI with blockchain applications. 

Blockchain provides a transparent real-time chronology of transactions using a decentralized public ledger. As each transaction creates a block, every person in the network receives a copy of the ledger. This makes alterations difficult and provides a complete audit trail of each transaction. 

Money transfers, direct payments, transaction tracking, and fraud reduction can be completed quickly using blockchain, as the transaction can be monitored by all parties every step of the way, and blockchain encryption provides an extra layer of security. Blockchain can reduce costs and provide enhanced transparency, an enhanced audit trail, and accountability.

Algorithmic Trading

Machine learning and AI can monitor and track trade volumes, analyze historical trade data, and then use the information to formulate recommendations for future investment strategies. In addition, AI can automatically execute a trade based on preset buy/sell/hold instructions which will be triggered when criteria such as time, price, volume or call and put option instructions. 

As trading volumes increase and client expectations become more complex, the pressure on trading desks to improve execution performance is steadily increasing. Machine learning enables algorithms to “learn” how to make different decisions and consider myriad data points to make smarter trades. Core trading algorithms will become increasingly intelligent and complex, evolving into a sort of contextual playbook versus a strict set of rules.

Final Thoughts

Financial services firms are now using machine learning to predict cash flow events, fine-tune credit scores, and detect fraud, among other important functions. This refactoring of the financial services industry, being driven by advancements in technology and rapidly evolving customer expectations, will propel businesses that are positioned to capitalize on the opportunities to the next level.

With 15+ years in BFSI, including 1200 dedicated engineers and expertise in regulatory compliance and control, Global Logic is helping its partners reshape their businesses – and the industry as a whole. How can we help you embrace these digital trends and transform your business? Get in touch and let’s find out.

Enterprise Blockchain for Digital Coupons

Over the years, digital coupons have become more popular with customers and throughout enterprise marketing strategies. Companies can distribute coupons throughout their website, apps, and social media to promote discounts and create opportunities to maximize their revenue.

Additionally, companies utilize third-party services with blockchain, distributed ledger technology, and smart contracts to minimize coupon cost management and distribution.

There are numerous advantages and use cases for companies to utilize blockchain technology and platforms for coupon campaigns. Learn about the impactful ways to incorporate blockchain into your coupon marketing strategy and the critical components behind it. 

The Future of Cloud-Driven Manufacturing: Built to Scale

The last two years have upended the global marketplace and the manufacturing sector is no exception. As the global pandemic rolled across the world, plants shut down, supply chains were disrupted, and widespread socio-economic instability ensued. In an effort to minimize future disruptions, mend operational inefficiencies revealed by the pandemic, and get ahead of changing consumer expectations, the manufacturing sector is undergoing a digital transformation. Welcome to the era of Industry 4.0.

Digital transformation is driving big gains for businesses by improving operational efficiency, reducing costs, improving product quality, and enabling quicker responses to evolving market requirements and customer demands. There are benefits as far as eco-positivity via reduced energy, lower material consumption, proactive monitoring, etc. as well.

These innovations mean we’re now seeing robotic processing automation, artificial intelligence, machine learning, augmented reality (AR) and virtual reality (VR) all working together in the Industrial Internet of Things (IIoT) to provide manufacturers resilient, agile solutions for persistent challenges. 

A recent Gartner survey found that 36% of manufacturing enterprises realize above-average business value from IT spending in digitalization at a reasonable cost when compared with peers. Is your manufacturing operation on trend? Let’s take a look at what cloud-driven smart manufacturing and Industry 4.0 look like in practice.

Digital Twins

One of the challenges of innovation in manufacturing is the sheer size of equipment, space, and logistics. Shutting down a production line to repair, replace or add a part or piece of equipment is expensive, potentially hazardous, and time-consuming.  In addition, despite the best measurements, fixed structures, wires, overhead beams or doors can be missed in the design phase, requiring costly changes and repeated shutdowns until the repair or part is installed, tested, and completed. 

Artificial intelligence uses augmented reality (AR) and virtual reality (VR) to create a 3-D model of the equipment, component or space, and then developers and engineers can work with this digital twin technology to design, tinker, adjust and perfect the equipment in virtual simulation before it is built and installed. 

According to a study by Gartner, 13% of organizations that have implemented Industry 4.0 and IoT are employing digital twin technology, and a further 62% are in the process of implementation. 

Some of the benefits of digital twin technology include reduced risks, accelerated production time, remote monitoring, enhanced collaboration, and improved fiscal decision-making thanks to advanced analytics and rapid testing in the cloud. 

GlobalLogic is a leader in building digital twin technology. Learn more about how it works in “If You Build Products, You Should Be Using Digital Twins.”

Predictive Maintenance

Equipment breakdowns and malfunctions are costly, time-consuming, and potentially dangerous to employees. One advantage of Industry 4.0 and digital twin technology is the ability to perform predictive maintenance in VR/AR. Unlike preventative maintenance, which is performed on a schedule whether the servicing is actually required at that point in time, predictive maintenance relies on data to predict when the maintenance should be performed. Successful predictive maintenance capabilities are dependent on the use of artificial intelligence, sensors, and cloud solutions. 

According to the US Department of Energy, an investment in a PdM strategy can reduce maintenance costs by up to 30%, reduce the number of unexpected breakdowns by ¾ and reduce the number of downtime hours by almost half. If properly implemented, it is also 25% cheaper compared to preventive maintenance.

The IoT sensors generate big data in real-time, and artificial intelligence and machine learning can analyze, flag anomalies, and initiate repair protocols before a problem halts production. Digital twin technology can scan a production operation from all angles continuously, and make recommendations for predictive maintenance that can be scheduled rather than completed on an emergency basis. This saves time, decreases production downtime, increases efficiency and safety, and mitigates risk.

Robotics & Autonomous Systems

Whether it is a full-scale automated robotic processing system, or a single station collaborative robot (cobots), robotics and autonomous systems have been changing the manufacturing landscape. 

Since the pandemic, however, robotics and autonomous systems have been driving digital transformation. Robots can work 24/7/365, they don’t take vacation, sick days or personal time off. They can provide rapid ROI and improve productivity while freeing human workers to do higher-value tasks. 

In recent years, the incorporation of AI, VR/AR, and machine learning has been employed to work side-by-side with human workers, and cobots with end-of-arm-tools equipped with machine learning can be moved by a worker in “teach” mode, and then it operates autonomously, becoming more efficient as it “learns” the task. 

The next innovation in robotics is individual microsystems, designed to work as autonomously as possible, while still collaborating with other microsystems. That way, if other microsystems crash, the others can continue to operate. Each microsystem is “choreographed” to work with others in collaboration while doing its part. It can be easily scaled and coordinated. Think of it as a colony of bees, each worker autonomous, but contributing to the whole synergy. Check out “Collaborating Autonomous Systems” to learn more about GlobalLogic’s work with microsystems. 

Connected Devices and the IIoT

Just as the IoT connects your smartphone to your thermostat, television, tablet, or speakers, the Industrial Internet of Things (IIoT) connects smart applications in manufacturing and industry. For example, the IIoT connects sensors on a cobot with the engineer’s tablet in another building, or the alarm system that activates if a sensor detects a chemical spill or heat increase. IIoT relies on cloud technology so that the data can be accessed from anywhere.

The data from these many sensors, controllers, and attached servers is often distributed across many remote locations. The data is uploaded continuously to the cloud, allowing for real-time updates at any time across multiple locations. McKinsey predicts that IIoT will be a $500 billion market by 2025.

One advantage of the IIoT is it provides simultaneous data from multiple locations and sources, whether within the same manufacturing facility or spread across multiple facilities or geographic locations. The cloud allows for centralized management of all the IIoT resources, but that management can happen from anywhere in the organization or the world. It provides business continuity and resilience if one location experiences an emergency or natural disaster, as operations can continue at the other locations, and real-time updates allow for quick response. 

Traditional IT storage requires hardware, system ware, servers and massive databases, and if the location goes down, the data can be lost. With IIoT cloud technology, the data is protected and accessible, while being encrypted and safeguarded by the cloud cybersecurity protocols.

IIoT is a form of edge computing, where the goal is to bring the resources from traditional data centers and bring them as close as possible where they are needed while maintaining safety, data protection, and guarding against cyber-attacks. 

GlobalLogic’s “Immunizing Edge Computing” takes a more in-depth look at how to protect data when working on the edge.

Conclusion

These are just a few examples of how cloud technology is transforming the manufacturing space. Intelligent automation, including VR/AR, artificial intelligence, machine learning, automated robotic processing, and autonomous microsystems are leading smart manufacturing innovations.

As more automation moves to edge computing – whether it’s a sensor, a pump, a car or a gateway – this trend will continue as the costs of computing power and related resources continue to decline. Determining precisely how to use the cloud and what can happen at the edge is an integral part of your smart manufacturing strategy and working with an expert in cloud technology is an important part of your intelligent automation business plan.

As innovation continues to evolve, the “edges” will get smarter, allowing for more powerful collaboration. With machine learning, the more the edge nodes “learn”, analyzing data, sensing the environment, and processing data, the more information will be available to share, whether peer-to-peer or through a network. IIoT will allow for smart edge collaboration in one form or another. 

Single station cobots, warehouse robots, and self-driving autonomous cars will continue to be innovation-driven, representing the span of collaborating autonomous systems, with no limits on the horizon. Intelligent automation, the IIoT, and other applications will continue to evolve robust, scalable, powerful systems with nuanced behavior. 

See how we can help you harness the power of cloud to engineer products to scale your manufacturing here.

Best practices for selecting a software engineering partner

More than ever, sports betting companies are under pressure to enhance customer value by tapping into the full potential of digital technology. However, traditional outsourcing models are not suited for digital engineering products. So understanding the difference between conventional outsourcing vendors and today’s software engineering partners is critical to success.

Cloud-Driven Innovations: What Comes Next?

Cloud computing has revolutionized the ways we store, manage, and process data, with fully 82% of enterprise survey respondents indicating they’d deployed a hybrid cloud in their organization by late 2020. Cloud technology is resolving some of the most persistent computing challenges and roadblocks. Beyond this, though, it is transforming business in myriad ways.

The business benefits of the cloud are many; for starters, it’s enabling organizations to build better, smarter applications much faster, cutting down the time to implement and deploy, as well as capital expenditures. This higher velocity development cycle empowers companies to take their products to market much quicker, even as opportunities for experimentation and testing increase. What once took three months to deploy on-site takes just minutes in the cloud – no new server hardware or enterprise software required.

Here are just a few examples of how the cloud is making the product and platform innovations an exceptional reality.

DevSecOps

Over three-quarters of security team professionals continue to report that devs find too few bugs too late in the process, according to GitLab’s 2021 Global DevSecOps Survey.

Bugs can have massive consequences. Last year, the average cost of a data breach topped $4.24 million USD, the highest level in the 17 years IBM has tracked this figure. What’s more, researchers found that the maturity of an organization’s cloud migration had a direct and measurable impact on the severity and cost of a breach.

“Organizations further along in their cloud modernization strategy contained the breach on average 77 days faster than those in the early stage of their modernization journey,” the report states.

Taking a security-first approach to DevSecOps is mission-critical. Take the financial services industry, for example, where cloud-enabled services are facilitating more seamless, rapid transactions than ever before across authentication systems, transaction processing systems, payment and billing systems, and databases. Relying on reported data to uncover fraudulent activities leaves the institution vulnerable and reactive.

Using AI-enabled technology and machine learning on data from these various sources in the cloud enables us to train the algorithm in fraud detection so anomalies can be flagged immediately. You can see here how this technology helped a retail bank with 14 million+ active customers and multiple brands achieve a significant reduction (50%) in certain types of fraud and save a predicted $1.3 to $3.9 million USD each year. 

API-led Integrations

Point-to-point integrations are difficult to maintain and modify, requiring a great deal of time and labor and, too often, still resulting in failure.

API-led integration ensures the seamless flow of data between applications for improved accessibility, enhanced visibility, and greater agility. Solutions become reusable and can be accessed by multiple users inside and outside the organization.

Low Code No Code (LCNC)

Today, Gartner research says that on average, 41% of employees outside of IT are tasked with customizing or building data or technology solutions. Gartner predicts that half of all new low-code clients will come from business buyers that are outside the IT organization by year-end 2025, too.

And as my colleague Nitin Unni pointed out in his recent paper Agile Transformation of the Modern Enterprise, with an increasing need for agile digital transformation and market realities, LCAP/NCAP solutions could become a game-changer for modern businesses. Be sure to give that a read for the factors you should consider in choosing the right platform.

LCNC in the cloud enables developers to build solutions much faster, as well as redesigning and automating workflows. However, it’s still important to have safeguards including data and process governance and security in place to prevent any unintended consequences. 

DesignOps

DesignOps is essential for building standardization and scale into workflows where teams are building digital products. When coupled with cloud technology, this operational management of designers and optimization of design processes can help your organization overcome technical debt, reduce overhead costs, and mitigate performance gaps.

DesignOps is helping brands reimagine the customer experience, as well. When an iconic fast food brand sought to express itself in a digital context, for example, we designed and delivered operational service enhancements to ensure a consistent, coherent experience for customers worldwide. The cloud facilitates rapid iteration, testing, and deployment at a scale that would take years to achieve in the real-world environment alone.

Universal Observability/Open Telemetry

Formed through the merger of OpenTracing and OpenCensus, Open Telemetry provides standardization in distributed telemetry data. It enables developers to instrument code once, then swap out monitoring tools, compare competing solutions, and run numerous different monitoring solutions in production for varying needs.

What Comes Next

Much innovation and digital transformation are already happening in the cloud. It’s a trend we can expect to see continue as companies increasingly move their offerings to cloud-native platforms, seeking to automate their most time-consuming, labor-intensive processes.

Microservices platforms, the most commonly deployed architectures in the creation of new experiences, is another technology that is working to reduce friction for DevSecOps by enabling horizontal scale, reducing time to market, and improving ROI. 

Early results are promising; GlobalLogic has seen savings of up to 75 person-months for a typical large-size deployment project. It’s already the foundation platform for companies as diverse as a leading pet insurance provider and a well-known third-party administrator of safety and compliance solutions.

Decision-makers must also consider how the metaverse and web3.0 are going to impact cloud-driven innovations in tech. With content and applications in the cloud and consumers experiencing them in increasingly immersive ways, ecommerce and marketing are ripe for a dramatic transformation. 

How can cloud adoption help your organization modernize, improve customer experiences, achieve efficiencies, and scale successful workflows? GlobalLogic’s cloud engineering professionals are leading the way in design led product and platform engineering and transition clients to fully digital. Let’s explore the possibilities together – reach out to Cloud@GlobalLogic.com.

Security Requirements for the Development Team

In my last post, "Security Training for the Development Team," I shared the experience of building a security training program for the development team. 

Today, I will cover another essential step of securing engineering: security requirements. After reading this blog, I hope you will have a better understanding and be able to improve your security processes by implementing the additional requirements you’ll find here from the beginning of the product engineering phase. 

Understanding a system's security and privacy requirements is critical to building a secure system. Therefore, security requirements must be updated regularly to reflect current requirements and the constant threat of landscape changes. Security requirements should also be defined along with the functional requirements, as this will help the design team to create a better system that can both cover these requirements and build a new security culture and mindset. 

Below, we will discuss an agile development environment and how security requirements can be captured and tracked in different ways.

1. Security Epics And User Stories 

A team can track security requirements similar to the functional and feature requirements in the form of epics and user stories. 

Epics and user stories can document and track business-related security requirements like security requirements for policy/compliance, legal, contractual and regulatory requirements, or system functionalities, design levels like multi-factor authentication, logging, and tracing. These can be prioritized alongside other epics and user stories. 

Security epics and user stories ensure security requirements are considered from the beginning, making the team fully aware of these requirements. The team can also tag these epics and user stories by tracking them separately across a security board. 

2. Security Tasks

In addition to epics and user stories, a team can also track security requirements as part of the security tasks. Security tasks are more suitable for granular or implementation levels, for example, when a task is used to implement data encryption while capturing and storing PII data. 

Security tasks ensure that security is not missed while implementing business features and functionalities. 

3. Security Debt

Security tasks can also be captured and tracked through a function called security debt. Debt is those tasks that cannot be properly implemented because of the prioritization part of this list. 

But as with technical debt, a team should be cautious when tracking security debt, as this can grow rapidly and become a significant risk if it is not taken care of within a reasonable time. 

4. Operational Security Tasks 

The team can also capture and track operational-related security tasks as part of their operational security tasks. 

These tasks are typically not related to security requirements, meaning they are operational. Examples could include updating the development environment regularly, updating open-source libraries regularly (at least before each major release), resolving issues reported by SAST, etc.

5. Specialized Security Tasks

The engineering team can implement all of the above without any significant help from the security team. 

But there are a few tasks that require a deep understanding of security. These can be captured and tracked through specialized security tasks such as the performance of threat modeling, PEN testing, environment hardening, etc.

The Evolution of Data & Analytics Technologies

A Google paper on the Google File System, published in October 2003, and Jeffrey Dean and Sanjay Ghemawat’s MapReduce paper in 2004, kicked off the era of big data technologies. Shortly thereafter, Doug Cutting and Mike Cafarella, then working with Yahoo! on a search engine called Apache Nutch (based on Apache Lucene indexing), created the new Hadoop subproject for running a large-scale computation on large clusters of commodity hardware in January 2006. 

Since these early efforts, the big data technology landscape has been enriched with numerous innovations and evolved in leaps and bounds. In part one of this blog series, we’ll look at the evolution of the data and analytics space across their core aspects.

Data Platforms Evolution

 

OSS based → Packaged Distributions → Cloud-Native Stack → Cloud Agnostic stack → Composable Data Analytics

 

Soon after Hadoop released an Apache open source project, it spawned several frameworks on top of Hadoop to perform different types of processing. Apache Pig, Apache Hive, Apache HBase, Apache Giraph, and Apache Mahout were a few of the diverse frameworks that allowed different ways to process data stored in a Hadoop cluster. 

In addition, there were parallel stacks that replaced one or more frameworks with Kafka, Cassandra, or Cascading. The initial deployments required teams to build and deploy software on commodity hardware based on open-source Hadoop ecosystem components. 

After Hadoop’s project came the commercial distribution of Cloudera, Hortonworks, MapR, Hadapt, DataStax, and Pivotal Greenplum, which packaged the required software in a user-friendly fashion and provided premium support. Then, Amazon EMR released the first cloud-based Hadoop distribution. 

Now there are cloud-specific Hadoop and Spark-based distributions like Azure Synapse Analytics GCP DataProc that come pre-installed with the required software and computing power. 

From there, cloud-agnostic stacks such as Snowflake and DataBricks evolved to work efficiently across different clouds. These platforms are adding innovative features which cater to key performance and cost metrics. As a result, these technologies are getting quite popular, with many enterprises now moving towards such cloud-agnostic stacks.

Enterprises are increasingly looking at a multi-cloud strategy to avoid lock-ins by a particular cloud and use the best technology for various purposes. The trend for the future is to move towards composable data analytics, where companies will build data platforms using components from two to three different technologies and cloud providers.

Data Architecture Evolution

 

Data Warehouse → Data Lakes / LakeHouse → Data Mesh / Data Fabric

 

For decades, data warehouses such as Teradata and Oracle have been used as central repositories for storing integrated data from one or more disparate sources. These data warehouses store current and historical data in one place that can create analytical reports for workers throughout the enterprise.

With the advent of big data frameworks like Hadoop, the concept of a data lake became incredibly popular. Typically, a data lake is a singular data storage and includes raw copies of source system data, sensor data, social data, and transformed data for reporting, visualization, advanced analytics, and machine learning tasks.

Data mesh is an organizational and architectural paradigm for managing big data that began to gain popularity in 2019. It is a process and architectural approach that delegates responsibility for specific data sets to business members who have the subject matter expertise to use the data properly. With the data mesh architecture, data domains become prominent with a base data platform that individual domain teams can use with their own data pipelines. 

In this situation, data resides within the foundational data platform storage layer (data lake or data warehouse) in its original form. Individual teams will choose how to process this data and then serve the datasets they own in a domain-specific manner.

Data fabric is a design concept defined by Gartner that serves as an integrated layer (fabric) of data and connecting processes. A data fabric utilizes continuous analytics over existing, discoverable, and inference metadata assets to support the design, deployment, and utilization of integrated and reusable data across all environments, including hybrid and multi-cloud platforms. 

Data mesh and data fabric are concepts that provide a unified view of the data distributed across the underlying data lakes and data warehouses. They, however, differ in how users access them. While data mesh is about people and processes, a data fabric is an architectural approach to tackle the complexity of the underlying data. Experts believe these concepts can be used simultaneously and will work on top of the existing data lakes and warehouses.

Data Processing Evolution

 

Batch Processing → Stream / Real-time Processing → Lambda → Kappa / Delta

 

Initially, the big data solutions were typically long-running batch jobs to filter, aggregate, and otherwise prepare the data for analysis. Usually, these jobs involved reading source files from scalable storage like the Hadoop Distributed File System (HDFS), processing them, and writing the output to new files in scalable storage. The key requirement for these batch processing engines is the ability to scale out computations in order to handle a large volume of data.

The stream, or real-time processing, deals with data streams captured in real-time and processed with minimal latency to generate real-time (or near-real-time) reports or automated responses. Frameworks like Apache Kafka, Apache Storm, Apache Spark Streaming, Amazon Kinesis, etc., help enable this capability.

The next evolution was the Lambda architecture, a data-processing architecture designed to handle massive quantities of data by taking advantage of both batch and stream-processing methods. This approach to architecture attempts to balance latency, throughput, and fault-tolerance by using batch processing to provide comprehensive and accurate views of batch data while simultaneously using real-time stream processing to provide views of online data

Jay Kreps then proposed the Kappa architecture as an alternative to the Lambda architecture. It has the same primary goals as the Lambda architecture but with an important distinction: all data flows through a single path, using a stream processing system.

Another alternative is the Delta architecture, which introduces a table storage layer to handle stream and table storage accessed through a single code base. Databricks proposed this architecture, with Delta Lake at the center of the architecture. Delta Lake is an open-source atomicity, consistency, isolation, durability (ACID) table storage layer over cloud object stores.

The Lambda architecture and its variants, the Kappa and Delta architecture, will continue to be valuable architectures in the near future.

This concludes the first part of the blog series. We’ll continue to explore the evolution of the data and analytics space in subsequent blog posts in this series in the coming months.

Resources:

Questioning the Lambda Architecture, Jay Kreps, 2014.

Edge Computing for Facial Recognition & Emotion Detection

In this document, we will discuss the key features of Edge Computing, the models of operation, how a distributed framework can move compute resources within the same data servers, and look at the benefits of using Edge Computing in accordance with facial recognition. 

We will also explain the technology behind facial recognition and facial emotion detection by delving deeper into the different hardware and software tools used, so you can gain an accurate understanding of how these considerations impact modern facial recognition technology and development.

  • URL copied!