SoftElegance's Blog

Dynamic Software Business Solutions

What’s new in a-Gnostics 2.0. Industrial AI service focused on anomaly detection and equipment failure prediction

Reading Time: 8 minutes

We are excited to release a-Gnostics 2.0, the service for rapid development of predictive analytics models, and would like to share details about the platform architecture and the new features available in this release.

Anomaly Detection and Equipment Failure Prediction General

Anomaly Detection and Equipment Failure Prediction General

Background

a-Gnostics, SoftElegance company, implements an Industrial AI service focused on anomaly detection and equipment failure prediction. The service is tailored to multivariable processes and timeseries data, retrieved from industrial equipment to automatically and accurately indicate normal, pre-failure, and failure statuses. The main objective is to apply machine learning and artificial intelligence to predict failures before they occur.

Failure Prediction service at a-Gnostics DataDome

Failure Prediction service at a-Gnostics DataDome

There are a variety of services known as A-SETS that are offered to the customers by the a-Gnostics platform.

The A-SETS all fall into the following high-level categories:

  • The forecasting of electricity consumption by counties and regions, with 96–99% accuracy.
  • The forecasting of energy resources (electricity and natural gas) consumption by large factories, with about 95% accuracy.
  • The forecasting of solar (PV) stations generation, with the accuracy of up to 90%.
  • Failure prediction, anomaly detection, and predictive analytics for industrial equipment, such as boilers at thermal power plants.
  • The list of services is getting extended based on new use cases and implementations of industrial AI and machine learning. For example, a recently added a-Gnostics service is the solution to electrical motor failures using failure prediction to help manufacturers reduce costs of repairs and maintenance.

    Introduction

    One of the main challenges of industrial AI/ML solutions is a need to process a variety of data from different sources. A flexible infrastructure that can scale proportionally to host the data and execute hundreds and thousands of models is also essential. More complexity is added with high security, compliance, data, and model governance requirements. The a-Gnostics platform addresses all these challenges via the abundance of out-of-the-box features and customization capabilities that allow tailoring the services to client’s needs.

    Top-level a-Gnostics architecture

    Top-level a-Gnostics architecture

    The diagram above shows the high-level architecture of the a-Gnostics platform (the most challenging tasks are shown in orange on the diagram).

    The system is composed of the following functional components:

  • Storage — a scalable cloud or on-premises data storage.
  • DataOps — the processes that implement automated data retrieval, validation, and cleansing in preparation to transferring it to the data storage.
  • ModelOps — the automation of AI models governance and lifecycle management.
  • API — an enabler for a-Gnostics services consumption by external systems, analytical platforms, and out-of-the-box UI presenting dashboards and metrics to end users.
  • One of the key principles and advantages of the a-Gnostic platform is its independent infrastructure, in terms of where the services may be deployed and launched. a-Gnostics supports multiple deployment and consumption models:

  • Software-as-a-Service / Model-as-a-Service approach allows the client to connect and use the platform’s services from our secured cloud environment, based on a subscription model.
  • Corporate cloud environment. a-Gnostics supports any cloud provider (AWS, Azure, Google, etc.), used by organizations.
  • Corporate on-premises infrastructure: a-Gnostics may be deployed at a company’s data centers.
  • The video shows an overview of the tasks that need to be solved to develop an automated data pipeline for analyzing industrial data using machine learning methods.

    Modern enterprise encompasses heterogeneous and complex digital environments. To obtain a high-quality training dataset, it is important to have several standard equipment connectors to get the raw data, processing modules to validate and clean the data, and storage that can handle high volumes and variety of data types and formats.

    As companies are gradually moving their infrastructure to the cloud, the data gathering tasks from industrial equipment may be streamlined by utilizing cloud services such as AWS IoT Greengrass, AWS IoT SiteWise, Azure IoT Edge, and more. This also reduces the costs of these integrations and allows greater focus on building AI/ML models and services.

    The main development language for the platform is Python. The system consists of modules and follows the microservices architecture. Each service is represented as a Docker container.

    Data

    The data processed by the a-Gnostics platform can be categorized below:

  • Historical data — mostly represented by csv, excel, and text file formats. Additionally, the system can process industry standard formats, such as WITSML/PRODML, depending on the industry domain as well as proprietary formats established at some organizations. For specific proprietary formats, it is necessary to develop separate processing modules.
  • Real-time data — usually, these are data streams captured from equipment sensors. Direct readings of the data from equipment sensors by external data consumers via the MQTT protocol is a rare possibility. In fact, there is a variety of technologies and protocols for such a data capture and processing. In most cases the intermediate bridge in the cloud environment (AWS S3, AWS Kinesis, Azure Blob, Azure FileShare, etc.) or a separate standing server (e.g. FTP) should be configured to access the data. While establishing such a connectivity, the security measures should be taken into considerations to protect against external interference. The approach that utilizes standard mechanisms brings benefits by shortening the time that needs to be spent on initial infrastructure setup and there is no need, in most cases, to develop custom connectors to handle specific protocols, such as Modbus.
  • Open data — any supporting information from open sources, such as weather facts and forecasts, that will improve the training set or complement the data visualization.
  • Proprietary data — information that is not available in open sources or from the customer. This type of data is used when there is a need to extend the training set to enhance the model and provide additional color to already available data categories. For example, for weather related information, such as cloudiness or solar radiation may enhance resulting model.
  • Regardless of the data categories, most of the data that a-Gnostics processes is time-series data.

    To highlight, high-quality and complete datasets is an essential prerequisite for building accurate models, especially in such an area as industrial AI. It is also very important to provide comprehensive data visualization so end users can easily interpret the data and make fast decisions.

    DataOps

    a-Gnostics utilizes multiple data processing patterns, such as ETL (Extract Transform Load) and ELT (Extract Load Transform). These processes may be automated by utilizing DataOps techniques. DataOps techniques provide automation for building, deploying, and executing data pipelines to achieve scalability to thousands of data processing tasks and AI models.

    a-Gnostics provides a number of ready-to-use data pipelines for converting the raw data to learning datasets for the energy industry, as well as the data processing for general purposes, like weather, which are used in a variety of tasks.

    Raw industrial data needs to go through stages of verifications, cleansings, and transformations.

    Specialized reusable modules have been developed for that, allowing easy integration into data pipelines.

    This collection of data pipelines, reusable transformation modules, and utilizing standard frameworks and libraries streamlines the implementation process and allows concentrating on unsolved problems by avoiding building from scratch.

    In terms of using standard frameworks, Apache Airflow is a good example of when a framework is ideal for tasks that involve a complex chain consisting of multiple datasets.

    In summary, componentization, reusability, and the best practices and frameworks, provided by the a-Gnostics platform is a very powerful capability on the data processing side.

    The data governance framework maintains all necessary meta information about the datasets: versions, original data sources, timing of creation, associated tasks, etc. This provides good visibility for managing, tracking, and analyzing the processes.

    Data Storage

    a-Gnostics stores the following entities:

  • Metainformation.
  • Raw data.
  • Proceed data.
  • Datasets ready for machine learning.
  • Models.
  • Model outputs (forecasts, failure predictions, etc.).
  • PostgreSQL is used as a data store for meta information, model results, and processed datasets. In some cases, the data is represented by files of different formats that are stored in the file system or cloud services such as AWS S3, Azure Blob, Azure FileShare, etc. There are exceptions when it is more optimal to put real-time data into specialized storages for working with time series databases.

    Models

    Development of machine learning models for the Industry 4.0 is a challenging task. A variety of problem types, data sources, etc. carries the risk of slipping into custom development for each model. Therefore, it is important to focus on groups of tasks that are as common as possible for their categories and with good potential for reusability in related areas. To solve real-life tasks in each domain area, deep and wide knowledge of machine learning techniques is required. For example, in tasks like forecasting the consumption/generation of energy commodities (electricity, natural gas, etc.), or predicting the behavior and breakdowns of equipment, it is required to apply both supervised, unsupervised, and semi-supervised learning. Additionally, it is very useful to have a wide outlook and knowledge in the related field to be able to add a new synthetic feature to the model, based on classic math, physics, or chemistry concepts. This combination of approaches helps achieving better model accuracy.

    To make it possible to scale models to hundreds of objects, and train thousands of models, it is critical to automate all stages of model building: data preparation, model training, storage and application to industrial operations. A-Gnostics uses the best practices of the ModelOps approach for the lifecycle management of machine learning models. While we try to utilize standard frameworks such as MLFlow as much as possible, the proprietary services that integrated with MLFlow were built for such tasks as model registry.

    Out-of-the-box, a-Gnostics contains a number of pre-built pipelines for training and deploying models. It allows use of a combination of industry standard algorithms (LSTM, XGBoost, Random Forest) and unique models or complex chains. To automate the process of choosing the most precise model for regression problems, a special walking forward algorithm that accepts a dataset and a set of models as input was developed. As a result, the algorithm returns the most precise model for specific tasks, along with detailed metrics for each model.

    a-Gnostics Electricity Consumption Forecasts

    a-Gnostics Electricity Consumption Forecasts

    To simplify the work with common problems, wrapper classes were created around the models that make it easier to work with regression tasks for timeseries data.

    A good example illustrating the importance of applying ModelOps automation is the task of forecasting of electricity consumption. Mathematically this is a regression problem for timeseries data. In theory retraining the model for the timeseries data should be done as often as possible so that the model remains relevant to the input data stream. New data (facts of electricity consumption) are received every day. After conducting a series of experiments, we determined that for long term benefits it is more profitable to retrain the model every day (while still being able to change the length of the training dataset from time to time). As a result, having N objects for forecasting, as well as M different models, it is required to train N*M models daily. Even with small values of M and N, it is not possible to perform this task manually.

    a-Gnostics enables consumption of the models in the production based on the Model-as-a-Service (MaaS) principle. For example, to get an electricity forecast, the consumer needs to connect with the corresponding module (ElectricityService) via the REST API. A-Gnostics’ modular architecture allows for consumption functionality via an API on the module level.

    API

    To have a fully functional production system, a simple execution of the model is not enough. A software interface with multiple servicing functions is required, so the consumer can interact with the model. For example, data such as the next day manufacturing plan should be supplied to the model to get the forecast.

    For these purposes a-Gnostics provides a variety of API, containing different sets of calls for different tasks, for example, api/v1/forecast gives access to forecast functions, and api/v1/equipment opens the access to industrial equipment data.

    In addition to API, a-Gnostics provides data visualization capabilities via DataDome, a user interface that enables end users to get the full view on the data and to make data-driven decisions.

    Security

    High Security standards are important to follow for any software product, especially for a platform that integrates data and systems across organization. Implementation in the enterprise environment is accompanied by a very scrupulous check by the Information Security Department. To be ready in advance and to minimize the number of iterations during deployment, our team follows the best security practices during implementation, routinely conducting static and dynamic security scans and addressing all potential vulnerabilities.

    We also try to use up-to-date versions of all 3rd party components and services, to stay current with security requirements.

    If the services are launched in the cloud, it is better to avoid running on virtual machines and recommended to use the appropriate cloud services (AWS ECS, AWS EKS, Azure AKS, Azure App Service, etc.) instead.

    Conclusion

    The a-Gnostics platform is successfully used by leading energy and agritech companies, as well as clients from other equipment-heavy industries.

    As of August 2021, more than 100,000 trained models were used in production, and more than 1,000,000 forecasts were executed.

    Merry Christmas and Happy New Year 2021

    Reading Time: 1 minute

    Wishing you Merry Christmas & a Happy New Year from SoftElegance!

    Continue reading

    Clutch Award 2019

    Reading Time: 2 minutes

    SoftElegance Named Top IT & Business Services Provider in Ukraine by Clutch

    We are excited to announce that SoftElegance is officially featured as a leading company in the recent Clutch’s report highlighting the top service providers in Ukraine.

    When your company is suffering from an information technology issue, you want to find a team that has proactive management, is experienced, and understands your business to fix it. The deliverables offered should also be graceful and stylish in appearance. In other words, you want those solutions to be elegant.

    At SoftElegance, our talented developers provide reliable services for implementation of SaaS systems, sophisticated business solutions, engineering applications, and business processes automation software.

    We are excited to announce that  our analytics expertise has earned us a 2019 Clutch Eastern Europe award as a leading IT and business services provider in Ukraine.

    “Being awarded by Clutch is a big honor for our company. For years we recognize Clutch to be the best way to reach the right audience and get exceptional long-term partnerships,” – CEO, SoftElegance.

    Continue reading

    Report from Interpipe Industry 4.0 conference 2018

    Reading Time: 1 minute

    We have some last Industry 4.0 news!

    5-6 July there was a conference “Technologies 4.0 in the industry” in Dnipro on Interpipe location. Representatives from leading companies were discussing industry issues and innovations.

    Yaroslav Nedashkovsky and Volodymyr Vargola on Interpipe Industry 4.0 conference

    Yaroslav Nedashkovsky and Volodymyr Vargola on Interpipe Industry 4.0 conference


    SoftEleganse was represented by Data Architect Yaroslav Nedashkovsky and Business Development Director Volodymyr Vargola.
    Continue reading

    SoftElegance celebrates 25th anniversary of the company!

    Reading Time: 1 minute

    Great celebration! 25 years on the market

    Actually, we want to share with you our joy.
    The previous weekend we had company’s anniversary and us proud that our outsourcing ship on market waves already 25 years!
    Of course, we’d celebrate this date grandiosely. It’s summer now, so our little company’s family ridden out on the barbecue in the countryside.

    Rewards for 25th anniversary SoftElegance

    Rewards for SoftElegance’s employees


    We had bow and arbalest shooting range, volleyball and beach sitting bags. Some of us even opened a swimming season!
    And of course after all activities, we had catering and all employees got rewards for their years of work.

    Work hard, play hard!
    For more updates please follow us on Twitter, Facebook and LinkedIn.

    SoftElegance became a member of APPAU

    Reading Time: 1 minute

    Now we’re part of Industry 4.0 Ukraine. Hurray!

    Not so long ago SoftElegance became APPAU member.
    For us, it’s an honour to be part of such innovative direction as Industry 4.0 in Ukraine.

    Industry 4.0 in Ukraine

    Industry life in Kyiv, Ukraine


    We proud to be part of Ukrainian Industry 4.0 society because it’s a new significant part of future in general. And our European and American partners too close to this future. We have to join forces for going hand in hand with world leaders and direct this power to new technologies development.
    Continue reading

    Why Businesses Need Artificial Intelligence For Marketing

    Reading Time: 1 minute

    Guest post about relationship between Marketing and AI

    Hi all!
    We want to share a new food for thought. And today it’s a guest post from the content team at websitesthatsell.com.au.
    So here’re a little introduction and infographic:

    “Whenever we talk about technology we are referring to something that is always in a state of flux and change. Therefore we might have come across technologies only in books and other such imaginary situations. However, the same are no longer imaginary but are slowly becoming real. Today the buzzword in technology is artificial intelligence or AI as many people would like to call it. It is being used selectively in some areas. However, over the years it is quite likely to have much wider acceptance and become important as far as businesses, trade and commerce are concerned. We are sharing some useful information which we are sure will help you to understand AI a bit more.”

    Why

    For more updates please follow us on Twitter, Facebook and LinkedIn.

    AI & Big Data Day 2018 and Cybersecurity case study

    Reading Time: 1 minute

    Yaroslav Nedashkovsky presented Cybersecurity case study

    On 10 March there was AI & Big Data Day. It’s one of the biggest innovation technologies event in western Ukraine.

    AI & Big Data Day 2018

    Yaroslav Nedashkovsky on AI & Big Data Day 2018


    40 speakers and more than 400 participants: data scientists, data engineers, developers, startup entrepreneurs, investors and stakeholders from Ukraine, Germany, the Czech Republic, Switzerland, the USA, Canada and Eastern Europe.
    Continue reading

    European Industry Day in Kyiv

    Reading Time: 1 minute

    EU Industry Day — it’s unique annual forum for the first time in Ukraine

    21 February was held European Industry Day in Kyiv. The main forums themes were Industry 4.0 and Internet Of Things.

    Volodymyr Vargola on EID in Kyiv

    Volodymyr Vargola on European Industry Day in Kyiv

    Continue reading

    SoftElegance one of trusted Ukrainian software development company!

    Reading Time: 1 minute

    We became the 2nd company in Ukraine on TOP trusted outsourcing companies

    Not so far ago we received a message from Jigar Modi that published companies rating. On his blog, you can find everything from blockchain to outsourcing leading companies.

    Negotiations

    Business relationships in IT


    We became the 2nd company in Ukraine on TOP trusted outsourcing companies. For us, it’s a big pleasure and responsibility too!

    Continue reading

    « Older posts

    © 1993—2021 SoftElegance's Blog

    Theme by Anders NorenUp ↑