Datactics, a leader in data quality software has achieved ISO 27001:2022 Certification for Information Security Management System.

The ISO 27001 certification is recognised globally as a benchmark for managing information security. The rigorous certification process, conducted by NQA and Vertical Structure, involved an extensive evaluation of Datactics’ security policies, procedures, people, and controls. Achieving this certification demonstrates Datactics’ dedication to safeguarding client data and maintaining information assets’ integrity, confidentiality, and availability.

Victoria Wallace, Senior DevOps & Security Specialist, stated: “Security is at the heart of everything that Datactics does and achieving ISO 27001:2022 certification is a testament to the team’s unwavering commitment in this technical field. Showcasing the extensive work that went into this prestigious achievement proves that dedication and determination can lead to significant success, both within Datactics and across our client ecosystem. Achieving and maintaining this certification is a key part of Datactics’ progress in enhancing our secure, process-driven, and powerful data quality platform.”

Tom Shields, Cyber & Information Security Consultant at Vertical Structure, said “It was a pleasure working with the team at Datactics. Their enthusiastic approach to ISO 27001 Information Security and the associated business risk mitigation was evident in every interaction. Involvement from top to bottom was prioritised from day one, allowing us to integrate into their team from the very outset. The opportunity to guide such organisations in certifying to ISO 27001 is a privilege for us, and we look forward to continuing to work alongside their team in the future.

About ISO 27001:2022 Certification

Datactics’ accreditation has been issued by NQA, a leading global independently accredited certification body. NQA has provided assessments (audits) of organisations to various management system standards since 1988.

Founded in 2006, Vertical Structure is an independent cyber security consultancy with a ‘people-first’ approach. Vertical Structure specialises in providing people-focused security and penetration testing services for web applications, cloud infrastructure and mobile applications.

Vertical Structure also conducts technical security training, helping companies to achieve certification to international standards such as ISO 27001, Cyber Essentials and CAIQ and are proud to be an Amazon Web Services® Select Consulting Partner.

The post ISO 27001:2022 Certification Success appeared first on Datactics.

A data quality firewall is a key component of data management. It is a form of data quality monitoring, using software to prevent the ingestion of messy or bad data.

It’s a set of measures or processes to ensure the integrity, accuracy, and reliability of data within an organisation, and helps support data governance strategies. This could involve controls and checks to prevent the entry of inaccurate or incomplete data from data sources into data stores, as well as mechanisms to identify and rectify any data quality issues that arise. 

In its simplest form, a data quality firewall could be data stewards manually checking the data. However, this isn’t recommended, as it’s considerably inefficient and could cause inaccuracies.  Instead, a more effective approach is the use of automation.

An automated approach

Data quality metrics (e.g. completeness, duplication, validity etc.) can be generated automatically and are useful for identifying a data quality issue. At Datactics, with our expertise in AI-augmented data quality, we understand that the most value is derived from data quality rules that are highly specific to an organisation’s context. This includes rules focusing on Accuracy, Consistency, Duplication, and Validity. The ability to execute all the above rules should be a part of any data quality firewall. 

The above is perfectly suited to an API giving an on-demand view of the data’s health before ingestion into the warehouse. This real-time assessment ensures that only clean, high-quality data is stored, significantly reducing downstream errors and inefficiencies.

What Features are Required for a Data Quality Firewall? 

The ability to define Data Quality Requirements 

The ability to specify what data quality means for your organisation is key. For example, you may want to consider whether data should be processed in situ or passed through an API, depending on data volumes and other factors. Here are a couple of other questions worth considering when defining data quality requirements- 

• Which rules should be applied to the data?  It goes without saying that not all data is the same. Rules which are highly applicable to the specific business context will be more useful than a generic completeness rule, for example. This may involve checking data types, ranges, and formats, or validation against sources of truth. Reject data that doesn’t meet the specified criteria.
• What should be done with broken data? Strategies for dealing with broken data should be flexible. Options might include quarantining the entire dataset, isolating only the problematic records, passing all data with flagged issues, or immediately correcting issues, like removing duplicates or standardising formats. All the above should be options for the user of the API.  The point is, not every use case is the same and a one-size-fits-all solution won’t be sufficient. 

Key DQ Firewall Features:

Data Enrichment 

Data enrichment may involve adding identifiers and codes to the data entering the warehouse. This can help with usability and traceability. 

Logging and Auditing 

Robust logging and auditing mechanisms should be provided. Log all incoming and outgoing data, errors, and any data quality-related issues. This information can be valuable for troubleshooting and monitoring data quality over time. 

Error Handling 

A comprehensive error-handling strategy should be provided, with clearly defined error codes and messages to communicate issues with consumers of the API. Guidance on how to resolve or address data quality errors is provided. 

Reporting 

Regular reporting on data quality metrics and issues, including trend analysis, helps in keeping track of the data quality over time.

Documentation 

The API documentation should include information about data quality expectations, supported endpoints, request and response formats, and any specific data quality-related considerations. 

How Datactics can help 

You might have noticed that the concept of a Data Quality Firewall is not just limited to data entering an organisation. It’s equally valuable at any point in the data migration process, ensuring quality as data travels within an organisation. Wouldn’t it be nice to know the quality of your data is assured as it flows through your organisation?

Datactics can help with this. Our Augmented Data Quality (ADQ) solution uses AI and machine learning to streamline the process, providing advanced data profiling, outlier detection, and automated rule suggestions. Find out more about our ADQ platform here.

The post What is a Data Quality Firewall?  appeared first on Datactics.

Firstly, what is a support desk?

Most people have a support desk at their company and know how to make use of it. For those who don’t, ITIL’s (Information Technology Infrastructure Library) definition is helpful for explaining it. It states that a support desk (also commonly known as a service desk) is “the single point of contact between the service provider and the users. A typical service desk manages incidents and service requests and also handles communication with the users”. So, a support desk is a means for end users in a company to gain the help and support of the IT support staff. The benefits of a well-managed service desk are far-reaching, from enhancing employee productivity to safeguarding critical business processes.

Typical uses of a support desk are:

• Unlocking an account
• Printer errors (as always)
• Requests for access to resources

In Datactics, the DevOps team is kept busy by the support desk. Whilst every day brings with it new requests, there are two which occur the most often:

1. Password changes/ account lockout: These are definitely the most popular. Recently our LDAP service provided a way for users to reset passwords when locked out by themselves, so we get fewer of these requests now than we used to but it is still quite a popular request!
2. Next up is admin access. Security is paramount to any company so we, like most others, work on the principle that users need given permission to do certain tasks (like downloading software on their laptops). As a result of this, we get a lot of requests for admin access on our support desk.

At the beginning of 2023, the DevOps team at Datactics set out a roadmap for the year ahead. One of the tasks in the roadmap was to move the support desk onto JIRA, as the current solution was nearing the end of its contract. Whilst it can be easy to think, ‘Why can we not just allow staff to message us, email their requests, or ask in person?’, the process of moving the support desk to JIRA shone a light on why it is so crucial to an organisation.

Single Point of Contact

There are a few reasons why, the most obvious being from ITIL’s definition- ‘the single point of contact’. The support desk provides one place where the staff knows where to go for help and where the team providing the help will check in regularly. This simplifies communication within the organisation and ensures that issues are properly tracked and managed. When end-users send requests on many different platforms it creates multiple places of contact, which can be confusing and difficult to keep track of. Imagine 60+ people sending you messages on different platforms, all requiring support in one form or another; like spinning plates. It can drive you crazy. Having one place to see all the tasks together helps prevent this (and helps keep us sane.)

Enable self-service

On top of this, support desks can allow for self-service. A support desk can host knowledge articles so that users can self-serve in fixing common problems e.g. resetting passwords.  Similarly, using HYCU as a backup solution allows end users to restore a deleted file or email without having to contact support. This saves time for the end user and also for the support staff- win-win.

Easily prioritise tasks

A support desk also allows for the prioritisation of tasks. When all of the support tickets are raised in one place, it allows the team to compare them to each other and determine which ones can wait until the urgent ones are completed. If a message comes in with an urgent request, but the team is already working on one, the message could be quickly forgotten about until it is too late. They prioritize and allocate resources based on the criticality of issues, ensuring that the most pressing matters receive immediate attention.

Streamlined issue resolution

Moreover, a support desk reminds the team of all of their pending tasks in one place. Imagine having to trawl through all your messages, emails, and memories of conversations(!!!) to find all the pending tasks- it can cause a lot of headaches, or worse- leave some tasks unnoticed.

It can also track how quickly a team can complete a task and how much time is spent on it; helpful for organisations calculating chargeable time. Without the feature of a support desk, it would be much more difficult to track the time taken to complete tasks.

Finally, support desks create a paper trail of work the support team completes. This helps keep the team accountable and allows for transparency. It is also easy to underestimate just how much is done by the support team, and an inaccurate representation of the volume of work could be given without using a support desk.

In conclusion, the service and support desk plays a pivotal role in the heart of any organisation’s IT infrastructure. It is the frontline that bridges the gap between technology and people, ensuring that technical issues are resolved and operations can run smoothly. Whether you’re an IT professional or a business leader, never underestimate the importance of a robust service and support desk for your organisation.

The post 4 reasons why your business needs a Support Desk appeared first on Datactics.

Year after year, the volume of data being generated is increasing at an unparalleled pace. For businesses, data is critical to inform business strategy, facilitate decision-making, and create opportunities for competitive advantage.

However, leveraging this data is only as good as its quality, and traditional methods for measuring and improving data quality are struggling to scale.

This is where Augmented Data Quality comes in. The term describes an approach that leverages automation to enable systems to learn from data and continually improve processes. Augmented data quality has led to the recent emergence of automated tools for monitoring and improving data quality. In this post, we’ll explain what exactly is augmented data quality, where it can be applied, and its positive impact on data management. 

Why Are Traditional Approaches Struggling? 

First, let’s set the scene. With an ever-growing reliance on data-driven decision-making, businesses are looking for ways to gain accurate insights, deep business intelligence, and maintain data integrity in an increasingly complex business environment.

However, measuring data quality is challenging for enterprises, due to the high volume, variety, and velocity of data. Enterprises grapple with ensuring the reliability of data that has originated from multiple sources in different formats, which can often lead to inconsistencies and duplication within the data.

The complexity of data quality management procedures, which involve data cleansing, integration, validation, and remediation, further increases the challenge. Traditionally, these have been manual tasks carried out by data stewards, and/or using a deterministic-based approach, both of which are not scalable as the volume and veracity of data grows.  Now, enterprises are turning to highly automated solutions to effectively handle vast amounts of data and accelerate their data management journey and overall data management strategy.

What Is Augmented Data Quality? 

Augmented Data Quality is an approach that implements advanced algorithms, machine learning (ML), and artificial intelligence (AI) to automate data quality management. The goal is to correct data, learn from this, and automatically adapt and improve its quality over time, making data assets more valuable. 

Augmented data quality promotes self-service data quality management, empowering business users to execute tasks without requiring deep technical expertise. Moreover, it offers many benefits, from improved data accuracy to increased efficiency, and reduced costs, making it a valuable asset for enterprises dealing with large volumes of data. 

Although AI/ML solutions can speed up routine DQ tasks, they cannot fully automate the whole process. In other words, augmented data quality does not eliminate the need for human oversight, decision-making, and intervention; instead, it complements it by leveraging human-in-the-loop technology, where human expertise is combined with advanced algorithms to ensure the highest levels of data accuracy and quality.

“Modern data quality solutions offer augmented data quality capabilities to disrupt how we solve data quality issues. This disruption – fueled by metadata, artificial intelligence/machine learning (Al/ML) and knowledge graphs – is progressing and bringing new practices through automation to simplify data quality processes.”

-Gartner^®, ‘The State of Data Quality Solutions: Augment, Automate and Simplify; By Melody Chien, Ankush Jain, 15 March 2022.

GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used herein with permission. All rights reserved.

How Can Augmented Data Quality Help A Data Quality Process?

Routine data quality tasks, such as profiling and rule building, can be time-consuming and error-prone. Fortunately, the emergence of augmented data quality has revolutionized the way routine data quality tasks as performed, reducing manual effort and saving time for users. Below are some examples of where automation can add value as part of a data quality process:

Data profiling and monitoring

ML algorithms excel at recognizing patterns. For example, ML can enhance a system’s capability to manage data quality proactively, by identifying and learning patterns in data errors and corrections. Using these learnings, ML can be applied to automate routine tasks like data cleaning, validation, and deduplication.

Data Deduplication

ML can be used to identify and remove duplicate entities. Rather than simply looking for exact matches, ML algorithms such as  natural language processing can identify duplicates even with minor variations, such as spelling mistakes or different formats.

Automated Validation

ML can be used to automate the data validation process. For a feature such as  automated rule suggestion, the system applies ML to understand underlying data and match relevant rules to this data. The process can be further enhanced by automatically deploying suggested rules using a human-in-the-loop approach, making the process faster and more efficient. 

Why Enterprises Are Embracing Augmented Data Quality

Augmented data quality is useful for any organization wanting to streamline its data quality management. Whether it’s for digital transformation or risk management, augmented data quality holds immense value. Here are a few examples of where our clients are seeing the value of augmented data quality:

Regulation and Compliance: Industries like healthcare and financial services are confronted with increasing regulatory changes. Yet, organizations often struggle to meet the demands of these regulations and must adapt quickly. By leveraging AI/ML methods to help identify data errors and ensure compliance with regulatory requirements, enterprises can efficiently minimize the potential risks associated with poor data quality. 
Use Cases: Single Customer View, Sanctions matching.

Business analytics: With complete, and consistent data, organizations can leverage analytics to generate accurate insights and gain a competitive edge in the market. Through AI/ML, data quality processes can be automated to quickly produce analytics and predict future trends within the data.
 Use Cases: Data preparation & Enrichment, Data & Analytics Governance.

Modern Data Strategy: Data quality is a foundational component of any modern data strategy, as data sources and business use cases expand. By leveraging augmented data quality within a modern data strategy, organizations can experience greater automation of manual processes, such as rule building and data profiling. 
Use Cases: Data Quality Monitoring & Remediation, Data Observability

Digital Transformation: Enterprise-wide digital transformation is taking place across all industries to generate more value from data assets. Automation plays a crucial role in enabling scalability, reducing costs, and optimizing efficiencies. 
Use Cases: Data Harmonization, Data Quality Firewall

Adopting augmented data quality within an organization represents a transformative step towards establishing a data-driven culture, where data becomes a trusted asset that drives innovation, growth, and success. The automation of process workflows reduces dependence on manual intervention, saving time and resources while enhancing efficiency and productivity. Moreover, augmented data quality increases accuracy, reliability, and compliance, enhancing customer experiences and improving an organization’s competitive advantage.

In conclusion, the seamless integration of augmented data quality within essential business areas offers significant benefits to organizations seeking to maximize the value of their data.

Find out more about Datactics Augmented Data Quality platform in the latest news from A-Team Data Management Insight.

The post What Is Augmented Data Quality And How Do You Use It? appeared first on Datactics.

Datactics has been awarded the Bronze Diversity Mark Accreditation for its efforts in championing Diversity, Equality and Inclusion (DEI) in the workplace. 

Diversity Mark, established by Women in Business NI, helps businesses across the UK and Ireland build more diverse and inclusive workplaces, through accreditation, insights, best practice and peer support. Evaluated by an independent assessment panel, Datactics has achieved Bronze Award, which marks the beginning of the organisation’s journey in advancing diversity and inclusion.

Datactics Head of People, Elspeth Flenley, said:

“Achieving this award hot on the heels of International Women’s Day feels befitting. We have long championed the role of women in the workplace, especially in technology companies, and securing this award demonstrates our longstanding commitment to a diverse workforce. It marks the start of a continual process of action and review and provides us with a solid framework to demonstrate progress against our DEI ambitions at Datactics.” 

Recognising Datactics’ ambitions, Nuala Murphy, Director at Diversity Mark said:

“We are delighted that Datactics has been awarded the Bronze Diversity Mark Accreditation for its commitment to advancing Equality, Diversity and Inclusion in the workplace. Our independent assessment panel applauded Datactics’ forward thinking attitude in relation to culture within the organisation. Huge congratulations to all the team on this robust achievement.”

About Datactics:

Datactics is an award-winning, Belfast-based tech company with a team of over 70, providing AI-augmented data quality and matching software to a growing number of international clients across industries. Solutions are data-agnostic and offer interoperability with data lineage, governance, and metadata management tools, especially critical in the deployment of data fabric and data mesh architectures. Datactics received Investors in People Gold accreditation in March 2022, having propelled from Bronze. Twelve months later, they secured additional recognition in the Diversity Mark Bronze Award.

About Diversity Mark:

The Diversity Mark was the first accreditation of its kind and was developed with the support of businesses that were committed to building more diverse and inclusive workplaces. The accreditation was officially launched in September 2017. Within 3 years, over 80 progressive companies from all business sectors had signed the Charter and are committed to transforming workplace culture and attitudes for good.

Diversity Mark offers accreditation, expert support, and guidance to organisations that are navigating the Diversity and Inclusion (D&I) landscape. It is the only accreditation where award applications and annual reports are assessed independently by assessors. Their assessment process is highly robust and, as such, is held in high regard as a mark of progress for DEI. 

For more from Datactics, find us on Linkedin, Twitter or Facebook.

The post Datactics achieves Diversity Mark Award for Diversity and Inclusion   appeared first on Datactics.

Every math enthusiast’s favorite day of the year has arrived – Pi day. Celebrated on 14^th March every year, it represents the first 3 digits of Pi – 3.14 (in the American date format). As it happens, it is also the same date as Albert Einstein’s birthday!

Senior Software Developer, Edele Copeland, explains why she loves Pi and shares a few ways to celebrate Pi Day 2023, including how to find your birthday in Pi .

What is Pi?

Represented by the Greek letter, π, most people would have come across this magical number in secondary school, where it would have been taught as part of calculating the area of a circle or the volume of a sphere. At school, Pi was just a large, irrational number that existed; a symbol on your calculator that can be represented by the number 3.14. But there is an actual meaning behind where it originates from.

If you were to draw a perfect circle using a compass, you could place a string on top of the circle, going around exactly once. When straightening out the string, the length would represent the circumference of the circle (hopefully some GCSE Maths knowledge is coming back?), which can then be measured using a ruler. 

Next, you would measure the diameter of the circle (i.e. any line that goes through the center of the circle – where the compass would have been placed – from one point on the circle to an opposite end). Using these two measurements, if you divide the circumference by the diameter, you would get an answer around 3.141592653589793238… or π. This will always be the case, no matter how big or small you draw the circle.

Where is it used?

Most commonly, Pi is used in calculations involving circles, spheres, and ellipses – this is what most people would have learned in school. But it also has essential use in trigonometry, defining infinite series, and Calculus.

Pi is used in real-world applications every day. To name a few, it is used to track statistical data like population dynamics, in GPS navigation, and in biochemistry to understand DNA structures. It can also be used to test how powerful a computer is, as the calculation of Pi can determine how accurately the computer’s hardware and software are working and whether changes need to be made. 

Why do I celebrate Pi day?

At school, maths was something I really enjoyed. I liked that there are so many real-world applications of maths, and applying logical thinking to work out a problem was what I was good at (unlike writing an essay, maths had one right answer, and there was no subjectiveness about it!).

Whilst studying for my A-Levels, I decided to do maths and further maths. One teacher, in particular, was a great influence on how my passion for maths grew, encouraging me to continue the subject into university.

During my four-year university course, I studied a mix of Pure, Statistical Maths, and Applied Maths. Within these 3 areas, Pi was always creeping into formulas, whether it was part of Number Theory, Classical or Fluid mechanics, or modeling within biology and medicine. It was used everywhere during my studies and in real life. Even during my final year, I was constantly using Pi within formulas and computations to analyze and prove my dissertation hypothesis.

Pi remains an important constant within maths and deserves recognition in the form of its own day!

Fun facts to help you celebrate Pi Day

1. Physicist Larry Shaw started celebrating March 14th as Pi day at San Francisco’s Exploratorium science museum. There he is known as the Prince of Pi.
2. It’s actually part of Egyptian mythology – people believe the Pyramids of Giza were built on the principles of Pi.
3. Pi was used in the famous O.J. Simpson trial, where an FBI agent’s findings in the case were discarded due to his wrongful use of Pi.
4. The Greek letter π is the first letter of the word periphery and perimeter. Pi is the ratio of a circle’s “periphery” (i.e. its circumference) to its diameter.
5. Even today, people are still trying to calculate more digits of Pi. In 2010, a Japanese engineer and an American computer wizard broke the record for the most number of Pi digits by calculating up to 5 trillion digits of Pi.

There are lots of ways to get involved with Pi Day, including testing yourself with this Pi Quiz and, of course, finding your birthday in Pi using this calculator– mine lies at position 46,377.

Happy Pi Day everyone!

The post How to find your birthday in Pi appeared first on Datactics.

Datactics has won ‘Best Data Quality Analysis Tool’ at A-Team Group’s Data Management Insight Awards 2022 for its innovative Self Service Data Quality (SSDQ) platform. The awards, now in their tenth year, recognise leading providers of data management solutions, services and consultancy to capital markets participants within Europe. This is the fourth consecutive year that Datactics has been recognised for its business user focused technology, as voted for by their customers.  

Stuart Harvey, CEO at Datactics, commented that, 

“We’re really pleased that we have received recognition for being an industry-leading data quality solution provider. Winning this award reflects our ongoing investment in our AI driven capabilities in order to provide our customers with greater automation of data quality tasks, including data cleansing, profiling and matching.  

“It’s brilliant to be nominated amongst best-in-class data quality solutions and we’d like to thank all those who voted for us. We look forward to further developing our Self Service Data Quality product offering in 2023 as an indispensable partner to our clients’ data quality ambitions.” 

Angela Wilbraham, CEO at A-Team Group and host of the Data Management Insight Awards Europe 2022, commented: “These awards recognise leading providers of data management solutions, services and consultancy to capital markets participants in Europe.  Many congratulations to Datactics for winning Best Data Quality Analysis Tool, they should be deservedly proud of their achievement in a closely fought and highly competitive contest.” 

About Datactics:  

Datactics provides business user-focused, no-code data quality and matching tools helping financial firms gain value from their data and reduce regulatory risk. Its award-winning platform integrates with multiple data sources, governance, and lineage systems with intelligent automation. It allows Chief Data Officers and senior data leaders to measure, report and fix their data, and match across multiple internal and external sources and systems. Using AI, the platform significantly reduces the manual effort required to make decisions, with full transparency. For more information or to book a demo please contact us.  

For more information on how Datactics’ Self-Service Data Quality platform can add value to your business please contact Kieran Seaward.  

And for more from Datactics, find us on Linkedin, Twitter or Facebook. 

The post Datactics wins ‘Best Data Quality Analysis Tool’ at the Data Management Insight Awards Europe 2022 appeared first on Datactics.

Datactics provides AI-augmented self-service data quality and matching software, empowering CDOs, CIOs and data leaders to rapidly measure, match, report and fix data assets. Solutions are data-agnostic and offer interoperability with data lineage, governance and metadata management tools, especially critical in the deployment of data fabric and data mesh architectures.  

According to Gartner, “Magic Quadrant reports are a culmination of rigorous, fact-based research in specific markets, providing a wide-angle view of the relative positions of the providers in markets where growth is high and provider differentiation is distinct. Providers are positioned into four quadrants: Leaders, Challengers, Visionaries and Niche Players. The research enables you to get the most from market analysis in alignment with your unique business and technology needs.”

Stuart Harvey, CEO comments that,

“We’re delighted to be recognised in Gartner Magic Quadrant for Data Quality Solutions 2022. In the last year, we’ve invested in expanding our development and ML teams, which has enabled us to enhance our product roadmap with AI driven capabilities and provide further automation for our customers.

At Datactics, we’re proud to be a ‘best of breed’ solution. However, we recognise the need to be interoperable with other data management tools and therefore have prioritised data integration and strategic partnerships with other leading lineage and catalog vendors as a key element of our delivery process in the last twelve months. Through an abundance of connectivity options available in the Datactics platform, we can integrate with existing systems and vendors in order to make it easier for businesses to establish a sustainable Data Fabric and deliver ready-to-use data throughout the enterprise.”

Download a complimentary copy of the Magic Quadrant report to learn more about Datactics Self-Service Data Quality platform here.

Gartner disclaimer

Gartner, Magic Quadrant for Data Quality Solutions, Ankush Jain, Melody Chien, 1 November 2022.

GARTNER and MAGIC QUADRANT are registered trademarks and service marks of Gartner, Inc. and/or its affiliates in the U.S. and internationally and are used herein with permission. All rights reserved.

Gartner does not endorse any vendor, product or service depicted in our research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

About Datactics

Datactics, a leading and innovative provider of solutions in the data quality solutions market is headquartered in Belfast and offers expertise worldwide. Learn more information at https://www.datactics.com/.

The post Datactics placed in 2022 Gartner® Magic Quadrant™ for Data Quality Solutions appeared first on Datactics.

Datactics, an award-winning self-service data quality and matching software vendor based in Belfast, UK, and MANTA , a world class data lineage platform, are today announcing a new technology partnership.

The integration with MANTA will see clients able to leverage the combined capabilities of the two firms to visualise data quality in-motion, without being tied to one single enterprise data governance and quality platform. This provides more options to firms exploring their data governance strategies, seeking to make use of ‘best of breed’ technology that aligns with their business operational models and user profiles, especially in the context of data in motion.

MANTA’s unique automated lineage capabilities enables enterprises to achieve a complete and comprehensive view of their data. The award-winning data lineage platform automatically scans an organisation’s data environment to build a map of the data ‘in-motion’, as it flows between various sources and transformations, displaying a holistic view of the data to both technical and non-technical users.

Datactics’ customers will benefit from the new partnership in addressing complexities around data in motion, by leveraging MANTA’s  lineage capabilities alongside Datactics data quality and matching expertise through their Self-Service Data Quality platform.

Datactics CEO, Stuart Harvey commented,

“As data management requires advanced capabilities at every turn, business users are finding that they need a reliable overview of their data as it flows through their organisation, from where it started to where it ended. This integration with MANTA’s data lineage platform will enable our customers to gain increased visibility around their enterprise data. By combining Datactics’ Self-Service Data Quality, a platform that identifies broken critical data elements and returns them to a data steward for fixing or automatic correction, and MANTA’s data lineage visualisation, the client will be empowered to make better decisions, prioritise and fix key data quality issues, and address the negative impact of poor data quality.”

Stuart Harvey

MANTA representative, Ernie Ostic, SVP of Product  said,

“With this partnership, Datactics and MANTA will increase visibility into what happens to data as it moves through the enterprise. We look forward to working together to provide customers with the ability to measure data quality in the context of their data pipelines and thus establish more accurate priorities for application of their data quality rules.”

Ernie Ostic

For more information, get in touch with Datactics or MANTA today.

The post Datactics partners with MANTA to Deliver Advanced Data Quality and Lineage Capabilities    appeared first on Datactics.

Datactics won the A-Team Innovation award for the creation of Rapid Match, a system which allowed financial analysts to understand the geographical allocation of financial loans made by UK government during the 2020/21 Covid crisis augmented with a breakdown of industry type.   

Datactics was selected and funded by Innovate UK to investigate the use of its self-service platform to solve a challenge for information analysts: How does an organization reduce time in wrangling data to make it fit for good quality analytics? The firm has reported that it has been regularly asked this question by clients spending too much time on manual preparation of data and seeing little return on investment (ROI) for their efforts. 

The Datactics solution focused on challenges associated with data preparation when an organisation seeks to create rapid data analytics but is faced with disparate data sources, (often stored in different formats), siloed information and poor underlying data quality.   

Summary of business benefits of the Datactics solution: 

• Addresses data quality and matching at scale challenges associated with joining large amounts of messy, incomplete data in varying formats, from a multiple sources​. 
• Provide a reliable ‘match engine’ allowing government and organisations to accurately and securely integrate diverse sources of data. 
• Automated and reproducible platform to ingest, cleanse and match and update datasets for downstream analysis. 
• Provides systematic, reproducible pipelines to address time consuming data quality, preparation and matching tasks to produce complete, high quality and timely data for decision making.

Datactics’ Head of Software Development, Dr Fiona Browne, who led the project said, 

“We were very aware that Companies House information is used in almost every KYC/AML system in the UK and we wanted to develop automated techniques for improving the quality of this information and to make it easy to ingest and to query. I’m glad to say that we achieved this ambition through building systematic, reproducible data quality pipelines that address time consuming tasks such as data wrangling and matching. We are augmenting our early work by looking at applications of machine learning and network analysis in this space for downstream tasks such as fraud detection and onboarding which are underpinned by data quality. ”  

Read more from Dr Browne on how Datactics demonstrates rapid matching capabilities on open datasets.

In detail: How it works 

The system extracted information from multiple financial and geographical data sources and used automated data pipelines for preparing, scrubbing and matching multiple data types before presenting the data in a very easy to query dashboard. These pipelines are systematic and re-producible enabling the process to be re-run when datasets are updated, or new datasets added.  

The process is transparent for auditability. Furthermore, the process provides a reliable ‘match engine’ allowing organisations to accurately and securely integrate diverse sources of data. A particular focus of the project was on using UK Companies House as a master source relating to company name/address validation and understanding the relationship between companies and their owners. A view of the resulting analytical dashboard is presented in Figure 1 below.   

Figure 1: Analytics Dashboard Screen for Rapid Match illustrating the breakdown of loans across industries in England  

Stuart Harvey, CEO of Datactics commented:

“Ultimately we wanted to prove that it was possible to create an easy-to-use solution in which a data team can address time-consuming data quality, preparation and matching tasks in order to create complete, high quality and timely data for decision making.

Rapid Match took advantage of existing strengths in the Datactics’ platform and added benefits from machine learning. We addressed data quality and matching at scale – joining large sets of messy data in varying formats from multiple sources. We implemented a reliable ‘match engine’ which allowed for fuzzy matching from these diverse sources. We made use of data op’s automation to ingest, clean and match data with minimum human input.

A key part of the challenge in building Rapid Match was seamless integration with Companies House information as a trusted data source. Companies House data is used in almost every KYC and AML process involving UK entities. It contains over 4 million companies updating information on 500,000 of these per year. Since it’s a fairly raw source of open data and many end users have significant challenges with data quality. The Datactics’ solution provided easy access to clean Companies House data via REST API and we used sophisticated network analysis to understand the relationship between companies and their owners.

We are grateful to Innovate UK for their support of this work and we’d welcome further opportunities to collaborate with clients or partners if our recent work in this area is of interest to you”.

Angela Wilbraham, CEO of the A-Team Group, who hosted the A-Team Innovation Awards 2022, commented, 

“Our A-Team Innovation Awards 2022 celebrate and reward those companies at the forefront of innovation within our industry. We congratulate Datactics in winning the Most innovative data quality initiative award in recognition of their excellence in driving forward progress in capital markets capabilities.”  

To read more about the A-Team’s Innovation awards and view the full report of worthy winners, please take a look here.

The post Datactics wins Most Innovative Data Quality Initiative at the A-Team awards appeared first on Datactics.

Open datasets consist of structured data that is machine readable and typically used by individuals, financial services and public sector bodies. The Open Data Handbook provides the following definition: 

“Open data is data that can be freely used, reused and redistributed by anyone – subject only, at most, to the requirement to attribute and sharealike.” 

The uses for open datasets are innumerable- from economic indicators, business information and public health statistics. Similarly, the advantages of open datasets are widely acknowledged, including greater transparency and accountability from our public bodies. 

However, open datasets come with caution around data quality. Due to the scale of the datasets, and the lack of sensitive information included, there is unlikely to be a great deal of maintenance and data quality control taking place before they are released. In order to tackle the issues around accuracy, usability and quality of open datasets, advancing AI technologies are taking the opportunity to provide their solutions to the problem.  

In order to extract valuable insights from open datasets (i.e. data that is accessible by the public) sufficient tooling is needed to ensure the data is of high quality. In this blog we look at 4 features of a data quality tool. We have used Companies House – the UKs open register of companies as a Use Case to illustrate the impact of standardising the data quality and preparation process for datasets. 

Companies House is an open register sponsored by the UK Government’s Department for Business, Energy & Industrial Strategy. Here, company information is registered and made available to the public. Details can include company addresses, persons of significant control and disqualified directors, as well as insolvency information which are often entered as free form text. As such, entities using these data often undertake data quality and preparation work before using this information for downstream tasks.   

It’s estimated that millions of professionals rely on Companies House data every day, with real-world applications including KYC onboarding and AML processes. In these applications, public data becomes critical as Companies House data is matched against financial crime and terrorism lists. Therefore, data needs to be of high quality in order to prevent fraudulent and potentially dangerous activity.   

We summarise below in the Companies House Use Case how Datactics developed systematic data quality and matching pipelines which were run against these data. By using automated pipelines, users save time on designing and maintaining the programmes needed to run against the open data. Moreover, the pipelines have been developed by data experts in order to achieve high quality data without needing tech expertise.   

What is data profiling and why do we need it?  

Data profiling is the process of analysing, inspecting and creating helpful summaries of big data. Through running a diagnosis of the data, including its sources and metadata, bad data can be detected and amended before becoming actionable intelligence.  

Data profiling is an important first step in improving data sets. For organisations like financial services or public sector bodies with vast citizen datasets, profiling can detect where there are spelling mistakes or a lack of standardisation. The downstream benefits of this is that organisations are left with accurate citizen data. Without profiling, data quality issues may creep into downstream analysis and result in poor business decision (and worse, potential fines). 

Application of Data Quality Profiling through Datactics Platform  

Data profiling is implemented by the Datactics tool using different techniques including frequency analysis through to co-occurance metrics. This information can be used to identify potential outliers in a dataset through to understanding the underling data and data quality rules that can be applied. In the Companies House Use Case, these open data sets were initially profiled to identify data quality issues before being remediated. This including profiling on names and country columns. Profiling was performed on forenames and surnames using string length (the length of a set of characters), unexpected characters and numerals to identify outliers, or data elements which didn’t conform to any common pattern within the data set.  

‘Regular expressions’, a method used for matching patterns in text, were run against the column to pick up on non-name words such as ‘to’, ‘the’ and ‘a’ and output an accurate list of names. 

For ‘country’ columns, Datactics engineers looked at identifying outliers such as misspellings and city names being included in country lists. This data profiling work resulted in an accurate list of countries ready to be cleansed and standardised.   

2. What is data cleansing?   

Data Cleansing refers to the process of removing and remediating inaccurate or incomplete data from a dataset. Data cleansing helps produce accurate data which are fed into downstream processes. For the Companies House Use Case, having accurate data plays a key role in financial services processes, including KYC and detection of financial fraud.  

Significant effort is required to cleanse and standardise datasets. Using the Datactics platform, this process can be automated and re-run resulting in efficiencies, transparency and auditability in the process.  

How is it used in Datactics data quality tool?  

The Datactics platform provides a self service approach in building data quality rules along with out of the box data cleansing rules which can be executed against datasets. For the Companies House Use Case, information in the Companies and Persons of Significant Control datasets, including company name and addresses, are typically input as free text. As such, this can result in mistakes such as misspelling which requires remediation before it can be used downstream.

Datactics data engineers developed data quality cleansing pipelines applied to the data to achieve high quality, standardised and updated data sets which could be used for downstream analysis. This included the substitution of special characters with safe alternatives to address potential issues such as SQL injections (where fraudsters or bad actors use SQL phrases such as maliciously delete records), along with the validation of countries and addresses.  

Data quality checks against the ‘identification.country’ registered column in the dataset identified over a quarter of a million instances (nearly half of the total dataset) where the name entered was not found within the standard list of the United Nations Geoscheme.   

3. What is data matching?  

Data matching is the process of comparing two data sets to identify similar matches and detect outliers. This is an integral data quality tool feature that calculates the probability that two similar entities are in fact the same, helping organisations achieve better data quality and identify potentially fraudulent activity. 

Common types of data matching include ‘exact’ and ‘fuzzy’ matching. Exact matching refers to the process of finding all of the entries in a dataset that are clearly identical. Fuzzy matching works in slightly less clear circumstances, where it may have to account for errors e.g. typos or abbreviations to determine whether entities are close enough to be a match.   

How is it used in Datactics data quality tool?   

For the Companies House Use Case, matching was undertaken using built in functionality within the Datactics platform. The example in this Use Case was using matching to verifiy countries entered in the country column and match them to the United Nations Geoscheme of countries and subregions. These matched countries were standardised to the list of the Geoscheme country names. Entries in the country column were compared against this list using exact and fuzzy matching. 

Exact matching highlighted when a country name was completely identical to one from the Geoscheme list. In comparison, fuzzy matching identified similarities which could be considered a match for example, identifying countries that may have a spelling mistake. Data matching is useful for facilitating and merging diverse datasets at scale. As a result, users are left with a standardised dataset which makes downstream activities much easier.  

4. What is low-code?   

Low-code applications are easy to use and reproducible and can conduct data quality tasks such as cleansing and matching. They alleviate users from having to write coding scripts, such as Python, in order to achieve their business goals and manage tasks. Platforms typically feature a user-friendly GUI (Graphical User Interface), where non-technical users can select the data management components they need. Low-code platforms help users satisfy business needs and solve problems in a management workflow.  

How is a low-code interface used in Datactics data quality tool?  

Having an easy-to-use tool makes data quality projects simple, efficient, and reliable. Datactics developed data quality and matching pipelines which are automated, reproducible, and auditable. A systematic approach enables the consistent re-running of the data quality projects without the need for development, tuning or support. Developing an in-house solution may require the resources to update and maintain scripts weekly. 

Datactics developed a data quality tool feature with a low-code interface, to produce transparent data quality projects. As part of the Companies House project, 67 data quality and standardisation rules were developed and 16 were presented in a PowerBI dashboard. In this instance it was presented in a Power BI dashboard, however, templates can be built in most data visualisation tools.  

As the data quality pipelines are reproducible and auditable, the trend of data quality can also be tracked over time, with visualisations of Data Quality metrics by data source and by dimensionality. Moreover, it highlights where most of the DQ breaks occur, in terms of dimensionality and the total number of failing records. 

Summary  

The functions associated with a data quality tool offer significant benefits to the end user, whether that is a KYC officer wrangling with messy data during onboarding, or an AML professional needing accurate, up to date information to prevent financial crime.  

This report highlights a real-life, relevant use-case in dealing with open datasets namely the ‘Companies House’ and ‘Persons of Significant Control’ data. For these particular datasets, having these 4 features are interoperable so that end-users don’t have to manually extract, transform and load data (ETL) in order to be able to use it.  

In short, better data quality means better decision making and business insights.  

If you would like to explore this solution for open data, please reach out through our contact page.  

Roisin Floyd  

The post 4 Data Quality Tool Features for Improving Open Datasets appeared first on Datactics.

The National Digital Policing Strategy discusses some of the challenges the police are facing in terms of achieving digital transformation. Among them, legacy technology lock-in, conservative risk appetite and inconsistent understanding of data are listed as key barriers. In our experience, data quality monitoring and matching is key to achieving operational efficiency.

 ‘Policing in the UK remains world leading and sets the standard for law enforcement agencies across the world, however, our service is under pressure.’ (NPDS 2020-2030)

 The challenge of ‘inconsistent understanding of data’ struck a chord with us. Given the amount of growing pressures on forces to deliver exceptional services and ensure compliance with regulations, it is essential that data is interpretable. In other words, that the data is clean, accurate and available. Access to high quality data is crucial for efficient decision making and problem solving; having messy data in different systems and silos is worse than having no data at all! This blog will look at two recommendations for police and public sector to use their data as an asset and achieve operational excellence.

Data Monitoring

As the saying goes, you can’t manage what you can’t measure. Using data quality metrics, police and other organisations can monitor their data to ensure it is up to date and compliant. The six core data quality dimensions- completeness, coverage, conformity, consistency, accuracy, duplication, timeliness, are good benchmarks for evaluating your data quality efforts. Not only are these guiding principles useful for continually improving your data, they also help to safeguard against compliance risks. For example, the timeliness of data is inherent in the RRD process, whereas validity and completeness are key to GDPR compliance.

Using automation eases the burden of manual monitoring of data for police forces. Automation can be used to monitor data that is subject to simple accuracy checks (e.g. postcode formats) or more complex rules around gazetteer data, improving data quality by continuously monitoring for rule defying data.

Data stewards can then review the improvement or degradation of data quality over time through a dashboard, enabling better data governance and strategic decision making. Datactics Self Service Data Quality platform automatically benchmarks data against a set of pre-determined rules.

Reliable, real world information underpins every aspect of efficient operational policing. When faced with outdated systems, data can end up being pulled from multiple sources, resulting in inefficient decision making and increased data quality issues. Adopting the practice of data preparation and monitoring can equip forces with the skills needed to achieve long term data integrity. Ultimately, utilising data-driven technology is a key enabler for protecting the public they serve.

Data Matching for Single Citizen View

Policing (and public sector more generally) can improve their key operations by using data matching to tackle crimes related to citizen data.

Data matching compares one data set against another, in order to get a single record of a citizen or customer (also known as a 360 degree view). Within the financial services industry, correlating disparate data into a summary of a customer allows for better understanding of their behaviours and circumstances. Insights into customer experience can benefit their business decisions and KYC efforts, with a clearer understanding of a customer’s risk potential, or their likelihood to buy new products.

Opportunities for change within Public Sector

In the same way, public services can benefit from a lean manufacturing approach to gain greater insights into citizen data. A common obstacle within public sector is merging multiple data sets stored in legacy systems. Without greater data sharing, justice and emergency services can suffer from a loss of critical insights.

A data matching exercise tackles this by starting with powerful matching on large data sets and de-duplication logic, allowing for highly configurable fuzzy matching on information such as names and addresses. The result is a single golden record or a series of candidate records rated for how closely they match. However, previous metadata is not ignored, as this is still useful in understanding a citizen’s past.

The automated nature of data matching tools highlights errors for further investigation, reducing the risks associated with missed matches and making it a regulatory imperative for public sector services. Pairing this capability with data quality tooling ensures that the standard of data is up to the level that digital transformation requires.

Tackling hidden crimes

Data matching offers benefits within policing. In policing, data matching can address issues around data falsification, where a suspect provides inaccurate information to avoid detection e.g. name or address. Moreover, it allows for maximized regulatory reporting and improved predictive analytics; transforming data from a liability to an asset.

The National Fraud Initiative (NFI) is a good example of data matching happening right now. Fraud accounts for 40% of all crimes in the UK and is one of the most enduring threats to the public purse. As the Counter Fraud Function says ‘fraud is a hidden crime, in order to fight it you have to find it’. Operated by the Cabinet Office, the NFI’s sole purpose is detecting and preventing fraud on a large scale, preventing £245 million cases of fraud and error between 2018-2020.

Digitally enabled crimes are getting more sophisticated, as shown by the National Policing Digital Strategy. To mitigate these risks, technology-driven strategies are helpful with decision making and optimising resource deployment across public services.

Data matching provides a cost-efficient solution for operations in public services. It can help identify potential risks at an early stage and deliver vital services as efficiently as possible.

Through a combination of measuring and monitoring data, organisations can achieve better data quality and operational excellence. Public sector organisations can benefit from the Shingo model’s guiding principles, which is to say, continuous improvement and a systematic approach to improving company culture. The first steps of investigating your data governance framework can help you benefit from these principles.

Improved police data quality is a cornerstone of a data management strategy that can deliver both operational excellence and a more reliable, trusted public serving body.

Roisin researches and writes on data management for Datactics.

And for more from Datactics, find us on Linkedin, Twitter, or Facebook.

The post How to Improve Operations with Data Monitoring and Matching appeared first on Datactics.

On May 18th 2021, the UK Government published its response to the consultation on the National Data Strategy (NDS) and next steps for making the UK a world leading data economy. Outlined are five key priority ‘missions’ that underpin the delivery of the strategy, including a commitment to transforming government use of data, in order to improve public services.

The NDS makes recommendations for dealing with cultural challenges and technical barriers, often experienced by stakeholders when trying to achieve a standardised, consistent approach to data use. Within policing, issues around data availability and interoperability are not uncommon, as well as access to the right skills to respond to an increasing demand for transparency and accountability. The strategy proposes actions to improve data quality, availability and interoperability for those working in public sector.

The Office for National Statistics plays a key role here, acting as a leading delivery partner on a number of initiatives for the NDS. The DQ Hub, based at the ONS, provides ongoing support and training for stakeholders to ensure their data is fit for purpose, a key challenge identified by respondents to the NDS. Other policies mentioned include the Integrated Data Programme and Reference Data Management Framework, which allow for secure data sharing across government departments and public sector bodies to improve decision-making.

In line with these key policies, last month the ONS announced a new method for counting knife-enabled crimes in England and Wales. To improve data quality the Home Office, together with police forces, has developed the National Data Quality Improvement Service (NDQIS), a computer-assisted classification tool which reviews records involving a knife or sharp instrument.

NDQIS- a new methodology

NDQIS automates the process of classifying crimes through codifying human decision-making into rule sets based on Home Office classifications. As well as reducing their exposure to potentially distressing data, NDQIS offers data stewards quantifiable benefits in time saved reviewing crime records. Whilst human intervention is still required to review certain classifications, the automation means that detecting and fixing data quality checks is a much faster operation.

For respondents who called for more common data standards and governance performed via APIs, NDQIS offers a solution- greater standardisation allows for more consistent categorisation of crimes and improved comparability between forces. Over 90% of surveyed police forces expressed interest in further investment in cloud infrastructure and technology this year, indicating a need for flexible, secure data sharing. The enhanced infrastructure of NDQIS allows for secure cloud implementation for each force submitting data for regulatory reporting, whilst encrypting highly sensitive information from anyone not authorized to view the data.

Tech remedies for data stewards

The NDS comments that ‘data driven technologies are fundamental for improving public services’ and the NDQIS is just one example of how tools can help tackle the complex data challenges existing within public sector. Not only does it improve regulatory reporting, but good quality data also emboldens forces to make informed policy decisions on their deployment of resources; accurate data helps public services reach the communities most in need.

NDQIS is a key enabler for improving data quality related to knife crime, but with ever growing demands on our public services, the opportunities to embrace data-driven technologies are greater than ever.

The following blogs in this series will be looking at some of the typical data issues experienced by those working in the public sector. As an experienced data scientist will spend 80% of their time doing data preparation (cleaning, de-duplicating and matching), future blogs will look at efficient ways to ease the burden of data preparation, from data metrics and self-service data improvement, to data matching and migration. Read more about this in our Policing Data Quality E-Book here.

Roisin researches and writes on data management for Datactics, specialising in Govtech and Fintech. Holding a Master’s from Newcastle in Communication & Media, Roisin also covers PR for the company, and has a passion for the impact of music and arts on society.

The post Knife Crime – Using Technology to improve results at ONS appeared first on Datactics.