SlideShare a Scribd company logo
1 of 53
Download to read offline
Tackling Compliance When it Becomes
Your Biggest Performance Bottleneck
By Craig Saunders & Harsh Gautam
Classified as General
Premise
© Kindred Group plc 2022 2
“
When developing software in heavily regulated markets,
compliance is not an optional nicety. But compliance is every
developer’s nightmare when trying to write performant
software, especially when you need to communicate with
3rd party software or a Regulator’s services that can’t give
you the performance guarantees you require.
This talk’s premise is to talk through such challenges we
faced and how we used event-driven architecture and
asynchronous events to handle different latencies to ensure
that different market performance issues didn’t have system-
wide effects.
Classified as General
Our Opinions
© Kindred Group plc 2022 3
Classified as General
Who are we?
© Kindred Group plc 2022 4
Classified as General
Bio Cont… Craig Saunders
© Kindred Group plc 2022
01 Software Engineer with 26 years experience
02 Work in the strategic side of software development
03 Work closely with Architects, Stakeholders and POs
04 Worked with Microservices for about 6 years
01 Huge tech advocate – love playing with the toys
Classified as General
Bio Cont… Harsh Gautam
© Kindred Group plc 2022
01 Software Engineer with 14 years experience
02 Passionate about solving complex technical challenges
03 Background with distributed systems and microservices
04 Love developing and designing new systems
05 Passion for learning new technologies and keeping myself current
Classified as General
Bio Cont…
© Kindred Group plc 2022
Ultimately, We’re geeks who likes playing with the toys!!
Classified as General
Who is Kindred?
© Kindred Group plc 2022 8
Classified as General
Who is Kindred?
© Kindred Group plc 2022 9
A tech company that operates in the online gambling space
Classified as General
Tenet of Kindred’s Core Values
© Kindred Group plc 2022 10
Player sustainability and compliance is non-negotiable
Classified as General
© Kindred Group plc 2022 11
Kindred Core Values – Goes Beyond
Kindred is about making gambling 100% enjoyable
Classified as General
Set the Scene
© Kindred Group plc 2022 12
Classified as General
© Kindred Group plc 2022 13
Problem Domain?
01
Event-driven microservice landscape on Kubernetes
At Kindred we have adopted the event-driven and microservice paradigms for a long time. This
throws up may challenges with needing to be. fault-tolerant.
02
Zero downtime operation conditions
We are a 24/7 product meaning customers are constantly engaging with our product; hence
ensuring we have no occurrences of the site being down is critical. Deployments and migrations
strategies need to bear this in mind.
03
50 million wallet transactions per day
We have an average of 50 million transactions per day and can peak over 60 million; meaning we
have to scale to a lot of customer activity – especially around large tournament such the World
Cup.
04
Campaigns targeting upwards of 3 mil customers
At certain times in the year Marketing can do large campaign send-outs targeting millions of
customer; hence we need to scale to huge burst in traffic.
05
Real-time customer communication and engagement
Modern marketing strategy is personalised content delivered at the right time to the customer,
which can be onsite; hence execution speed is paramount. Couple this with large batch send-
outs, we need to ensure that batch send-outs don’t impact the marketing strategy.
Classified as General
Let’s Get Down to the Tech
© Kindred Group plc 2022 14
Classified as General
Platform Interactions
© Kindred Group plc 2022 15
Campaign Back-
office Platform
Audience
Integration
Platform
Communications
Platform
Player Account
Platform
Customer
Verification
Regulatory
Integration
Player
Sustainability
Execution
Platform
Classified as General
Execution Platform Raison D’Etre
© Kindred Group plc 2022 16
“
The execution platform is a decision agnostic platform that enable
safe, reliable and predictable execution of a marketing strategy to
offer rewards and/or send communications.
It enables 3rd party decision tools to be able to seamlessly
integrate with Kindred platform
Classified as General
Execution Platform
© Kindred Group plc 2022 17
Send-out
Execution
Marketing Compliance
Orchestration
Classified as General
© Kindred Group plc 2022 18
Execution Platform
Campaign
Send-out
Snapshot
ADAPTER
Campaign
Association
ADAPTER
Test Accounts
Adapter
ADAPTER
Campaign
Send-out
Executor
ADAPTER
Campaign
Customer
Details
ADAPTER
Campaign
Orchestrator
ADAPTER
Campaign
Scheduler
ADAPTER
Orchestration
Marketing
Compliance
ADAPTER
Marketing
Compliance
Profile
ADAPTER
Marketing Compliance
Campaign
Model
ADAPTER
Campaign
Back-office
Execution
Rewards
Notification
Service
ADAPTER
Classified as General
© Kindred Group plc 2022 19
Challenges
01 Required to communicate with regulator components with little guarantees
02 Different markets require more checks that others
03 Real-time customer engagement whilst the customer is on-site
04 Large batch-based send-outs targeting up to 3 millions customers
Classified as General
© Kindred Group plc 2022 20
Example
Half time when we need to target customer quickly for second half opportunities
Classified as General
Time to Cry?
The latency from
third parties
makes this
impossible
How do we know
we’ve achieve
success?
How’s this
going to be
possible?
20+ markets, how
can we guarantee
any consistency?
Classified as General
Impossible Task? No
© Kindred Group plc 2022 22
It’s just another challenge we developers dream about
Classified as General
So What Do We Do About It?
© Kindred Group plc 2022 23
Let’s Dive In
Classified as General
© Kindred Group plc 2022 24
Technique 1: Microservices Principles
01 Single responsibly for each component
02 Optimize for the responsibility E.G., connections, memory, CPU, etc
03 Reduce points of failure. Can we continue when a service is down?
04 Reduced resource usage means key services can have more instances
01 Required to communicate with regulator components with little guarantees
Classified as General
© Kindred Group plc 2022 25
Technique 1: Example
01 Heavy reads, low writes; hence can optimise for reading with caches
02 Low reads and caching means we can optimize for many connections
03 Because it’s standalone, it’s less likely to interfere with the Executor
04 If it falls over, it’s very quick to restart which reduces the impact
01 Smaller resource usage; hence easier to have more instances
Campaign Send-out
Snapshot
ADAPTER
Classified as General
© Kindred Group plc 2022 26
Focus Area for Potential Bottleneck
Campaign
Send-out
Executor
ADAPTER
Marketing
Compliance
ADAPTER
Marketing
Compliance
Profile
ADAPTER
Marketing Compliance Execution
Classified as General
Let’s Define the Terms
© Kindred Group plc 2022 27
Marketing Compliance Report Marketing Compliance Profile
How can we engage with a customer
in a compliant way.
Typical action are can be target with
rewards or what channel are allowed.
Can we engage with the customer at
all?
Aggregates all the data points that
are needed to build a compliance
report.
Continually monitors the other
systems to ensure the profile is up-
to-date.
Classified as General
© Kindred Group plc 2022 29
Component Responsibilities
Campaign
Send-out
Executor
ADAPTER
Execution
Marketing
Compliance
Profile
ADAPTER
Marketing
Compliance
ADAPTER
Marketing Compliance
Request a compliance report
Decides on what actions can be done based on the compliance report
Returns a compliance report based on the rules
Runs the compliance rules for the customer’s jurisdiction
Returns a valid compliance profile
Aggregates the compliance profile from a myriad of services
Classified as General
© Kindred Group plc 2022 30
Technique 2: Asynchronous Hand-offs
Used two queue for asynchronous communication to hand-off long running
queries
Hand-offs meant that slower queries were not holding up newer ones
The compliance profile was asynchronously updated in the background
Needed mechanism to know when the profile was consistent
Simple requests, where the profile was already complete, are faster
Classified as General
Technique 2 – Asynchronous Hand-offs
© Kindred Group plc 2022
ADAPTER
Compliance Profile
Retrieval
Update Compliance
Profile
Compliance Execution
ADAPTER
Regulatory
Engine
ADAPTER
Regulatory Integration
Rofus
ADAPTER
Third Party System
Compliance Report Request
Compliance Profile Request
Marketing Compliance Profile
Retrieve Batch of Customer to Check
Request a Batch of Customer to Check
Marketing Compliance Service
Execution Platform
Compliance Profile Response
Obtain Profile From the DB
Update Profile in the DB
Compliance Report Response
Request to Update Profile in the DB
Classified as General
Technique 2: Sequence
© Kindred Group plc 2022 32
Classified as General
© Kindred Group plc 2022 33
Is This Over?
Classified as General
© Kindred Group plc 2022 34
No!
How do we know we’ve succeeded?
Classified as General
© Kindred Group plc 2022 35
Inspiration from the Theory of Constraints
the constraint
In any journey, there’s only one bottleneck
Classified as General
© Kindred Group plc 2022 36
Inspiration from the Theory of Constraints
the constraint
the constraint
Everything else to the
constraint
Identify
Exploit
Subordinate
Elevate
Don’t stop now
the constraint
the constraint
everything else
to the constraint
the constraint
Find your next to
constraint
Classified as General
© Kindred Group plc 2022 37
Inspiration from the Theory of Constraints
Resolve
Protect
If you can’t reduce the
constraint, then protect it
Ensure efficiency in and out of
the constraint
The constraint will define your
throughput
Any improvement either side
of the bottleneck is pointless
Understand the bottleneck,
make the change and then
check
If the bottleneck has moved,
rinse-repeat with the new
location
Classified as General
Technique 3: Performance Monitoring
© Kindred Group plc 2022
01 Highlight where the bottleneck is in the stages of the send-out
02 Be able to slice and dice by campaign, market, and real-time vs batch
03 Understand throughput numbers by market
04 Understand where time is being spent that can affect end-to-end time
05 Understand common traffic patterns to replay in our PT environment
Classified as General
© Kindred Group plc 2022 39
Performance Monitoring Examples
Classified as General
© Kindred Group plc 2022 40
Performance Monitoring Examples
Classified as General
So Where is the Constraint?
© Kindred Group plc 2022
ADAPTER
Compliance Profile
Retrieval
Update Compliance
Profile
Regulatory
Engine
ADAPTER
Regulatory Integration
Compliance Profile Request
Marketing Compliance Profile
Retrieve Batch of Customer to Check
Request a Batch of Customers to Check
Execution Platform
Compliance Profile Response
Obtain Profile From the DB
Update Profile in the DB
Request to Update Profile in the DB
Classified as General
What Was the Issue?
© Kindred Group plc 2022 42
Batches of requests for 1,000
customers are sent every 5 seconds
per market
Scenario: 100,000 customers for one market
100,000 divided by 1,000 multiplied by 5 seconds = 8 minutes 40 seconds
Conclusion: The last few batches will have a natural 7 to 8 minutes delay
Classified as General
Time to Fix? Can it be Fix?
© Kindred Group plc 2022 43
Of course, we can
Classified as General
© Kindred Group plc 2022 44
Technique 4: Non-blocking Queue
Non-blocking
Blocking
Consumes customer
selections one-by-one and
stores in a DB
Every 5 seconds create a lock
Collates a batch and sends via
a message to the Regulatory
Integration component to
process
No batching, DB or locks
required
Reads the batch of customer
directly from the queue up to
the 1K limit
Leveraging the power of
Solace to poll and
acknowledge message
Classified as General
© Kindred Group plc 2022 45
Technique 4: Recap Sequence
Area of focus
Classified as General
Technique 4: Sequence
© Kindred Group plc 2022 46
Classified as General
What Did the Solution Bring?
© Kindred Group plc 2022 47
Batches of requests for 1,000
customers are sent continuously for
the queue per market
Scenario: 100,000 customers for one market
100,000 divided by 1,000 multiplied by 100 milliseconds = 10 seconds
Conclusion: Reduced 7 to 8 minutes delay to 10 seconds
Classified as General
© Kindred Group plc 2022 48
Technique 5: Only Request What You Need
Concept: Based on the campaign
strategy and the rules that need to be
executed for the respective market just
request the necessary sections of the
compliance profile
Advantage: Based on the campaign strategy and the
rules that need to be executed for the respective
Example 1: A newsletter campaign strategy doesn’t
require bonus block/opt-outs, etc.
Example 2: A campaign strategy that rewards a
customer without any communication doesn’t need
to check
Classified as General
© Kindred Group plc 2022 49
New Problem – Real-time vs Batch
Real-time needs to be processed with very low latency
Real-time send-outs are spread out over the day with occasional peaks
Batch targets large quantities of customers at the same time - up to 3 mil
Batch sends can block real-time by flooding the queue
Batch sends can tolerate a slow throughput through the system
Classified as General
© Kindred Group plc 2022 50
Technique 6: Bulkhead Pattern
“
The bulkhead pattern is based on a concept from building ships.
With a bulkhead design, a ship is divided into completely
segregated and watertight compartments called bulkheads.
Even if the ship’s hull is punctured, because the ship is divided
into watertight compartments (bulkheads), the bulkhead will
keep the water confined to the area of the ship where the
puncture occurred and prevent the entire ship from filling with
water and sinking.
Classified as General
Variant of the Bulkhead Pattern
© Kindred Group plc 2022 51
Campaign
Send-out
Executor
ADAPTER
Unica Adapter
(batch
sending)
ADAPTER
MMC (real-
time sending)
ADAPTER
Audience
Campaign
Feeder (batch
sending)
ADAPTER
Audience Integration
Platform
Execution Platform
Batch Queue
Real-time Queue
The real-time queue is a
constraint that needs to be
protected.
With the influence
bulkhead pattern, we can
build a second queue
dedicated to real-time;
hence large batch sends
affect real-time less.
Classified as General
Santa Claus’ Present to You
© Kindred Group plc 2022 52
https://microservices.io/patterns/microservices.html
Classified as General
© Kindred Group plc 2022 53
Keys to Success
01 Life throws up challenges that software can solve with the right mindset
02 Event-driven and asynchronous messaging is perfect to reduce delays
03
Performance monitoring in production is essential to understand if
you’ve solved the problem
04
You are never finished. Constant monitoring to assess the constraint is
important
05 These are challenges techies love to solve
Classified as General
© Kindred Group plc 2022 54

More Related Content

Similar to Tackling Compliance When it Becomes Your Biggest Performance Bottleneck.pdf

Partnership with Zettagrid Indonesia
Partnership with Zettagrid IndonesiaPartnership with Zettagrid Indonesia
Partnership with Zettagrid IndonesiaDolly Indra
 
Arx mindbusinessportfolio
Arx mindbusinessportfolioArx mindbusinessportfolio
Arx mindbusinessportfoliopankaj joshi
 
Cloudflare Partner Program 2020
Cloudflare Partner Program 2020Cloudflare Partner Program 2020
Cloudflare Partner Program 2020Dan Hollinger
 
Rate&Bill: 20 reasons to choose it
Rate&Bill: 20 reasons to choose itRate&Bill: 20 reasons to choose it
Rate&Bill: 20 reasons to choose ithosnihamdi2009
 
GTM Lab Data Slide Deck.pptx
GTM Lab Data Slide Deck.pptxGTM Lab Data Slide Deck.pptx
GTM Lab Data Slide Deck.pptxDemandbase
 
Align Your ITSM and SecOps Strategy for Unstoppable IT
Align Your ITSM and SecOps Strategy for Unstoppable ITAlign Your ITSM and SecOps Strategy for Unstoppable IT
Align Your ITSM and SecOps Strategy for Unstoppable ITSecPod
 
What is Hybrid Cloud and understanding what's right for your business?
What is Hybrid Cloud and understanding what's right for your business?What is Hybrid Cloud and understanding what's right for your business?
What is Hybrid Cloud and understanding what's right for your business?Emma Large
 
G-Cloud #AccreditCamp
G-Cloud #AccreditCamp G-Cloud #AccreditCamp
G-Cloud #AccreditCamp G-Cloud
 
Accredcamp
AccredcampAccredcamp
AccredcampG-Cloud
 
SCG Sales Tech Guide
SCG Sales Tech Guide SCG Sales Tech Guide
SCG Sales Tech Guide Robbie Burns
 
Democrance - InsurTech Innovation Award 2022
Democrance - InsurTech Innovation Award 2022Democrance - InsurTech Innovation Award 2022
Democrance - InsurTech Innovation Award 2022The Digital Insurer
 
MWD Advisors: Process Visibility-Enabling Business Transformation - Keynote P...
MWD Advisors: Process Visibility-Enabling Business Transformation - Keynote P...MWD Advisors: Process Visibility-Enabling Business Transformation - Keynote P...
MWD Advisors: Process Visibility-Enabling Business Transformation - Keynote P...Software AG
 
Getting Demo & POV Ready
Getting Demo & POV ReadyGetting Demo & POV Ready
Getting Demo & POV ReadyThousandEyes
 
Cgs channel enablement services nn bm
Cgs channel enablement services  nn bmCgs channel enablement services  nn bm
Cgs channel enablement services nn bmBill McCready
 
BreakingOutofTradeInsertiaWhileAchievingDigitalTransformation.pptx
BreakingOutofTradeInsertiaWhileAchievingDigitalTransformation.pptxBreakingOutofTradeInsertiaWhileAchievingDigitalTransformation.pptx
BreakingOutofTradeInsertiaWhileAchievingDigitalTransformation.pptxNilay Kumar
 
Symantec Corporate Presentation May 31, 2013
Symantec Corporate Presentation May 31, 2013Symantec Corporate Presentation May 31, 2013
Symantec Corporate Presentation May 31, 2013InvestorSymantec
 
Presentación de Ryan Manchee, DGMediaMind
Presentación de Ryan Manchee, DGMediaMind Presentación de Ryan Manchee, DGMediaMind
Presentación de Ryan Manchee, DGMediaMind IAB México
 
Finding the Digital Sweet Spot
Finding the Digital Sweet SpotFinding the Digital Sweet Spot
Finding the Digital Sweet SpotRyan Manchee
 

Similar to Tackling Compliance When it Becomes Your Biggest Performance Bottleneck.pdf (20)

Partnership with Zettagrid Indonesia
Partnership with Zettagrid IndonesiaPartnership with Zettagrid Indonesia
Partnership with Zettagrid Indonesia
 
Arx mindbusinessportfolio
Arx mindbusinessportfolioArx mindbusinessportfolio
Arx mindbusinessportfolio
 
Cloudflare Partner Program 2020
Cloudflare Partner Program 2020Cloudflare Partner Program 2020
Cloudflare Partner Program 2020
 
Rate&Bill: 20 reasons to choose it
Rate&Bill: 20 reasons to choose itRate&Bill: 20 reasons to choose it
Rate&Bill: 20 reasons to choose it
 
GTM Lab Data Slide Deck.pptx
GTM Lab Data Slide Deck.pptxGTM Lab Data Slide Deck.pptx
GTM Lab Data Slide Deck.pptx
 
Align Your ITSM and SecOps Strategy for Unstoppable IT
Align Your ITSM and SecOps Strategy for Unstoppable ITAlign Your ITSM and SecOps Strategy for Unstoppable IT
Align Your ITSM and SecOps Strategy for Unstoppable IT
 
What is Hybrid Cloud and understanding what's right for your business?
What is Hybrid Cloud and understanding what's right for your business?What is Hybrid Cloud and understanding what's right for your business?
What is Hybrid Cloud and understanding what's right for your business?
 
Startup InsurTech Award - iCede
Startup InsurTech Award - iCedeStartup InsurTech Award - iCede
Startup InsurTech Award - iCede
 
G-Cloud #AccreditCamp
G-Cloud #AccreditCamp G-Cloud #AccreditCamp
G-Cloud #AccreditCamp
 
Accredcamp
AccredcampAccredcamp
Accredcamp
 
SCG Sales Tech Guide
SCG Sales Tech Guide SCG Sales Tech Guide
SCG Sales Tech Guide
 
Democrance - InsurTech Innovation Award 2022
Democrance - InsurTech Innovation Award 2022Democrance - InsurTech Innovation Award 2022
Democrance - InsurTech Innovation Award 2022
 
MWD Advisors: Process Visibility-Enabling Business Transformation - Keynote P...
MWD Advisors: Process Visibility-Enabling Business Transformation - Keynote P...MWD Advisors: Process Visibility-Enabling Business Transformation - Keynote P...
MWD Advisors: Process Visibility-Enabling Business Transformation - Keynote P...
 
Getting Demo & POV Ready
Getting Demo & POV ReadyGetting Demo & POV Ready
Getting Demo & POV Ready
 
Cgs channel enablement services nn bm
Cgs channel enablement services  nn bmCgs channel enablement services  nn bm
Cgs channel enablement services nn bm
 
BreakingOutofTradeInsertiaWhileAchievingDigitalTransformation.pptx
BreakingOutofTradeInsertiaWhileAchievingDigitalTransformation.pptxBreakingOutofTradeInsertiaWhileAchievingDigitalTransformation.pptx
BreakingOutofTradeInsertiaWhileAchievingDigitalTransformation.pptx
 
Symantec Corporate Presentation May 31, 2013
Symantec Corporate Presentation May 31, 2013Symantec Corporate Presentation May 31, 2013
Symantec Corporate Presentation May 31, 2013
 
Test 2
Test 2Test 2
Test 2
 
Presentación de Ryan Manchee, DGMediaMind
Presentación de Ryan Manchee, DGMediaMind Presentación de Ryan Manchee, DGMediaMind
Presentación de Ryan Manchee, DGMediaMind
 
Finding the Digital Sweet Spot
Finding the Digital Sweet SpotFinding the Digital Sweet Spot
Finding the Digital Sweet Spot
 

Recently uploaded

VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 

Recently uploaded (20)

VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 

Tackling Compliance When it Becomes Your Biggest Performance Bottleneck.pdf

  • 1. Tackling Compliance When it Becomes Your Biggest Performance Bottleneck By Craig Saunders & Harsh Gautam
  • 2. Classified as General Premise © Kindred Group plc 2022 2 “ When developing software in heavily regulated markets, compliance is not an optional nicety. But compliance is every developer’s nightmare when trying to write performant software, especially when you need to communicate with 3rd party software or a Regulator’s services that can’t give you the performance guarantees you require. This talk’s premise is to talk through such challenges we faced and how we used event-driven architecture and asynchronous events to handle different latencies to ensure that different market performance issues didn’t have system- wide effects.
  • 3. Classified as General Our Opinions © Kindred Group plc 2022 3
  • 4. Classified as General Who are we? © Kindred Group plc 2022 4
  • 5. Classified as General Bio Cont… Craig Saunders © Kindred Group plc 2022 01 Software Engineer with 26 years experience 02 Work in the strategic side of software development 03 Work closely with Architects, Stakeholders and POs 04 Worked with Microservices for about 6 years 01 Huge tech advocate – love playing with the toys
  • 6. Classified as General Bio Cont… Harsh Gautam © Kindred Group plc 2022 01 Software Engineer with 14 years experience 02 Passionate about solving complex technical challenges 03 Background with distributed systems and microservices 04 Love developing and designing new systems 05 Passion for learning new technologies and keeping myself current
  • 7. Classified as General Bio Cont… © Kindred Group plc 2022 Ultimately, We’re geeks who likes playing with the toys!!
  • 8. Classified as General Who is Kindred? © Kindred Group plc 2022 8
  • 9. Classified as General Who is Kindred? © Kindred Group plc 2022 9 A tech company that operates in the online gambling space
  • 10. Classified as General Tenet of Kindred’s Core Values © Kindred Group plc 2022 10 Player sustainability and compliance is non-negotiable
  • 11. Classified as General © Kindred Group plc 2022 11 Kindred Core Values – Goes Beyond Kindred is about making gambling 100% enjoyable
  • 12. Classified as General Set the Scene © Kindred Group plc 2022 12
  • 13. Classified as General © Kindred Group plc 2022 13 Problem Domain? 01 Event-driven microservice landscape on Kubernetes At Kindred we have adopted the event-driven and microservice paradigms for a long time. This throws up may challenges with needing to be. fault-tolerant. 02 Zero downtime operation conditions We are a 24/7 product meaning customers are constantly engaging with our product; hence ensuring we have no occurrences of the site being down is critical. Deployments and migrations strategies need to bear this in mind. 03 50 million wallet transactions per day We have an average of 50 million transactions per day and can peak over 60 million; meaning we have to scale to a lot of customer activity – especially around large tournament such the World Cup. 04 Campaigns targeting upwards of 3 mil customers At certain times in the year Marketing can do large campaign send-outs targeting millions of customer; hence we need to scale to huge burst in traffic. 05 Real-time customer communication and engagement Modern marketing strategy is personalised content delivered at the right time to the customer, which can be onsite; hence execution speed is paramount. Couple this with large batch send- outs, we need to ensure that batch send-outs don’t impact the marketing strategy.
  • 14. Classified as General Let’s Get Down to the Tech © Kindred Group plc 2022 14
  • 15. Classified as General Platform Interactions © Kindred Group plc 2022 15 Campaign Back- office Platform Audience Integration Platform Communications Platform Player Account Platform Customer Verification Regulatory Integration Player Sustainability Execution Platform
  • 16. Classified as General Execution Platform Raison D’Etre © Kindred Group plc 2022 16 “ The execution platform is a decision agnostic platform that enable safe, reliable and predictable execution of a marketing strategy to offer rewards and/or send communications. It enables 3rd party decision tools to be able to seamlessly integrate with Kindred platform
  • 17. Classified as General Execution Platform © Kindred Group plc 2022 17 Send-out Execution Marketing Compliance Orchestration
  • 18. Classified as General © Kindred Group plc 2022 18 Execution Platform Campaign Send-out Snapshot ADAPTER Campaign Association ADAPTER Test Accounts Adapter ADAPTER Campaign Send-out Executor ADAPTER Campaign Customer Details ADAPTER Campaign Orchestrator ADAPTER Campaign Scheduler ADAPTER Orchestration Marketing Compliance ADAPTER Marketing Compliance Profile ADAPTER Marketing Compliance Campaign Model ADAPTER Campaign Back-office Execution Rewards Notification Service ADAPTER
  • 19. Classified as General © Kindred Group plc 2022 19 Challenges 01 Required to communicate with regulator components with little guarantees 02 Different markets require more checks that others 03 Real-time customer engagement whilst the customer is on-site 04 Large batch-based send-outs targeting up to 3 millions customers
  • 20. Classified as General © Kindred Group plc 2022 20 Example Half time when we need to target customer quickly for second half opportunities
  • 21. Classified as General Time to Cry? The latency from third parties makes this impossible How do we know we’ve achieve success? How’s this going to be possible? 20+ markets, how can we guarantee any consistency?
  • 22. Classified as General Impossible Task? No © Kindred Group plc 2022 22 It’s just another challenge we developers dream about
  • 23. Classified as General So What Do We Do About It? © Kindred Group plc 2022 23 Let’s Dive In
  • 24. Classified as General © Kindred Group plc 2022 24 Technique 1: Microservices Principles 01 Single responsibly for each component 02 Optimize for the responsibility E.G., connections, memory, CPU, etc 03 Reduce points of failure. Can we continue when a service is down? 04 Reduced resource usage means key services can have more instances 01 Required to communicate with regulator components with little guarantees
  • 25. Classified as General © Kindred Group plc 2022 25 Technique 1: Example 01 Heavy reads, low writes; hence can optimise for reading with caches 02 Low reads and caching means we can optimize for many connections 03 Because it’s standalone, it’s less likely to interfere with the Executor 04 If it falls over, it’s very quick to restart which reduces the impact 01 Smaller resource usage; hence easier to have more instances Campaign Send-out Snapshot ADAPTER
  • 26. Classified as General © Kindred Group plc 2022 26 Focus Area for Potential Bottleneck Campaign Send-out Executor ADAPTER Marketing Compliance ADAPTER Marketing Compliance Profile ADAPTER Marketing Compliance Execution
  • 27. Classified as General Let’s Define the Terms © Kindred Group plc 2022 27 Marketing Compliance Report Marketing Compliance Profile How can we engage with a customer in a compliant way. Typical action are can be target with rewards or what channel are allowed. Can we engage with the customer at all? Aggregates all the data points that are needed to build a compliance report. Continually monitors the other systems to ensure the profile is up- to-date.
  • 28. Classified as General © Kindred Group plc 2022 29 Component Responsibilities Campaign Send-out Executor ADAPTER Execution Marketing Compliance Profile ADAPTER Marketing Compliance ADAPTER Marketing Compliance Request a compliance report Decides on what actions can be done based on the compliance report Returns a compliance report based on the rules Runs the compliance rules for the customer’s jurisdiction Returns a valid compliance profile Aggregates the compliance profile from a myriad of services
  • 29. Classified as General © Kindred Group plc 2022 30 Technique 2: Asynchronous Hand-offs Used two queue for asynchronous communication to hand-off long running queries Hand-offs meant that slower queries were not holding up newer ones The compliance profile was asynchronously updated in the background Needed mechanism to know when the profile was consistent Simple requests, where the profile was already complete, are faster
  • 30. Classified as General Technique 2 – Asynchronous Hand-offs © Kindred Group plc 2022 ADAPTER Compliance Profile Retrieval Update Compliance Profile Compliance Execution ADAPTER Regulatory Engine ADAPTER Regulatory Integration Rofus ADAPTER Third Party System Compliance Report Request Compliance Profile Request Marketing Compliance Profile Retrieve Batch of Customer to Check Request a Batch of Customer to Check Marketing Compliance Service Execution Platform Compliance Profile Response Obtain Profile From the DB Update Profile in the DB Compliance Report Response Request to Update Profile in the DB
  • 31. Classified as General Technique 2: Sequence © Kindred Group plc 2022 32
  • 32. Classified as General © Kindred Group plc 2022 33 Is This Over?
  • 33. Classified as General © Kindred Group plc 2022 34 No! How do we know we’ve succeeded?
  • 34. Classified as General © Kindred Group plc 2022 35 Inspiration from the Theory of Constraints the constraint In any journey, there’s only one bottleneck
  • 35. Classified as General © Kindred Group plc 2022 36 Inspiration from the Theory of Constraints the constraint the constraint Everything else to the constraint Identify Exploit Subordinate Elevate Don’t stop now the constraint the constraint everything else to the constraint the constraint Find your next to constraint
  • 36. Classified as General © Kindred Group plc 2022 37 Inspiration from the Theory of Constraints Resolve Protect If you can’t reduce the constraint, then protect it Ensure efficiency in and out of the constraint The constraint will define your throughput Any improvement either side of the bottleneck is pointless Understand the bottleneck, make the change and then check If the bottleneck has moved, rinse-repeat with the new location
  • 37. Classified as General Technique 3: Performance Monitoring © Kindred Group plc 2022 01 Highlight where the bottleneck is in the stages of the send-out 02 Be able to slice and dice by campaign, market, and real-time vs batch 03 Understand throughput numbers by market 04 Understand where time is being spent that can affect end-to-end time 05 Understand common traffic patterns to replay in our PT environment
  • 38. Classified as General © Kindred Group plc 2022 39 Performance Monitoring Examples
  • 39. Classified as General © Kindred Group plc 2022 40 Performance Monitoring Examples
  • 40. Classified as General So Where is the Constraint? © Kindred Group plc 2022 ADAPTER Compliance Profile Retrieval Update Compliance Profile Regulatory Engine ADAPTER Regulatory Integration Compliance Profile Request Marketing Compliance Profile Retrieve Batch of Customer to Check Request a Batch of Customers to Check Execution Platform Compliance Profile Response Obtain Profile From the DB Update Profile in the DB Request to Update Profile in the DB
  • 41. Classified as General What Was the Issue? © Kindred Group plc 2022 42 Batches of requests for 1,000 customers are sent every 5 seconds per market Scenario: 100,000 customers for one market 100,000 divided by 1,000 multiplied by 5 seconds = 8 minutes 40 seconds Conclusion: The last few batches will have a natural 7 to 8 minutes delay
  • 42. Classified as General Time to Fix? Can it be Fix? © Kindred Group plc 2022 43 Of course, we can
  • 43. Classified as General © Kindred Group plc 2022 44 Technique 4: Non-blocking Queue Non-blocking Blocking Consumes customer selections one-by-one and stores in a DB Every 5 seconds create a lock Collates a batch and sends via a message to the Regulatory Integration component to process No batching, DB or locks required Reads the batch of customer directly from the queue up to the 1K limit Leveraging the power of Solace to poll and acknowledge message
  • 44. Classified as General © Kindred Group plc 2022 45 Technique 4: Recap Sequence Area of focus
  • 45. Classified as General Technique 4: Sequence © Kindred Group plc 2022 46
  • 46. Classified as General What Did the Solution Bring? © Kindred Group plc 2022 47 Batches of requests for 1,000 customers are sent continuously for the queue per market Scenario: 100,000 customers for one market 100,000 divided by 1,000 multiplied by 100 milliseconds = 10 seconds Conclusion: Reduced 7 to 8 minutes delay to 10 seconds
  • 47. Classified as General © Kindred Group plc 2022 48 Technique 5: Only Request What You Need Concept: Based on the campaign strategy and the rules that need to be executed for the respective market just request the necessary sections of the compliance profile Advantage: Based on the campaign strategy and the rules that need to be executed for the respective Example 1: A newsletter campaign strategy doesn’t require bonus block/opt-outs, etc. Example 2: A campaign strategy that rewards a customer without any communication doesn’t need to check
  • 48. Classified as General © Kindred Group plc 2022 49 New Problem – Real-time vs Batch Real-time needs to be processed with very low latency Real-time send-outs are spread out over the day with occasional peaks Batch targets large quantities of customers at the same time - up to 3 mil Batch sends can block real-time by flooding the queue Batch sends can tolerate a slow throughput through the system
  • 49. Classified as General © Kindred Group plc 2022 50 Technique 6: Bulkhead Pattern “ The bulkhead pattern is based on a concept from building ships. With a bulkhead design, a ship is divided into completely segregated and watertight compartments called bulkheads. Even if the ship’s hull is punctured, because the ship is divided into watertight compartments (bulkheads), the bulkhead will keep the water confined to the area of the ship where the puncture occurred and prevent the entire ship from filling with water and sinking.
  • 50. Classified as General Variant of the Bulkhead Pattern © Kindred Group plc 2022 51 Campaign Send-out Executor ADAPTER Unica Adapter (batch sending) ADAPTER MMC (real- time sending) ADAPTER Audience Campaign Feeder (batch sending) ADAPTER Audience Integration Platform Execution Platform Batch Queue Real-time Queue The real-time queue is a constraint that needs to be protected. With the influence bulkhead pattern, we can build a second queue dedicated to real-time; hence large batch sends affect real-time less.
  • 51. Classified as General Santa Claus’ Present to You © Kindred Group plc 2022 52 https://microservices.io/patterns/microservices.html
  • 52. Classified as General © Kindred Group plc 2022 53 Keys to Success 01 Life throws up challenges that software can solve with the right mindset 02 Event-driven and asynchronous messaging is perfect to reduce delays 03 Performance monitoring in production is essential to understand if you’ve solved the problem 04 You are never finished. Constant monitoring to assess the constraint is important 05 These are challenges techies love to solve
  • 53. Classified as General © Kindred Group plc 2022 54