http://www.datadrivensolutions.com.au/

Data-Driven Solutions, Gold Coast (2026)

12/10/2026

We think we may have created a problem with the way we are measuring technology delivery.

Speed has become a measure of success.

So has quantity.

How quickly did we deliver?
How many features?
How many pipelines?
How many releases?
How many processes did we automate?

All very easy to measure.

Quality is harder.

Did we build the right thing?
Is the data complete?
Can we prove it?
What controls did we build around it?
What risk did we introduce while moving faster?

Those things are much harder to measure.

And that's where we think we have a problem.

We've become very good at measuring how much we deliver and how quickly we deliver it.

We're not convinced we put the same effort into proving that what we delivered is actually correct.

Speed matters.

Quantity matters.

But without quality and assurance, aren't we just getting better at producing more risk, faster?

hashtag hashtag hashtag hashtag

11/10/2026

Engineering tells you whether the data can move.
Assurance asks whether it moved correctly.
Privacy asks whether it should have moved in the first place.

Three very different questions.

But we often treat them as three completely separate conversations.

Take something as simple as moving customer data from one system to another.

Engineering asks:

Did the pipeline run?
Did the data arrive?
Did the schema hold?
Did anything fail?

All important.

But a technically successful pipeline doesn't mean the right data arrived.

Did every record that should have moved actually move?
Was anything dropped, duplicated or changed?
Can we reconcile source to destination?
Can we prove it?

That's assurance.

And then there's the question that is often asked far too late:

Should we have moved all that data in the first place?

Did the destination actually need every field?
Did we copy personal information simply because it was available?
Are we moving data because there is a genuine requirement, or because moving everything is easier?

That's privacy.

Which creates an interesting problem.

A data pipeline can be technically perfect and still be wrong.

It can successfully move data that should never have been moved.

Or it can report 99% success while the missing 1% contains exactly the records a regulatory process needed.

The job completed successfully.

The engineering worked.

But the outcome was still wrong.

That's why, at Data-Driven Solutions, we believe these three disciplines need to meet much earlier.

Engineering: Can we move it?

Assurance: Did we move it correctly?

Privacy: Should we move it?

Three questions about the same data, and getting only one of them right isn't enough.

hashtag hashtag hashtag hashtag

09/10/2026

Engineering tells you whether the data can move.

Assurance asks whether it moved correctly.

Privacy asks whether it should have moved in the first place.

Three very different questions.

But we often treat them as three completely separate conversations.

Take something as simple as moving customer data from one system to another.

Engineering asks:

Did the pipeline run?
Did the data arrive?
Did the schema hold?
Did anything fail?

All important.

But a technically successful pipeline doesn't mean the right data arrived.

Did every record that should have moved actually move?
Was anything dropped, duplicated or changed?
Can we reconcile source to destination?
Can we prove it?

That's assurance.

And then there's the question that is often asked far too late:

Should we have moved all that data in the first place?

Did the destination actually need every field?
Did we copy personal information simply because it was available?
Are we moving data because there is a genuine requirement, or because moving everything is easier?

That's privacy.

Which creates an interesting problem.
A data pipeline can be technically perfect and still be wrong.
It can successfully move data that should never have been moved.

Or it can report 99% success while the missing 1% contains exactly the records a regulatory process needed.

The job completed successfully.

The engineering worked.

But the outcome was still wrong.

That's why I think these three disciplines need to meet much earlier.

Engineering: Can we move it?

Assurance: Did we move it correctly?

Privacy: Should we move it?

Three questions about the same data and getting only one of them right isn't enough.

08/10/2026

A random conversation in a local organic shop reminded me of something important about AI and careers in data.

I went into Flannerys the other day looking for something for my wife and ended up talking to the naturopath.

She was incredibly knowledgeable, but what struck me most was how she thought. Curious. Analytical. Constantly questioning *why*.

Eventually she asked:

“So what do you do? You seem to have a curious and analytical mind.”

I explained that I'm a data consultant specialising in data engineering, improving data processes and automating regulatory reporting.

Her eyes lit up.

“I’m trying to get into that field. What should I focus on?”

Her concern was that she was coming into data relatively late, just as AI is changing the profession.

My advice?

Learn the technical foundations. Databases, data modelling, how information is stored, transformed and moved.

But then go deeper into **privacy and the people behind the data**.

AI will get increasingly good at writing SQL, building pipelines and generating technical solutions.

But someone still needs to ask:

**Should we be doing this at all?**

**What could someone infer from this data that we never intended?**

**Would the person whose data this is reasonably expect us to use it this way?**

Those aren't purely engineering questions.

They're questions of judgement, governance, privacy and critical thinking.

AI may reduce the value of some routine technical ex*****on. But it may simultaneously increase the value of judgement around what data should be collected, combined, exposed and retained.

**In an AI-driven world, knowing how to use data will matter. Knowing when we shouldn't may matter even more.**

07/10/2026

Visibility of a problem is not ownership of its cause.

Regulatory reporting teams often have the best visibility of data problems.

That’s because they sit at the end of the chain.

Missing classifications.
Incomplete customer data.
Incorrect mappings.
Broken interfaces.
Inconsistent source data.

Eventually, all of it arrives at reporting.

And something interesting happens.

Because the reporting team discovers the problem, the organisation gradually starts treating it as their problem to fix.

But finding a defect does not mean you created it.

And repeatedly correcting it downstream does not constitute a control.

It can actually hide the weakness upstream.

Good governance asks two different questions:

Who detected the problem?
Who controls the process that caused it?

Those answers are often very different.

Regulatory reporting should provide visibility and assurance.

It should not become the permanent repair shop for broken upstream processes.

Visibility is not ownership.

06/10/2026

CRS 3.0 isn't just a regulatory reporting change.

It's a data engineering change.

Australia moves to CRS Schema 3.0 from 1 January 2027, and we're already seeing the same transition taking place internationally.

But changing the XML schema is arguably the easy part.

The harder questions are:

• Is the underlying customer data complete?

• Can tax residency and controlling-person information be reliably sourced?

• Are TINs valid and consistently captured?

• Can every reported value be traced back to its source?

• And can you prove that nothing that should have been reported was lost between the source system and the final submission?

A file can pass schema validation and still contain poor data.

That's why regulatory reporting increasingly needs to be treated as a data engineering problem—not simply a reporting exercise.

Validation proves the file is valid.

Reconciliation helps prove the data is right.

That distinction matters.

04/10/2026

The transaction exists in the source system.
It never reaches AML monitoring.
No alert fires.

FATF’s September report on professional money laundering highlights fragmented data as a barrier to detection. For me, that raises a practical engineering question: did all the relevant data reach monitoring in the first place?

A failed join, an outdated filter or an unmapped transaction type can silently exclude records; even when the overnight job reports “successful”.

This is why data engineering is key to effective AML monitoring.

A successful job tells you the process finished. It doesn’t prove that every transaction requiring monitoring arrived.

That takes reconciliation: checking source transactions against monitoring inputs, accounting for exclusions and raising exceptions when records go missing.

The monitoring rules can only assess what they receive.

Can you prove what reached yours?

FATF report: Professional money laundering, underground banking and hawala

02/10/2026

Sometimes the biggest improvements in technology aren't the flashy new features.

They're the ones that remove a problem you've been working around for years.

One of those for me is native Regular Expression (Regex) support in SQL Server 2025.

Much of what I build centres around data quality, regulatory reporting and audit. Validating complex patterns accurately isn't optional; it's a fundamental requirement.

For years we've achieved that using SQL CLR.

CLR has served us well, but it also means enabling CLR at the server level.

That's perfectly acceptable in many environments, but in others it introduces another security and governance conversation that I'd rather not have.
Native Regex changes that.

Take Tax Identification Numbers (TINs) as an example. Every jurisdiction has its own format, its own rules and its own quirks. Validating those accurately without Regex quickly becomes messy, or pushes the problem outside SQL Server altogether.

In our case, Regex isn't about convenience; it's about accurately validating data that regulators expect us to get right.

Now that validation can remain where the data lives.
For me, that's the real value.
Not because Regex is a clever new feature.

Because it removes a workaround, reduces risk and makes the overall solution simpler.

The best technology improvements don't always add something new.
Sometimes they simply let you stop doing something you never wanted to do in the first place.

02/10/2026

One of the hardest lessons I've learnt is that being right too early can look exactly the same as being wrong.

A colleague sent an email around this week explaining a new process.

As I was reading it, I couldn't help but smile.
It was almost identical to an approach I'd suggested over a year ago.

Back then the feedback was along the lines of:
> "That's not what a database is for."
> "We should separate the functionality."
> "It's bad practice."

The idea never went anywhere.
Fast forward a year.

The business now needs to retain original source files for longer. Audit requests have increased. Regulatory scrutiny has increased. Being able to reproduce exactly what was received from upstream systems has become far more valuable.

The interesting thing?

The solution didn't really change. The business did.
Technology rarely stands still, but business requirements move even faster.

Good architecture isn't just about solving today's problem. It's about anticipating tomorrow's questions.

One thing I've learnt over the years is that even a good idea needs the right audience at the right time.

An idea that's rejected today isn't necessarily a bad idea. Sometimes the organisation simply isn't ready for it yet.

I've had ideas accepted immediately, others take years, and some never happen at all.

That's just part of engineering solutions for real businesses.
Have you ever watched an idea come back years later and suddenly become "the obvious approach"?

14/07/2026

After more than 30 years working with SQL Server, there's still one feature I find myself wishing it had.

Autonomous Transactions.

Not because they're flashy.
Because they solve a very real engineering problem.

Imagine a regulatory reporting process that's been running for three hours.
It has identified the reportable population, built hundreds of thousands of records, generated XML and is almost finished.....then one unexpected data condition causes the transaction to roll back.

That's exactly what should happen.

What shouldn't disappear is the diagnostic history explaining how far the process got before it failed.

Today, SQL Server gives us several ways to work around this—CLR, separate connections, Service Broker and external logging among them. In my experience, CLR provides the cleanest implementation when it's governed appropriately.

But imagine if SQL Server simply supported autonomous transactions natively, allowing diagnostic information to be committed independently while the business transaction still rolled back.

For those of us building long-running ETL pipelines, regulatory reporting frameworks and financial systems, that would be a genuine game changer.

I've written a deeper article exploring the problem, the current options and why I think this is one capability SQL Server is still missing. Link in comments.

How are you solving this today?

Data-Driven Solutions

12/10/2026

11/10/2026

09/10/2026

08/10/2026

07/10/2026

06/10/2026

04/10/2026

02/10/2026

02/10/2026

14/07/2026

Address

Website

Alerts

Contact The Business

Shortcuts

Share

Category

Listing takedown request