Data-Driven Solutions

14/07/2026

After more than 30 years working with SQL Server, there's still one feature I find myself wishing it had.

Autonomous Transactions.

Not because they're flashy.
Because they solve a very real engineering problem.

Imagine a regulatory reporting process that's been running for three hours.
It has identified the reportable population, built hundreds of thousands of records, generated XML and is almost finished.....then one unexpected data condition causes the transaction to roll back.

That's exactly what should happen.

What shouldn't disappear is the diagnostic history explaining how far the process got before it failed.

Today, SQL Server gives us several ways to work around this—CLR, separate connections, Service Broker and external logging among them. In my experience, CLR provides the cleanest implementation when it's governed appropriately.

But imagine if SQL Server simply supported autonomous transactions natively, allowing diagnostic information to be committed independently while the business transaction still rolled back.

For those of us building long-running ETL pipelines, regulatory reporting frameworks and financial systems, that would be a genuine game changer.

I've written a deeper article exploring the problem, the current options and why I think this is one capability SQL Server is still missing. Link in comments.

How are you solving this today?

09/07/2026

Over the past few weeks, I've spent a lot of time rebuilding my website.

On the surface it's a new look and feel, new case studies, new service pages and a much stronger portfolio.

But the biggest change wasn't the design.

It was taking the time to reflect on what Data-Driven Solutions actually does.

For years I've described myself as a SQL Server consultant, data engineer or regulatory reporting specialist.

The reality is that's only part of the story. What clients really engage me for is reducing operational and regulatory risk through disciplined data engineering.

Whether it's regulatory reporting, investment data, SQL Server optimisation, access governance or business automation, the outcome is always the same:

• Better quality data.
• Better governance.
• Better decisions.
• Less operational risk.

That reflection has shaped every page of the new website—from the language and case studies through to the solutions and business outcomes.

It's also been a reminder that sometimes it's worth stepping back and asking a simple question:

"What problem do I really solve?"

I'd love to hear your thoughts, and as always, any feedback on the new site is very welcome.

Reducing Regulatory & Operational Risk Through Data Engineering

17/05/2026

Most organisations don’t trust their data.
They trust the people manually fixing it before the meeting.

⸻

That’s one of the reasons I rebuilt the DDS website around a much clearer philosophy:

Reliable data systems should:
• drive action
• withstand failure
• remain auditable
• support trusted decisions

Because this isn’t just about pipelines.
It’s about building systems organisations can actually trust.

⸻

Much of this thinking now sits behind what I call:
The DDS Data Reliability Framework.
Built from years working in regulated and enterprise environments where auditability, resilience, and operational trust genuinely matter.

16/05/2026

Most data projects don’t fail in production.
They fail long before anyone notices.

• Built fast.
• Used briefly.
• Then quietly replaced.

Not because they didn’t work.

Because they didn’t last.

So I stopped thinking in pipelines…
and started thinking in systems.

This is what that looks like:

1. Understand the process
Fix the problem — not just the data.

2. Design for reuse
Build once. Stop rebuilding.

3. Make it auditable
No proof = no trust.

4. Design for action
If nothing happens, nothing changes.

5. Design for failure
If it breaks silently, it’s already broken.

6. Maintain control
Security, ownership, and continuous validation.

Individually, these are good practices.

Together — they’re the difference between
something that works…
and something that lasts.

This isn’t a pipeline.
It’s a system.

15/05/2026

Most data solutions don’t break.
They drift.

Step 6: Maintain control.

Security should never be an afterthought.

It’s part of the design.

Role-based access.
Least privilege.
Clear ownership.

Someone in the business
must be accountable for who has access —
and why.

Because access isn’t static.

• It changes.
• It expands.
• It gets forgotten.

And then there’s usage.

• Data evolves.
• Patterns change.
• People adapt how they use systems.

What was correct at launch isn’t guaranteed to stay correct.

If no one is reviewing it…
it’s already drifting.

Are people using it as intended?
Are decisions still aligned?
Are controls still valid?

Because a system that’s not reviewed
is a system that’s no longer controlled.

The biggest risk isn’t bad design.
It’s losing control after go-live.

Good data engineering delivers solutions.

Great data engineering
keeps them secure, relevant, and trusted over time.

14/05/2026

Every data solution works… until it doesn’t.

Step 5: Design for failure. Not success.

Most systems are built
assuming everything goes right.

That’s not reality.

• APIs fail.
• Data arrives late.
• Formats change.
• Dependencies break.

The question isn’t:
“Will this fail?”

It’s:
“When it fails — what happens next?”

Does it retry?
Does it alert?
Does it stop safely?
Does anyone even know?

If failure isn’t visible —
it isn’t managed.

Monitoring isn’t optional.

Row counts.
Schema changes.
Data drift.

Logging isn’t optional.

What ran?
What failed?
What changed?

The most dangerous failures aren’t the ones that break.
They’re the ones that keep running.

Good data engineering works when things are fine.

Great data engineering keeps working when they aren’t.

13/05/2026

Data that sits doesn’t create value.

Step 4: Design for action.

Most data solutions end at the dashboard.

They inform.

But they don’t drive anything.

Insight without action is just information.

Ask yourself:

• Who needs to act on this?
• What decision should this trigger?
• What system should this update?
• What happens if no one does anything?

If you can’t answer those… your data isn’t finished.

Design the output.
Then design what happens next.

Automate the handoff.
Trigger the workflow.
Close the loop.

Because the goal isn’t a report.

The goal is impact.

I’ve seen dashboards no one used and alerts no one acted on.
Not because people didn’t care.

Because the solution wasn’t built to drive action.

Good data engineering delivers insights.

Great data engineering drives outcomes.

12/05/2026

If you can’t prove your data — it won’t be trusted.

Step 3: Make it auditable.

It’s not enough to move data.
It’s not enough to reuse logic.

If you can’t explain the output —
it won’t be used.

Where did it come from?
What changed?
Who touched it?
Why did this number move?

If those questions take time to answer…
confidence is already lost.

Auditability isn’t a “nice to have.”

It’s what turns data
into something people rely on.

Every transformation should be traceable.
Every decision should be explainable.
Every output should be defensible.

Because when something goes wrong and it will —

The question isn’t:
“Can you fix it?”

It’s:
“Can you prove what happened?”

I’ve seen technically “perfect” pipelines
get rejected…

Not because they were wrong —
because no one could trust them.

Good data engineering delivers outputs.

Great data engineering delivers
confidence in every decision made from them.

11/05/2026

Most “one-off” data solutions aren’t one-off.
They’re just badly designed.

Step 2: Design for reuse.

Once you understand the process properly,
this is where most teams still get it wrong.

They build for the request.
Not for what comes next.

A new pipeline.
A new script.
A new model.

Every time.

Because it feels faster.

Until you’re maintaining
10 versions of the same thing.

The reality?

Most problems repeat.
Across teams.
Across systems.
Across time.

If you design for reuse from day one:

You reduce duplication.
You reduce maintenance.
You increase consistency.

But reuse doesn’t happen by accident.

It’s a decision.

Can this be parameterised?
Can this be extended?
Can this be used somewhere else?

If the answer is yes, it’s not a task.

It’s a data product.

I’ve seen teams rebuild the same logic
over and over again…

Not because they had to.
Because they didn’t design not to.

Good data engineering delivers solutions.

Great data engineering builds things once and uses them everywhere.

10/05/2026

If you automate a bad process, you’ve just made it worse.

Step 1: Understand the process.

Before writing a single line of code, this is where I start.

Not the documented version.
Not the “happy path.”

The one that actually runs.

Where it breaks.
Where people intervene.
Where workarounds exist.
Where decisions actually get made.

Because if you don’t understand that…
you’re not solving the problem.

You’re just wrapping it in code.

I’ve seen entire pipelines built
on assumptions that weren’t real.

If your process only works
because someone “knows what to do”— it’s already broken.

Good data engineering moves data.

Great data engineering fixes the process first.