Real World Leadership

Leadership One Day at a Time

Tag: Enterprise Data

  • The Final Mile – Data Delivery Problem

    The Final Mile – Data Delivery Problem

    Spend enough time around enterprise technology and you start to notice a pattern. Organizations invest heavily in data infrastructure. They build sophisticated ERP environments, modern data warehouses, and analytics platforms that can answer almost any question about the business in near real time. Then someone outside the organization needs to see some of that data, and after all that investment, someone opens Excel, hits export, and emails a PDF.

    That is where the money stops working.

    I have watched this play out in organization after organization, and the frustrating part is not that people make this choice. The frustrating part is that they usually have no better option. The tools we built to share data with the outside world are, in most cases, decades old and designed for problems that no longer exist. The gap between what enterprise data systems can do and what actually reaches the person who needs it is enormous, and almost nobody talks about it.


    1991 Called

    The PDF was a genuinely clever solution to a real problem. Adobe designed it to answer a question that mattered in the early 1990s: how do you make sure a document looks identical on every printer, regardless of the operating system or software on the machine doing the printing? Print fidelity. That was the job. And for that job, the PDF is still excellent.

    The problem is that nobody is printing anymore. Or rather, the reason people share documents has changed completely, and the format has not.

    When you export data as a PDF today, you are using a print fidelity format to do a data delivery job. The analyst building the report decides what to include and how to organize it. Those decisions get baked in permanently at the moment of export. If the recipient needs to see a different time period, a different vendor, a different product line, they cannot. They submit a data request and wait. If the underlying data has changed since the report was generated, there is no way for the recipient to know. If the file ends up somewhere it should not, there is no mechanism to pull it back.

    The document just circulates. Forever. On systems you cannot see, held by people you may no longer be able to account for.

    Research consistently suggests that static reports and statements typically surface somewhere between one and five percent of the information available in the underlying dataset. Everything else gets left behind, either because pulling it all in would make the document unwieldy or because the person building the report could not anticipate every question someone might eventually want to ask.

    That is not a small problem to work around. It is the format failing at its actual purpose.


    Portals Were Not the Answer Either

    The data portal was supposed to fix this. Stop sending static files and put the dashboards online. Give everyone a login. Let them explore.

    And portals did fix part of it. Genuine interactivity is a real improvement over a static PDF, and I do not want to dismiss that. But portals introduced a different set of problems that are just as significant in practice.

    Start with the connectivity assumption. A portal needs a live network connection for every single interaction. That sounds obvious, but think about where the people who need data actually work. A field technician in a hospital basement. An insurance adjuster in a flood-damaged neighborhood. An executive on an overnight flight reviewing board materials before a morning meeting. A small business owner in a rural area with unreliable service. A portal cannot reach any of them. It just stops working. The data exists, but the people who need it cannot get to it.

    Then there is the provisioning overhead. Every external person who needs portal access needs a login, a license, and properly configured permissions inside your source system. For your internal team, that is a manageable process even if it is expensive. For customers, vendors, auditors, and partners, it is enough friction that most organizations give up and send a PDF instead. Which is exactly where they started.

    The security picture is also more complicated than it looks. When a portal session is compromised, the attacker gets access to everything that session was authorized to see. That scope is almost always broader than the minimum necessary for the task the legitimate user was trying to perform. A portal breach exposes the session. That is a very different risk profile from a breach of a single scoped document.

    So portals solved interactivity and created new problems in portability, offline access, provisioning cost, and session security. Progress, but not a solution.


    And the Spreadsheet Is Not a Shortcut

    There is a third path that many organizations quietly rely on: just export to CSV or Excel and let the recipient figure it out. At least the raw data is there.

    The raw data being there is about the only thing that can be said in favor of this approach. Spreadsheet exports have no access controls after delivery. They create no audit trail. There is no mechanism to personalize them at scale without significant manual work. And critically, they move the entire analytical burden onto the recipient.

    For someone who lives in Excel, that might be fine. For most of the people who actually receive these files, including customers, executives, external partners, and field teams, it is not. Handing someone a data file when they need clear answers to specific questions is not a solution. It is a transfer of responsibility.


    What This Actually Costs

    The direct costs are visible if you look for them. Deloitte’s 2023 Internal Audit Technology Survey found that evidence collection and assembly accounts for 35 to 45 percent of total internal audit effort. Not a secondary task. The biggest single time consumer in the entire process. Multiply that across vendor reporting cycles, regulatory submissions, customer communication workflows, and executive briefing preparation, and the hours add up fast.

    But the more interesting costs are the ones that never appear in any budget review.

    When a CFO makes a capital allocation decision based on a report that did not include the regional breakdown they needed, that cost shows up in the outcome, not in the report production process. When a compliance team spends a week fielding follow-up questions from auditors because the evidence package was not filterable, that cost shows up as project delay. When customers make financial decisions based on statements that showed them a small fraction of what their account actually contains, that cost shows up as trust erosion over time, slow enough that nobody connects it back to the statement format.

    These are real costs. They are not small costs. They are just costs that get attributed to the wrong place, which is why the format that caused them rarely gets scrutinized.


    The Compliance Problem Is Not Going Away

    I want to spend a moment on the governance dimension because it tends to get underweighted in these conversations.

    Every document that leaves an enterprise system is, from that moment forward, outside your control. The governance frameworks that modern regulated industries operate under were not designed with that assumption in mind. GDPR’s data minimization principle requires that personal data be limited to what is actually necessary for the stated purpose. HIPAA’s minimum necessary standard says the same thing for protected health information. The security principle of least privilege has been foundational to enterprise information security practice for decades.

    Static document delivery, by its structure, tends to fail all three. A vendor performance report that includes fields unrelated to that vendor’s scope. A customer statement that carries account metadata the customer never requested. An audit evidence package sitting on an external laptop months after the engagement closed. None of these are unusual. All of them represent compliance exposure.

    This is not a criticism of the people building these documents. They are working with the tools available to them. The tools were designed before these regulatory frameworks existed, and they were not updated when the requirements changed.


    What Would Actually Work

    Here is what a data delivery format designed for the problems organizations actually face today would need to do.

    It would need to carry data to any recipient through any channel without requiring a platform license or a live connection. It would need to enforce access controls at the individual recipient level so that each person sees exactly what they are supposed to see and nothing else. It would need to maintain a complete audit trail from delivery through every subsequent interaction. It would need to support revocation after the fact. And it would need to work offline, because the real world does not reliably provide connectivity.

    None of these requirements is technically impossible. Parts of them are already solved in various contexts. Encrypted files handle some access control. Portals maintain some audit history. Offline-capable applications exist. The gap is not in any individual capability. It is in a format that delivers all of them together, in a package portable enough to reach any recipient through any channel.

    The PDF solved portability and sacrificed everything else. The portal solved interactivity and sacrificed portability, offline access, and minimized security scope. The spreadsheet preserved raw data and sacrificed governance, usability, and controlled delivery entirely.

    Every format we have was designed to solve one problem, and each one creates a different set of problems downstream. What the enterprise does not yet have, in any mainstream form, is a delivery format designed from scratch around the full set of requirements.


    Why This Keeps Getting Ignored

    Part of what makes this problem persistent is that it does not announce itself loudly. The data gets shared. The report gets sent. The recipient opens something. It works well enough that nobody flags it as broken.

    The failures are quiet. A decision made on incomplete information. A compliance gap that surfaces during an audit. A customer who stops engaging because they never felt they could understand their own account. A vendor relationship that deteriorates because SLA reporting was always a negotiation over what the data actually showed.

    None of these failures get traced back to the format. They get traced back to processes, to people, to systems. The format that connects all of them stays invisible.

    Regulatory pressure is tightening this. The expectation that organizations can demonstrate with precision what data was shared, with whom, when, and under what authorization is moving from advanced capability to baseline audit requirement. That shift will force some hard conversations about formats that have never had to justify themselves.

    When those conversations happen, the answer is not going to be a better PDF.