Real World Leadership

Leadership One Day at a Time

Category: Cloud

  • The Final Mile – Data Delivery Problem

    The Final Mile – Data Delivery Problem

    Spend enough time around enterprise technology and you start to notice a pattern. Organizations invest heavily in data infrastructure. They build sophisticated ERP environments, modern data warehouses, and analytics platforms that can answer almost any question about the business in near real time. Then someone outside the organization needs to see some of that data, and after all that investment, someone opens Excel, hits export, and emails a PDF.

    That is where the money stops working.

    I have watched this play out in organization after organization, and the frustrating part is not that people make this choice. The frustrating part is that they usually have no better option. The tools we built to share data with the outside world are, in most cases, decades old and designed for problems that no longer exist. The gap between what enterprise data systems can do and what actually reaches the person who needs it is enormous, and almost nobody talks about it.


    1991 Called

    The PDF was a genuinely clever solution to a real problem. Adobe designed it to answer a question that mattered in the early 1990s: how do you make sure a document looks identical on every printer, regardless of the operating system or software on the machine doing the printing? Print fidelity. That was the job. And for that job, the PDF is still excellent.

    The problem is that nobody is printing anymore. Or rather, the reason people share documents has changed completely, and the format has not.

    When you export data as a PDF today, you are using a print fidelity format to do a data delivery job. The analyst building the report decides what to include and how to organize it. Those decisions get baked in permanently at the moment of export. If the recipient needs to see a different time period, a different vendor, a different product line, they cannot. They submit a data request and wait. If the underlying data has changed since the report was generated, there is no way for the recipient to know. If the file ends up somewhere it should not, there is no mechanism to pull it back.

    The document just circulates. Forever. On systems you cannot see, held by people you may no longer be able to account for.

    Research consistently suggests that static reports and statements typically surface somewhere between one and five percent of the information available in the underlying dataset. Everything else gets left behind, either because pulling it all in would make the document unwieldy or because the person building the report could not anticipate every question someone might eventually want to ask.

    That is not a small problem to work around. It is the format failing at its actual purpose.


    Portals Were Not the Answer Either

    The data portal was supposed to fix this. Stop sending static files and put the dashboards online. Give everyone a login. Let them explore.

    And portals did fix part of it. Genuine interactivity is a real improvement over a static PDF, and I do not want to dismiss that. But portals introduced a different set of problems that are just as significant in practice.

    Start with the connectivity assumption. A portal needs a live network connection for every single interaction. That sounds obvious, but think about where the people who need data actually work. A field technician in a hospital basement. An insurance adjuster in a flood-damaged neighborhood. An executive on an overnight flight reviewing board materials before a morning meeting. A small business owner in a rural area with unreliable service. A portal cannot reach any of them. It just stops working. The data exists, but the people who need it cannot get to it.

    Then there is the provisioning overhead. Every external person who needs portal access needs a login, a license, and properly configured permissions inside your source system. For your internal team, that is a manageable process even if it is expensive. For customers, vendors, auditors, and partners, it is enough friction that most organizations give up and send a PDF instead. Which is exactly where they started.

    The security picture is also more complicated than it looks. When a portal session is compromised, the attacker gets access to everything that session was authorized to see. That scope is almost always broader than the minimum necessary for the task the legitimate user was trying to perform. A portal breach exposes the session. That is a very different risk profile from a breach of a single scoped document.

    So portals solved interactivity and created new problems in portability, offline access, provisioning cost, and session security. Progress, but not a solution.


    And the Spreadsheet Is Not a Shortcut

    There is a third path that many organizations quietly rely on: just export to CSV or Excel and let the recipient figure it out. At least the raw data is there.

    The raw data being there is about the only thing that can be said in favor of this approach. Spreadsheet exports have no access controls after delivery. They create no audit trail. There is no mechanism to personalize them at scale without significant manual work. And critically, they move the entire analytical burden onto the recipient.

    For someone who lives in Excel, that might be fine. For most of the people who actually receive these files, including customers, executives, external partners, and field teams, it is not. Handing someone a data file when they need clear answers to specific questions is not a solution. It is a transfer of responsibility.


    What This Actually Costs

    The direct costs are visible if you look for them. Deloitte’s 2023 Internal Audit Technology Survey found that evidence collection and assembly accounts for 35 to 45 percent of total internal audit effort. Not a secondary task. The biggest single time consumer in the entire process. Multiply that across vendor reporting cycles, regulatory submissions, customer communication workflows, and executive briefing preparation, and the hours add up fast.

    But the more interesting costs are the ones that never appear in any budget review.

    When a CFO makes a capital allocation decision based on a report that did not include the regional breakdown they needed, that cost shows up in the outcome, not in the report production process. When a compliance team spends a week fielding follow-up questions from auditors because the evidence package was not filterable, that cost shows up as project delay. When customers make financial decisions based on statements that showed them a small fraction of what their account actually contains, that cost shows up as trust erosion over time, slow enough that nobody connects it back to the statement format.

    These are real costs. They are not small costs. They are just costs that get attributed to the wrong place, which is why the format that caused them rarely gets scrutinized.


    The Compliance Problem Is Not Going Away

    I want to spend a moment on the governance dimension because it tends to get underweighted in these conversations.

    Every document that leaves an enterprise system is, from that moment forward, outside your control. The governance frameworks that modern regulated industries operate under were not designed with that assumption in mind. GDPR’s data minimization principle requires that personal data be limited to what is actually necessary for the stated purpose. HIPAA’s minimum necessary standard says the same thing for protected health information. The security principle of least privilege has been foundational to enterprise information security practice for decades.

    Static document delivery, by its structure, tends to fail all three. A vendor performance report that includes fields unrelated to that vendor’s scope. A customer statement that carries account metadata the customer never requested. An audit evidence package sitting on an external laptop months after the engagement closed. None of these are unusual. All of them represent compliance exposure.

    This is not a criticism of the people building these documents. They are working with the tools available to them. The tools were designed before these regulatory frameworks existed, and they were not updated when the requirements changed.


    What Would Actually Work

    Here is what a data delivery format designed for the problems organizations actually face today would need to do.

    It would need to carry data to any recipient through any channel without requiring a platform license or a live connection. It would need to enforce access controls at the individual recipient level so that each person sees exactly what they are supposed to see and nothing else. It would need to maintain a complete audit trail from delivery through every subsequent interaction. It would need to support revocation after the fact. And it would need to work offline, because the real world does not reliably provide connectivity.

    None of these requirements is technically impossible. Parts of them are already solved in various contexts. Encrypted files handle some access control. Portals maintain some audit history. Offline-capable applications exist. The gap is not in any individual capability. It is in a format that delivers all of them together, in a package portable enough to reach any recipient through any channel.

    The PDF solved portability and sacrificed everything else. The portal solved interactivity and sacrificed portability, offline access, and minimized security scope. The spreadsheet preserved raw data and sacrificed governance, usability, and controlled delivery entirely.

    Every format we have was designed to solve one problem, and each one creates a different set of problems downstream. What the enterprise does not yet have, in any mainstream form, is a delivery format designed from scratch around the full set of requirements.


    Why This Keeps Getting Ignored

    Part of what makes this problem persistent is that it does not announce itself loudly. The data gets shared. The report gets sent. The recipient opens something. It works well enough that nobody flags it as broken.

    The failures are quiet. A decision made on incomplete information. A compliance gap that surfaces during an audit. A customer who stops engaging because they never felt they could understand their own account. A vendor relationship that deteriorates because SLA reporting was always a negotiation over what the data actually showed.

    None of these failures get traced back to the format. They get traced back to processes, to people, to systems. The format that connects all of them stays invisible.

    Regulatory pressure is tightening this. The expectation that organizations can demonstrate with precision what data was shared, with whom, when, and under what authorization is moving from advanced capability to baseline audit requirement. That shift will force some hard conversations about formats that have never had to justify themselves.

    When those conversations happen, the answer is not going to be a better PDF.

  • The Hidden Price of Cloud Services: What CDOs and CFOs Must Know About Cloud Data Costs

    The Hidden Price of Cloud Services: What CDOs and CFOs Must Know About Cloud Data Costs

    We sold the board on agility and scale. We convinced the business that cloud would let teams experiment fast, spin up analytics, and iterate toward better decisions. And for the most part, that promise has been real.

    But there’s a quieter truth that doesn’t get as many slide deck minutes: cloud economics are variable, and in a world awash with data, that variability becomes the thing that keeps finance and data leaders awake at night. As both a CDO and CFO across multiple cloud migrations, I’ve seen the pattern too often: data gets created and uploaded cheaply; the expensive part is what we do with it afterward, how often we touch it, how we compute over it, and how and where we move it.

    Below I’ll walk through the behavioral and technical drivers of variable cloud cost, show the critical difference between creating/uploading data and consuming it, point to market data and reporting where possible, describe documented financial impact cases, and close with practical guardrails you can apply now to reconcile speed with fiscal discipline.

    Variable cost is the new normal

    Historically, IT costs were largely fixed: you bought servers, depreciated them, and budgeted for refresh cycles. Cloud flips that script. Storage, compute, and, critically, network transfers are metered. The bill arrives as a sum of thousands of operational decisions: how many clusters ran overnight, which queries scanned terabytes instead of gigabytes, which business intelligence dashboards refresh by default every five minutes.

    This pattern matters because many of those decisions are made by people who think in analytics and velocity, not dollars-per-GB. Engineers and data scientists treat compute as elastic, and they should, for innovation, but the elasticity becomes costly without governance. Recent industry reporting confirms that unexpected usage and egress fees are a leading cause of budget overruns. [1]

    Upload vs. download: the crucial distinction

    Cloud pricing is purposefully asymmetric. Ingress, uploading data into the cloud, is typically free or very cheap. Providers want your data on their platform. Egress, moving data out of the cloud, between regions, or to downstream consumers, is where the economics bite. That’s why uploading billions of log lines feels inexpensive, but serving those logs to users, copying datasets between regions, or exporting terabytes for partner analytics can produce bills that scale in minutes.

    For example: major cloud providers publish tiered network and storage pricing where ingress is minimal and egress ranges by region and destination. Amazon’s S3 pricing pages and general data transfer documentation show free or near-free ingress alongside non-trivial outbound transfer rates that vary by region and tier. [2]
    [3]

    Put differently: storing a terabyte for a month costs one thing; repeatedly reading, copying, or exporting that terabyte is another. A platform that charges separately for compute time (for queries and pipelines), storage, and network transfer will make consumption the dominant lever in your monthly bill. For example, some analytic platforms separate compute + storage + egress explicitly. [4] [5]

    Where consumption surprises come from (and why they compound)

    Consumption overruns aren’t a single root cause, they’re a system. A few common patterns show up repeatedly:

    • Unfettered experimentation. Teams spin up large clusters, train big models, or run broad scans ‘for a test.’ A single heavy job run at full scale can spike costs for the month.
    • Chatty pipelines and duplication. Every copy, transform, and intermediate table multiplies storage and compute. When teams don’t centralize or catalogue datasets, duplicates proliferate and get processed again and again, increasing cost with each duplication.
    • Always-on analytics and reports. Hundreds of dashboards (and linked on demand reports) refreshing by default, real-time streams with high retention, and cron jobs without review all turn predictable activity into persistent cost.
    • Cross-region and multi-cloud traffic. Moving data between regions or providers often carries egress or inter-region fees. That cost is small per GB but large in aggregate, and it’s often invisible until it’s not.
    • AI and ML compute consumption. Training and inference on large models use GPU/accelerator time, which is expensive and scales super-linearly with workload size. [6]

    Industry surveys back this up: finance leaders consistently say a lack of visibility into technical drivers is a main contributor to runaway spending. [7]

    What the market tells us about scale and trajectory

    Two useful frames help here: (1) total cloud spending trends and (2) raw data growth.

    Analyst forecasts show cloud spending continues to accelerate. According to Gartner’s 2025 public-cloud forecast, worldwide end-user spending on public cloud services is projected to exceed US $720 billion, a strong year-over-year jump that underscores how much budget is flowing into cloud platforms. [8]

    On the data side, Fortune Business Insights [9] series quantified the explosion of the global datasphere: past forecasts put the global datasphere in the hundreds of zettabytes by the mid-2020s.  The scale is staggering, tens to hundreds of zettabytes of created, captured, copied, and consumed data, with continuous growth driven by IoT, media, and especially AI workloads that train on massive datasets. Those macro trends mean the base unit (how much data is available to touch) is rising fast which, left unmanaged, makes consumption costs an ever-larger line on the P&L.

    Documented cases of financial impact due to cloud consumption and egress costs

    Several documented cases highlight the financial impact of cloud consumption and egress costs:

    • A large large insurance company that generates over 200,000 customer statements a month are spending over $10,000,000 yearly just on customer statement generation as they pay the server side compute and data egress costs.
    • Data Canopy’s $20,000 monthly egress fees: Data Canopy, a provider of managed co-location and cloud services, was paying $20,000 monthly in egress fees by using VPN tunnelling to connect clients to AWS. VPN routes often introduce latency, lack scalability, and result in unpredictable costs due to fluctuating data-transfer volumes.
    • A startup’s $450,000 Google Cloud bill: A startup reported on the OpenMetal blog received a $450K Google Cloud bill after compromised API keys triggered massive unauthorized transfers in 45 days.
    • $120,000 AWS bill from a stress test: An engineering team set up infrastructure for a product stress test that copied large files from S3 to an EC2 instance. The setup led to a $120,000 AWS bill over the weekend due to data-transfer and compute costs.

    These cases underscore the importance of understanding and managing cloud consumption and egress costs to avoid unexpected financial burdens.

    Hard numbers and egress examples

    Exact per-GB egress numbers vary by provider, region, and tier, and providers publish detailed tiered pricing tables. A representative comparison often quoted shows outbound transfer rates commonly between US $0.05’$0.12 per GB in many regions, with variation for cross-region or inter-cloud transfers. 

    For platform-specific color: some analytic platforms break billing into distinct components (storage, compute, data transfer) so a scan-heavy workload that reads lots of compressed data can run up compute credits far faster than storage alone would suggest. [4]

    Forecast: growth + consumption = more financial focus

    Two simple forces are converging: raw data volumes continue to expand (zettabytes of data in the global datasphere), and enterprises are running more compute-heavy workloads (AI, real-time analytics, large-scale ETL). The combination means consumption bills will grow faster than storage bills. Cloud-spending forecasts (hundreds of billions annually) and rapid AI adoption make this inevitable unless governance catches up. In practice, expect your cloud‐consumption line to be one of the fastest-growing operational expenses over the next 3’5 years unless you adopt stronger cost visibility and control. [8]

    Practical Guardrails for Leaders Who Want Both Speed and Control

    Innovation does not stop because you start measuring costs. But you can innovate more safely. Below are detailed guardrails based on industry feedback:

    1. Real-Time Cost Telemetry + Visibility

    Treat cloud cost as you treat service downtime metrics. Engineers should see cost, usage, and performance side-by-side. For example, when a data scientist launches a heavy job, they should know in real time the incremental cost in dollars, not just cluster hours. Create dashboards that show compute usage, egress GBs, and storage growth with mapped cost. Set alarms for unexpected surges.

    2. Workload Ownership with Showback/Chargeback

    Every dataset, every pipeline, every compute environment needs a ‘budget owner.’ That person or team receives monthly cost summaries, cost variances, and the ability to act. If a team treats the cloud like a sandbox with no accountability, costs balloon. Use tagging and cost-center attribution so every resource is traceable. Monthly cost reviews should include business teams, not just engineering.

    3. Automated Lifecycle & Data Tiering Policies

    Treat data like the asset it is: ephemeral unless activated. Implement rules: dev/test clusters auto-shutdown after inactivity; datasets not accessed for 90 days shift to cold storage or archive; raw ingestion copies truncated or summarized. Remove or archive intermediate copies automatically. Set retention policies aligned to usage and cost thresholds. The fewer idle TBs sitting and refreshing, the smaller the ‘always-on’ burden.

    4. Right-size Compute & Leverage Auto-Scaling / Spot Instances

    Large, fixed clusters are easy but wasteful. Use auto-scaling or spot/pre-emptible instances where appropriate, particularly for non-mission-critical workloads. Enforce policies: cluster size ceiling, job timeout limits, query concurrency limits. Review usage logs monthly to optimize resource sizing and avoid ‘large cluster for test’ scenarios. Encourage cost awareness in engineering planning.

    5. Eliminate Duplication, Enforce Data Catalogue & Reuse

    Multiple copies of the same dataset, processed in isolation across teams, drive duplicate storage and compute. Create a central data catalogue, promote reuse of datasets, and mark copies only when necessary. Standardize ingestion patterns so that processes don’t proliferate ad-hoc pipelines. Encouraging teams to search existing assets before creating new ones reduces waste and cost.

    6. Tagging, Attribution & Forecasting

    Resources without tags are cost-blind. Ensure every cluster, dataset, job has tags for business unit, project, owner, environment (dev/test/prod). Use this to attribute cost, forecast spend based on usage trends, and model scenarios. Don’t treat cloud invoices as ‘job done’ at month’s end, use them as input to forecasting, cost optimization, and decision-making. Run ‘what-if’ modelling: what happens if ingestion doubles? What if egress increases by 50%?

    7. AI/ML Spend Discipline

    Training large models and real-time inference pipelines are expensive. Require clear business use-case and cost estimates before spinning large GPU/cluster jobs. Use smaller batch trials in cheaper environments, then scale only for production. Monitor overarching GPU-hour consumption and set thresholds. Make AI spend visible and subject to the same ownership and budget discipline as ETL or BI pipelines.

    8. Negotiate Committed Use / Savings Plans Where Appropriate

    If you can forecast a baseline level of consumption, negotiate committed-use discounts or savings plans with your cloud provider. Treat that baseline separately from the variable tail. The tail, experimental work, ad-hoc data movement, new analytics, stays uncommitted so you retain agility while limiting surprise.

    9. Capacity Building + Cost Literacy in Data Teams

    Last but not least: make cost behavior part of your data culture. Engineers, architects, analysts should all understand that ‘every query is a financial decision.’ Include cost implications in your onboarding, training, and architecture reviews. Celebrate teams that reduce cost while delivering performance. Make cost reduction visible, not just cost growth.

    Final Word: Treat Consumption as an Operational Discipline, Not a Surprise

    Cloud gives us extraordinary capabilities. But capabilities without constraints create risk. Consumption is a behavioral and architectural problem as much as a pricing problem. The data is growing exponentially; so must our financial stewardship.

    If you are a CDO, your role now includes translating technical choices into economic outcomes. If you are a CFO, your role now includes translating invoices into operational levers that engineers can act on. When those two disciplines converge, when finance and data speak the same language and operate with the same telemetry, cloud becomes less of a gamble and more of a controlled advantage.

    The cloud will continue to win for those who learn to measure not just bytes at rest, but the dollars behind every byte moved and every CPU-second consumed.

    References (summarized)

    1. CIO Dive ‘ Cloud data storage woes drive cost overruns, business delays, Feb 26 2025https://www.ciodive.com/news/cloud-storage-overspend-wasabi/740940/  
    2. Amazon Web Services ‘ Amazon S3 Pricing. https://aws.amazon.com/s3/pricing
    3. Amazon Web Services ‘ AWS Products and Services Pricing. https://aws.amazon.com/pricing
    4. Snowflake Documentation ‘ Understanding overall cost. https://docs.snowflake.com/en/user-guide/cost-understanding-overall
    5. Microsoft Azure ‘ Azure Databricks Pricing. https://azure.microsoft.com/en-us/pricing/details/databricks
    6. CIO Dive ‘ What Wipro’s global CIO learned about AI cost overruns, Oct 6, 2025. https://www.ciodive.com/news/wipro-global-cio-generative-ai-agents-cost-deployment/801943
    7. CIO Dive ‘ Runaway cloud spending frustrates finance execs: Vertice, Sept 26, 2023. https://www.cfodive.com/news/runaway-cloud-spending-frustrates-finance-execs-vertice/694706
    8. CIO Dive ‘ Global cloud spend to surpass $700B in 2025 as hybrid adoption spreads: Gartner Nov 19, 2024 .
      https://www.ciodive.com/news/cloud-spend-growth-forecast-2025-gartner/733401
    9. Fortune Business Insights ‘ Data Storage Size, Share, Forecast, Oct 6, 2025. https://www.fortunebusinessinsights.com/data-storage-market-102991
    10. HelpNetSecurity ‘ Cloud security gains overshadowed by soaring storage fees, Mar 7 2025https://www.helpnetsecurity.com/2025/03/07/cloud-storage-fees/  
    11. ComputerWeekly ‘ Unexpected costs hit many as they move to cloud storage, Mar 5 2024https://www.computerweekly.com/news/366572292/Unexpected-costs-hit-many-as-they-move-to-cloud-storage  
    12. Academic paper ‘ Skyplane: Optimizing Transfer Cost and Throughput Using Cloud-Aware Overlays, Oct 2022. https://arxiv.org/abs/2210.07259  
    13. Gartner – Tame Data Egress Charges in the Public Cloud, Sept 2023. https://www.gartner.com/en/documents/4786031
    14. IDC ‘Future-Proofing Storage, Mar 2021. https://www.seagate.com/promos/future-proofing-storage-whitepaper/_shared/masters/future-proofing-storage-wp.pdf