# LLMs Are Here. The Real Work Starts Now.

*Discussing the Challenges of GenAI Implementation*

By [ByteByByte](https://bytebybyte.tech) · 2025-02-05

artificialintelligence, llms, aiops, enterpriseai, aigovernance, techleadership

---

I was chatting with a colleague about GenAI/LLMs recently and figured I’d share my thoughts here.

We were discussing the challenges many companies face when implementing LLMs. His perspective is that many organizations are overly focused on LLM reasoning capabilities. As a result, **many companies and technology leaders are waiting for the models to reach a near-perfect level of capability before putting their hats in the ring and joining the GenAI wave.** He also pointed out the [limitations LLMs](https://www.nytimes.com/2024/12/19/technology/artificial-intelligence-data-openai-google.html) have been running into.

He argued that LLMs are already powerful (I agree), but their effectiveness is largely determined by the context provided (I also agree - think [RAG](https://www.databricks.com/glossary/retrieval-augmented-generation-rag)). Innovative companies, he noted, are investing in collecting and organizing their structured and unstructured data while orchestrating LLMs effectively (again...I agree). That’s why he sees data preparation and structuring tools as particularly valuable right now. (Again, I agree.)

From my experience, 60% of the work that goes into data science solutions is data preparation and exploratory data analysis (EDA). Leveraging data preparation tools backed by GenAI to accelerate that process is a major opportunity right now.

**My issue with his point of view is that while some companies - or individuals within them - might be waiting for LLMs to improve, I don’t think most companies are.**

During my time supporting GenAI strategy and governance efforts, **I've seen three primary challenges that companies face when executing GenAI strategies and building GenAI tools**:

1.  **Managing what’s been built** – Once an LLM solution or product is developed, how do we operationalize it? What’s the “Ops” in DevOps, and does my organization have the right teams and resources to support and scale it effectively?
    
2.  **Understanding the GenAI black box** – Many companies struggle with visibility into how LLMs generate their outputs. This is especially critical for regulated industries like healthcare, utilities, and financial services, where understanding what data the model is using and ensuring accuracy are non-negotiable. We all remember the lawyer who unknowingly cited false legal precedents because an LLM hallucinated.
    
3.  **Ensuring adoption** – Beyond the technology itself, organizations struggle with governance and managing the change required for business users to adopt and sustain GenAI solutions successfully. Without these foundational elements, even the best models risk failing to deliver meaningful impact - or worse, being misused to the detriment of the company.
    

LLMs are powerful tools, and many companies recognize that. The primary challenge isn't whether LLMs are ready - it’s whether organizations are ready. Success in GenAI requires thoughtful execution, operationalization, and governance. While some technical leads and enterprise architects may be waiting for LLMs to advance, most businesses are grappling with how to manage, interpret, and adopt these new tools effectively.

---

*Originally published on [ByteByByte](https://bytebybyte.tech/llms-are-here-the-real-work-starts-now)*