Vin Vashishta

Why aren’t we living in a Generative AI-powered utopia yet? LLMs are harder to implement and deploy than most businesses expect.

It takes 6-12 months to come up to speed, develop, and release a customer-facing Generative AI app. It’ll take longer if the business’s data isn’t curated to support advanced model training.

For internal use cases, it can take as little as 2-3 months to deliver reliable solutions. Products are much harder because they must be more reliable and deliver a superior user experience.

Internal-facing LLM apps can get away with cutting corners that a customer won’t tolerate. Publicly available apps must also account for adversarial scenarios and intentional misuse.

Mastering Generative AI for internal use cases is the easy part. If the business is planning to build for customer-facing products, add time to account for the differences.

Ignore the hype about how easy it is to build LLM apps when it comes to customer-facing products. It’s a completely different process, and development doesn’t end once the product ships. There truly is no finish line.

#generativeAI #aistrategy #productmanagement

The job interview process is more toxic than Chornobyl in the Spring of 87. Here’s what I’m hearing from senior-level data professionals going through it.

Interviews are adversarial. People treat candidates like the enemy, not potential future teammates.

The process doesn’t respect the candidate’s time. It spans multiple rounds, and the work assignments can take over a day to complete.

Questions don’t relate to the job itself. Interviewers ask complex deep learning theory and implementation questions, but the team doesn’t apply those concepts.

The posted salary range is often misleading. Offer letters have salaries below the range, with bonuses barely pushing the total compensation into the low end.

Experienced professionals are going through what juniors have been for years. Most refuse to jump through the ridiculous hoops and find roles at more mature businesses.

If you’re wondering why your company can’t get talent when tons of amazing data professionals are looking for their next challenge, it’s probably your hiring process.

#datascience #hiringprocess

I’ve hired just over 50 data scientists & sat in on 3 times that number of interviews. While I have an array of questions to assess technical proficiency, I only need 1 to assess a candidate’s value to the business. “Give me a user persona from one of your previous projects.”

I get 3 kinds of answers. The first is confusion, “What’s that?” followed by, “Ah, gotcha. I didn’t interact with the users in my last job.” This response is an interview ender because this candidate will add no value to the business. A data scientist who’s that disconnected from users isn’t likely to produce anything a user would value.

The second is a few personas. “Our primary user was… & we also looked at…” These are good answers because the candidate took the time to remember who they were building for.

What gets me to lean in is the third answer. “Let me tell you about Carol…” When a data scientist has an actual user in mind, that’s my gal or guy.

I can train a candidate on algorithms or programming languages. It’s much harder to instill a genuine interest in the user. That interest leads to business value more than any other skill I’ve seen. Data scientists who care enough to talk to users are worth their weight in gold. Treat them accordingly.

People trying to break into data science or move ahead are held back by 1 mental mistake more than any other.

Knowledge of machine learning models and math are not in-demand skills. Neither is knowledge of Python, Java, R, and SQL. It is a mistake to focus your data science career path or learning journey on any of those.

There is only 1 in-demand skill; the ability to apply technology to drive an outcome or create value. Businesses need results, not technology. Customers buy functionality and utility, not technology.

Businesses hire people who are focused on outcomes and value creation. Stop emphasizing skill words and start emphasizing what you can do with those words.

Your job search and career path will take off as soon as you make the change.

#datascience #data #analytics #career

We’re still a long way from models that perform better than people, so that can’t be the standard. Why does everyone judge machine learning and deep learning against a perfect human?

Data scientists and researchers don’t talk about reliability or performance limitations. ChatGPT is different. It has an early version of a reliability framework.

When you ask it to predict the future or any number of other cases, ChatGPT will return a canned response redirecting the user to supported functionality. That’s a blueprint we should all be following.

It takes time to assess model reliability and decide what categories are supported. We need model explainability and QA frameworks to support the effort, but it has become much more feasible over the last 3 years.

Advertise model reliability standards. Implement failover behavior for unsupported classes or categories. Inform and redirect users when they get into unpredictable territory.

#datascience #machinelearning #deeplearning

Google is better at AI than Microsoft. They have better models, but Microsoft’s releases are outperforming Google’s. How is that possible?

Microsoft is better at productizing and commercializing AI. The fact that Microsoft can beat a more advanced competitor shows how critical those two capabilities are.

A competitor that is better at delivering solutions to customers will win even with less advanced capabilities and models. To succeed, AI products must:

1️⃣ Fit into customers’ workflows and meet a need they’re willing to pay for.
2️⃣ Fit the business model and how the company currently monetizes products.
3️⃣ Have a path to production that includes integration into existing product lines and support for continuous improvement post-release.
4️⃣ Meet not only functional but also reliability requirements.

Most businesses tackle these challenges last when it costs the most to resolve any problems they uncover. They call this the AI last mile problem, but it’s really a first mile problem.

Monetization, productization, and commercialization must happen upfront. Businesses need playbooks and frameworks to manage all 3 pieces without slowing initiatives down.

Data and AI Product Managers are critical success factors. Without them, no one is accountable for those 3 pieces, and they don’t get done until after the model is delivered.

#datascience #productmanagement #datastrategy

Some Twitter engineers are getting called back after being laid off. This is fairly common, and here’s what you should do if your employer has second thoughts after letting you go.

Offer hourly consulting. If you’re already into the job hunt and interviewing, hourly consulting can be an excellent way to double dip. You avoid the risk of returning to a job only to be let go again a short time later.

What rate should you ask for? If your former employer is in a tough spot, ask for 3X-5X your salaried hourly. Otherwise, 150% is standard.

Offer to return if they give you a golden parachute. You’re taking a risk by coming back to a role that’s obviously on the chopping block. Ask them for a 3 - 6 month guaranteed severance package.

Ask for a raise and/or promotion. If you’re planning to accept, why not ask? You’ve already been fired. They can’t do it twice, so there’s not much to lose.

If they had options, they wouldn’t be calling you. It’s essential to keep that in mind. Before saying “No,” ask yourself what it would take to accept. Whatever answer you come up with, make them the counteroffer.

I’ll come back if… I have seen companies in this spot accept some very big asks.

#layoffs #twitter

Data scientists who can write production-grade code don’t understand the models, and those who understand the models don’t know how to code. A data science manager wrote that on Reddit last night. There is an easy solution to that problem.

Pair them up. Have them work on their projects together for about 6-12 months. They code together, evaluate problem and solution spaces, evaluate trained models, build experiments, and so on.

That’s all it takes. Training and mentorship are part of leading. Few people on the team are fully baked. We put the frameworks in place to bring people from where they are to the next level. This is a core leadership function.

Directors and higher train leaders. I hear struggles like this one from data science managers all the time. They need training and mentorship just like their team does. I have been a leader for over 15 years. My mentors have taught me how to:

Communicate across the business
Lead through adversity
Develop ICs and leaders
Mentor people into unlocking their potential
Transition from leading small teams to large teams to business units and build all three of those
Hire and create hiring processes
Develop values, vision, purpose, and strategy
Creating an innovation pipeline
And on and on…

Don’t wait for perfect people. Develop them. Don’t leave your leaders hanging. They need development. It is a never-ending cycle. Every leader is part of the process.

#data #leadership #analytics #datascience