Introduction

We had our Community Day in Hamburg on the 15^th of March – we hold this in German but as I present the troubleshooting session and didn’t use slides, I decided to write a blog about this topic to ensure that not just German Speaking people understand what we did here.

(Video of the Session (Language is in German
https://youtu.be/R_JDKXctW5A )

Scenario

Imagine a Scenario where you launch you new VDI Image on Monday morning. I took one participant and gave him the task to run the App-V app, and to add a package from a local folder. Now the fun begins…

Problem

Starting an already published App-V Application shows this:

Trying to add a Package manually shows this error:

What is happening?

The first error could be a bit misleading pointing to VC++ components. The second error is much more straightforward and points to the App-V service not running. So let’s dig in:

Troubleshooting

Let’s try to start the service in PowerShell:

This is not helping much – so we do it the old style through the services.msc:

Ok, this is much more useful as it points us to a dependency service or a group of services which are not starting.

What dependencies do we have for App-V Client and how do we find them?

We have the services.msc open already, so use it and check the dependencies for the App-V Client service:

Now I know the dependencies but how do I know which of them is causing this?

Windows brings a tool called FLTMC – if you run this as Administrator on this box, you get the following output:

This shows us that the AppvStrm Driver is loaded but the AppVVFS and the Appvvemgr filter drivers are not there. As next step we need to focus on the AppvVemgr, as this is also a dependency of AppvVfs.

Let’s try to start it…

Now we are getting closer and closer – the actual error is a “File not found” error.

What is the first reaction of all engineers when they see this? – “Sure thing – the driver file is missing”

Ok – let’s check that.

Ok it’s there, now we need the tool we usually need to any type of troubleshooting…

PROCMON

I added a small tips section at the bottom of this post where I show you how to make Procmon show you more, which is not needed for this case but good to know…

How do we use Procmon?

What to trace? How to filter? There are so many events; what is important and what is not important? Questions over questions which gives some fear when we need that tool – but no worry, we will guide you through it here.

So, we start Procmon and start the capture and we just capture the startup of the driver (AppvVemgr.sys) as shown above.

When we have done that, we stop the capture and now we need to isolate the issue. At this point there are thousands of ways to dig into this deeper…. I show you one method which helps me in most of the cases. Let’s call it the Rewind Method.

Analysis with Procmon using Rewind and Assumption

In the default Setting I have ~ 94 K Events collected, and my actual view displays 4.5 K.

First, I go to the very last event as the issue happens nearly at that time when I stop the Trace.

Now I need to isolate my view a bit and here I need my knowledge. What do I know about the error? It says something was not found… OK, what have I done to get the error? I tried to start the AppvVemgr driver.

Which Process is actually starting the drivers? You see, with that knowledge that you get out of the error you can build an filter based on assumptions.

What does the filter look like?

After applying this filter, I got back four events:

This result matches 2 of three conditions we were thinking about:

1. I see the “not found”
2. It has to do with the driver we are looking for.

Now which Process is starting drivers? It’s not Services.exe it’s System. We don’t see anything from System here. If we would lookup that registry key on a working system we would see that this value “ObjectName” doesn’t exist so it’s not our root cause here.

Where is System?

Because it’s excluded by our view we don’t see it, so let’s make it look up either by changing the actual filter and remove this entry:

Or enable the Advanced Output from Procmon:

As we just want to focus on System we do the first option here. In addition, we modify our filter and no longer filter just on the driver (as the startup of the driver can look somewhere else to come up and we will ensure not to miss this). But as we want to know why the AppV driver is not coming up, and as a result the AppV service – we filter on Appv. The filter looks like this now:

After applying this filter, we have 15 results, where 5 are coming from the App-V registry key:

Time to take a closer look on the MAV keys in the registry of the machine.

Now when we go through the entries under Packages we notice that there is a difference:

All Packages look like this above except one:

This was also already shown in the Procmon above…

So what to do – as we are still in troubleshooting mode, we delete that key to see if this has any effect, and start the driver afterwards to see if this is now working:

So our troubleshooting method has a positive outcome now – the service is coming up fine.

TIP:

If you check the FLTMC output again, the drivers running on the system after all this troubleshooting are shown below:

What does this now mean – when you collect a Procmon trace you just see what is above the Procmon driver. In the case above, we would see what AppvVFS is doing but we would not see what the Appvvemgr driver or the Appvstrm driver will do.

If needed you must change the altitude of Procmon to run on a lower altitude. You can do this by open Regedit and find the used Procmon driver:

Change the Altitude value to 45100 (which will show you virtually everything that is happening on the machine).

You must also set the security on the “Process Monitor Instance” key and add deny rights for everyone for “delete” and “set value”. Reason being that Procmon will try to change its value back right away. You will have to uncheck “inherit permissions” to be able to set them at the Process Monitor Instance level.

Now after restarting the machine and starting Procmon, it’s running on the lower altitude.

Kind regards

Sebastian Gernert – Escalation Engineer App-V