Slow performance in Azure when in use by multiple users

[OrcaTech]

Hi, I have a puzzling issue with the performance of FoxPro after trying to use it on virtual machines in Azure. This issue occurs on a Remote Desktop VM in Azure and also on our test VM.

Essentially, when there is only one person using FoxPro the performance is fine, but if another person uses it and opens a table it will take minutes to open. This happens when RDPing to the server and then running the app, so it's being run directly on the server itself, but the same thing also happens when the app is run as a published app in Remote Desktop (which is ultimately how we want it to be used).

There shouldn't be any issue with network performance as both the app and the database is on the same drive on the server in Azure. The disks are also premium SKU. Enabling read/write caching on the disks makes no difference.

We even have a test server which has a copy of app and database on a drive on the server and this works fine with the server in the office. But after the server was migrated ('lift and shift') to Azure (the Azure VM specs matched those of the VM in the office), it too has the same performance issue when a 2nd person tries to use FoxPro and opens a table.

I'm not a FoxPro developer, so my knowledge of FoxPro is rather limited, but I've been assured by the developer I am liaising with that things like table buffering, locking, non-exclusive file access and rushmore optimisation have all been checked. As I say, it's working fine on the server in the office, but just has this issue when running on VMs in Azure! AV is Defender and the database files are excluded from AV scanning.

Apologies if this is all quite vague, happy to fill in any more details that you might require. If anyone could assist in figuring out what's causing this performance issue I will be extremely grateful!

Best Regards,

 

[Chris Miller]

both the app and the database is on the same drive on the server in Azure

That's good, in short. Or it should be good. Essentially what prevents the only well-known problem with SMB protocol oplocks in file access to cause the issue of slowing down whenever a secondary user starts an application and uses the same DBFs is avoiding a network between the client executing the EXE and the DBF files.

Or the other way around: The simplest solution to the oplocks problem aside from managing to turn them off is to avoid their usage, because this is a network performance acceleration construct failing for the way VFP accesses DBFs.

The most immportant point is not an EXE being on the same drive as DBFs, it's that an EXE starts on a terminal servers CPU (or virtual CPU in a VM, doesn't matter) and to that real or virtual computer (node) the DBFs are on local drives, i.e. file access does not need to go through network protocols. The often used construct of EXE and data on a share is not meaning local data access of the EXE, as clients load the EXE through the network and then data access also through that network.

I don't know, maybe the lift and shift of a VM to Azure is done in a way that the virtual hard drives are networked within the virtual environment, that's not making them local, even if they are simulated on the same VM host, i.e. the bad side effect of SMB protocol oplocks also happens in a virtual network.

Chriss

 

[OrcaTech]

Hi Chris,

Thanks for the quick reply and explanations, much appreciated. So looks like we can rule out it being an oplocks issue then.

The lift and shift was just done for the test VM. The other Azure VM (remote desktop VM) was created within Azure, not migrated. And both have the same issue. But as a test, I will copy the app and database to the C drive of the Azure test VM and try running it from there. That should rule out any connectivity/performance issue with running the app on the secondary drive.

Best Regards,

 

[OrcaTech]

Just an update - it doesn't make any difference if the app and database are located on the C drive of the VM. So I can rule out the app and database being on a separate disk causing any performance issue.

 

[Chris Miller]

Interesting, then you have a new problem, not oplocks/SMB. I understand you don't have the problem in the same configuration on a real server and users connecting to it via remote desktop?

Chriss

 

[OrcaTech]

Yes that's right, the issue can't be reproduced running FoxPro on the servers in the office. There is a Remote Desktop server in the office that some users use for accessing FoxPro remotely, but users mostly just run it from their own workstations by running the exe that's on an office server share (this share contains the app and database). The issue has only come to light since trying to move to Azure and run the app on an Azure server.

I've just been reading about 'Sys(3050)'. I'm not sure if this setting has changed after being installed in Azure (I can't imagine anyone has changed it, but will check). Maybe this needs tweaking?

Best Regards,

 

[Chris Miller]

If you don't explicitly set SYS(3050) low, Rushmore will have enough memory for its optimization. That's not likely to be the reason. It would also slow down your on premise situation, not only the Azure VM.

It seems to be VM related, doesn't it? So what I'd look into is, whether the second user actually connects to the VM the first user starts up or whether Azure starts a copy of a base VM for every user. I think that would also not work out for the data then is local to each VM user and not shared. It would only explain a performance hit for the second user at starting the application, not using the same table.

Chriss

 

[Chris Miller]

users mostly just run it from their own workstations by running the exe that's on an office server share (this share contains the app and database)

By the way, that's still a situation with networked data, and would have the SMB oplocks issue. If you don't have problems with that, then what's even the motivation to go Azure? Just general IT/server migration into the cloud?

Chriss

 

[OrcaTech]

what's even the motivation to go Azure? Just general IT/server migration into the cloud?

Yes, it's part of a larger migration to Azure. All other company apps and services have been migrated to Azure.

So what I'd look into is, whether the second user actually connects to the VM the first user starts up or whether Azure starts a copy of a base VM for every user.

The issue is happening on a several VMs I've spun up in Azure, not just a Remote Desktop (Azure Virtual Desktop) VM. The Azure Virtual Desktop configuration is set to 'Pooled' not 'Personal', so all users are running apps on the same VM (there is only a single VM in the pool).

Best Regards,

 

[Chris Miller]

Okay, so the rough situation is your application works without performance issues locally, even networked or with a remote desktop setup within your LAN.

Setup within an Azure VM that's a pooled virtual desktop configuration where you put data into a local drive of the desktop has a performance issue for a secondary user. It can't be explained by missing indexes or suboptimal Rushmore optimization or otherwise, you would have the same issue with the local setups.

Then all that's available is an issue with the VM. When you do remote desktop for multiple users you have to have enough resources to support multiple user sessions, that's all that remains for me.

I'm afraid I have no further ideas to investigate, but as far as it's outlined the fault needs to be in Azure. If I were in your situation I would try to prove it's Azure by creating a similar setup with Google cloud services and if that has no issues ask for support about the Azure performance issue from Microsoft.

Chriss

 

[OrcaTech]

Thanks Chris, I appreciate your input. At least we've been able to rule out some things. I'm not sure MS support will be able to assist though, as FoxPro is no longer a supported product.

 

[Chris Miller]

Well, don't make it a support call about VFP, make it a support call about VM, Azure itself. It won't matter what type of EXE you have in the VM, if you can show them it runs much faster within a Google cloud VM than in their Azure, they have the motivation to look into it.

Chriss