03Explorer
Technical User
Simple explanation of my project. I have nested loops that each are based on a linked table query (source = SQL Server tables).
Loop1 start
| Loop2 start
| | Loop3 start
| | | Loop4 start
| | | Loop4 end
| | | Loop5 start
| | | Loop5 end
| | Loop3 end
| | Loop6 start
| | | Loop7 start
| | | Loop7 end
| | | Loop8 start
| | | Loop8 end
| | Loop6 end
| Loop2 end
Loop1 end
Loops 5 & 8 are my destination points that populate a table based on the criteria generated from the outer loops.
I am experiencing two things that surprise me. The time to execute this VBA (using same criteria that define the outer loops) has such a varying run time. I have two groupings of test runs, one on the LAN and one via VPN (Off-site). The location is outside this discussion. If I run the script from LAN the execution time can vary anywhere from a few minutes to over 10 minutes. Even when executing back to back times. Very inconsistent from time to time or even day to day.
The second experience is this execution takes anywhere from 4 minutes up to 30 minutes. The information is in reference tables housed in SQL Server. I am expecting the output to be no more than 4 minutes based on queries execution time, when done manually, is instantaneous.
1. Where is the bottleneck in the design that causes this to take long to execute?
2. Why is the execution time so random for times for running?
Here is my function that contains the loops
Loop1 start
| Loop2 start
| | Loop3 start
| | | Loop4 start
| | | Loop4 end
| | | Loop5 start
| | | Loop5 end
| | Loop3 end
| | Loop6 start
| | | Loop7 start
| | | Loop7 end
| | | Loop8 start
| | | Loop8 end
| | Loop6 end
| Loop2 end
Loop1 end
Loops 5 & 8 are my destination points that populate a table based on the criteria generated from the outer loops.
I am experiencing two things that surprise me. The time to execute this VBA (using same criteria that define the outer loops) has such a varying run time. I have two groupings of test runs, one on the LAN and one via VPN (Off-site). The location is outside this discussion. If I run the script from LAN the execution time can vary anywhere from a few minutes to over 10 minutes. Even when executing back to back times. Very inconsistent from time to time or even day to day.
The second experience is this execution takes anywhere from 4 minutes up to 30 minutes. The information is in reference tables housed in SQL Server. I am expecting the output to be no more than 4 minutes based on queries execution time, when done manually, is instantaneous.
1. Where is the bottleneck in the design that causes this to take long to execute?
2. Why is the execution time so random for times for running?
Here is my function that contains the loops
Code:
Dim db As DAO.Database
Dim rs, rsPARENT, rsCHILD, rsServiceType, rsSkills, rsConsultingMatrix, rsSurveyGenerated As Recordset
Dim SQL, SQL2, SQL3, SQL4 As Variant
Dim origPractice As Integer
Dim newPractice As Integer
Dim Client As Variant
Dim Practice As Integer
Dim ShowPractice As Integer
Dim StartTime As Date
Dim n As Integer
Dim s As String
n = FreeFile()
Open "C:\temp\Synergy3" & "_" & SurveyNumber & ".txt" For Output As #n
Set db = CurrentDb
s = "starting Matrix: " & Format(Now, "h:nn:ss AM/PM")
Debug.Print s
Print #n, s
'get information pre looping
'original Practice
SQL = "SELECT PracticeID FROM tblProjectDetails WHERE aID = " & SurveyNumber & " ;"
Set rs = CurrentDb.OpenRecordset(SQL)
origPractice = rs!PracticeID 'practice will be used as base for what was selected and showpractice is for calculating
'ClientID
SQL = "SELECT ClientID FROM tblProjectDetails WHERE aID = " & SurveyNumber & " ;"
Set rs = CurrentDb.OpenRecordset(SQL)
Client = rs!ClientID
'rs.Close
'Set rs = Nothing
'Parent Loop rsPARENT : [URL unfurl="true"]https://stackoverflow.com/questions/5864160/code-to-loop-through-all-records-in-ms-access[/URL]
SQL = "SELECT RoleID, AssociateID FROM tblProjectsAssignedTo WHERE SurveyNumberID = " & SurveyNumber & " ORDER BY 1 ASC;"
Set rsPARENT = CurrentDb.OpenRecordset(SQL)
'Check to see if the recordset actually contains rows
If Not (rsPARENT.EOF And rsPARENT.BOF) Then
rsPARENT.MoveFirst 'Unnecessary in this case, but still a good habit
Do Until rsPARENT.EOF = True
' Debug.Print "Call Parent SQL " & Format(Now, "h:nn:ss AM/PM")
'Child Loop rsCHILD
'Debug.Print "Parent: " & rsPARENT!AssociateID & ", Role: " & rsPARENT!RoleID
SQL2 = "SELECT RoleID, AssociateID FROM tblProjectsAssignedTo WHERE SurveyNumberID = " & SurveyNumber & " ORDER BY 1 ASC;"
Set rsCHILD = CurrentDb.OpenRecordset(SQL2)
'Check to see if the recordset actually contains rows
If Not (rsCHILD.EOF And rsCHILD.BOF) Then
rsCHILD.MoveFirst 'Unnecessary in this case, but still a good habit
Do Until rsCHILD.EOF = True
'set parent / Child elapse timer
StartTime = Now
s = "Parent Role = " & rsPARENT!RoleID & " Child Role = " & rsCHILD!RoleID & " Time: " & Format(Now, "h:nn:ss AM/PM")
Debug.Print s
'' Print #n, s
'Debug.Print "Parent: " & rsPARENT!AssociateID & ", Role: " & rsPARENT!RoleID & " Child: " & rsCHILD!AssociateID & ", Role: " & rsCHILD!RoleID
'ServiceType Practice Loop rsServiceType (Technical; but NOT Behavioral skills)
s = "Technical Begins: " & Format(Now, "h:nn:ss AM/PM")
Debug.Print s
'' Print #n, s
SQL3 = "SELECT ServiceTypeID FROM tblProjectServiceTypes WHERE ProjectQueueForLcID = (SELECT Max(aID) FROM tblProjectQueueForLCs WHERE SurveyNumberID = " & SurveyNumber & ") ORDER BY 1 ASC;"
Set rsServiceType = CurrentDb.OpenRecordset(SQL3)
'Debug.Print "Call ServiceType SQL " & Format(Now, "h:nn:ss AM/PM")
'Check to see if the recordset actually contains rows
If Not (rsServiceType.EOF And rsServiceType.BOF) Then
rsServiceType.MoveFirst 'Unnecessary in this case, but still a good habit
Do Until rsServiceType.EOF = True
'Debug.Print "ServiceTypeID: " & rsServiceType!ServiceTypeID
'Consulting Matrix Part 1: Technical Skills
SQL4 = "SELECT Technical FROM tblConsultingMatrix WHERE ParentRole = " & rsPARENT!RoleID & " AND ChildRole = " & rsCHILD!RoleID & " ;"
Set rsConsultingMatrix = CurrentDb.OpenRecordset(SQL4)
'Check to see if the recordset actually contains rows
If Not (rsConsultingMatrix.EOF And rsConsultingMatrix.BOF) Then
rsConsultingMatrix.MoveFirst
Do Until rsConsultingMatrix.EOF = True
'ShowPractice logic here for child
Select Case rsPARENT!RoleID
Case 7, 6, 5
ShowPractice = origPractice
Practice = 5
Case 4
SQL = "SELECT aID FROM tblSkills WHERE practiceID = " & origPractice & " AND ProjectRoleID = " & rsCHILD!RoleID & " AND ServiceTypeID = " & rsServiceType!ServiceTypeID
Set rs = CurrentDb.OpenRecordset(SQL, dbOpenDynaset, dbFailOnError + dbSeeChanges)
'Debug.Print "Call skills Role4 SQL " & Format(Now, "h:nn:ss AM/PM")
If Not (rs.EOF And rs.BOF) Then
'records exist
ShowPractice = origPractice
Practice = origPractice
Else
'records do NOT exist
ShowPractice = origPractice
Practice = 5
End If
Case 3, 2, 1
ShowPractice = origPractice
Practice = origPractice
End Select
'Technical Skills to be populated HERE
'Behavioral needs be done outside/after ServiceType Loop
If rsConsultingMatrix!Technical = 1 Then
''Debug.Print "Begin Technical Skills " & Format(Now, "h:nn:ss AM/PM")
SQL = "SELECT aID, SkillNo, Skill, SkillVersion, BusinessTypeID, ShortForm "
SQL = SQL + "FROM tblSKills "
SQL = SQL + "WHERE PracticeID = " & Practice & " AND ProjectRoleID = " & rsCHILD!RoleID & " AND "
SQL = SQL + "SkillTypeID = " & rsConsultingMatrix!Technical & " AND ServiceTypeID = " & rsServiceType!ServiceTypeID & " AND "
SQL = SQL + "SkillVersion = (SELECT MAX(SkillVersion) FROM tblSkills WHERE PracticeID = " & Practice & " AND ProjectRoleID = " & rsCHILD!RoleID & " AND SkillTypeID = " & rsConsultingMatrix!Technical & " AND ServiceTypeID = " & rsServiceType!ServiceTypeID & ") AND "
SQL = SQL + "ActiveYn = 1 AND BusinessTypeID = 1 "
SQL = SQL + "ORDER BY SkillNo ASC "
Set rsSkills = db.OpenRecordset(SQL, dbOpenDynaset, dbFailOnError + dbSeeChanges) '--, dbSeeChanges)
'Debug.Print "Skill Recordset Defined: " & Format(Now, "h:nn:ss AM/PM")
If Not (rsSkills.EOF And rsSkills.BOF) Then
rsSkills.MoveFirst
Do Until rsSkills.EOF = True
'' Set rsSurveyGenerated = CurrentDb.OpenRecordset("tblSurveysGenerated", dbOpenDynaset, dbFailOnError + dbSeeChanges)
'' rsSurveyGenerated.AddNew
'' rsSurveyGenerated!SurveyNumberID = SurveyNumber
'' rsSurveyGenerated!ClientNumberID = Client
'' rsSurveyGenerated!ParentID = rsPARENT!AssociateID
'' rsSurveyGenerated!ChildID = rsCHILD!AssociateID
'' rsSurveyGenerated!SkillAutoID = rsSkills!aID
'' rsSurveyGenerated!SkillOrderNumber = rsSkills!SkillNo
'' rsSurveyGenerated!SkillVersion = rsSkills!SkillVersion
'' rsSurveyGenerated!ShowPracticeID = ShowPractice
'' rsSurveyGenerated!PracticeID = Practice
'' rsSurveyGenerated!RoleID = rsCHILD!RoleID
'' rsSurveyGenerated!SkillTypeID = rsConsultingMatrix!Technical
'' rsSurveyGenerated!ServiceTypeID = rsServiceType!ServiceTypeID
'' rsSurveyGenerated!BusinessTypeID = rsSkills!BusinessTypeID
'' rsSurveyGenerated!SurveyCreatedDate = Date
'' rsSurveyGenerated.Update
'
SQL2 = "INSERT INTO tblSurveysGenerated (SurveyNumberID, ClientNumberID, ParentID, ChildID, SkillAutoID, SkillOrderNumber, SkillVersion, ShowPracticeID, PracticeID, RoleID, SkillTypeID, ServiceTypeID, BusinessTypeID, SurveyCreatedDate) "
SQL2 = SQL2 + "VALUES (" & SurveyNumber & ", " & Client & ", " & Chr(34) & rsPARENT!AssociateID & Chr(34) & ", " & Chr(34) & rsCHILD!AssociateID & Chr(34) & ", " & rsSkills!aID & ", " & rsSkills!SkillNo & ", " & rsSkills!SkillVersion & ", " & ShowPractice & ", " & Practice & ", " & rsCHILD!RoleID & ", " & rsConsultingMatrix!Technical & ", " & rsServiceType!ServiceTypeID & ", " & rsSkills!BusinessTypeID & ", '" & Date & "') "
db.Execute SQL2
' DoCmd.SetWarnings False
' DoCmd.RunSQL SQL2
' DoCmd.SetWarnings True
rsSkills.MoveNext
Loop 'Skills
Else 'Skills
'Do not populate Skills
End If 'Skills
'Debug.Print "T:Skills done " & Format(Now, "h:nn:ss AM/PM")
Else 'Technical Skills
'Do not populate Technical Skills (0's)
End If 'Technical Skills
'Debug.Print "End ConsultingMatrix : " & Format(Now, "h:nn:ss AM/PM")
rsConsultingMatrix.MoveNext
Loop 'ConsultingMatrix
Else 'ConsultingMatrix
'Debug.Print "There are no records in the Consulting Matrix recordset."
End If 'ConsultingMatrix
'Debug.Print "Finished looping through CONSULTING MATRIX. " & Format(Now, "h:nn:ss AM/PM")
rsServiceType.MoveNext
Loop 'rsServiceType
Else 'rsServiceType
'Debug.Print "There are no records in the SERVICE TYPE recordset. " & Format(Now, "h:nn:ss AM/PM")
End If 'rsServiceType
s = "Technical Ends: " & Format(Now, "h:nn:ss AM/PM")
Debug.Print s
'' Print #n, s
'Begin Behavioral Skills HERE (OUTSIDE SERVICE TYPES)
'ServiceType Loop rsServiceType (Behavioral; NOT Technical skills)
s = "Behavioral Begins: " & Format(Now, "h:nn:ss AM/PM")
Debug.Print s
'' Print #n, s
SQL3 = "SELECT DISTINCT ServiceTypeID, ServNo FROM tblSkills "
SQL3 = SQL3 + "WHERE ProjectRoleID = " & rsCHILD!RoleID & " AND PracticeID in (5," & Practice & ") AND SkillTypeID = 2 AND ActiveYn = 1 AND BusinessTypeID = 1 "
SQL3 = SQL3 + "ORDER BY ServNo ASC"
Set rsServiceType = CurrentDb.OpenRecordset(SQL3)
'Check to see if the recordset actually contains rows
If Not (rsServiceType.EOF And rsServiceType.BOF) Then
rsServiceType.MoveFirst 'Unnecessary in this case, but still a good habit
Do Until rsServiceType.EOF = True
'Consulting Matrix Part 2: Behavioral Skills
SQL4 = "SELECT Behavioral FROM tblConsultingMatrix WHERE ParentRole = " & rsPARENT!RoleID & " AND ChildRole = " & rsCHILD!RoleID & " ;"
Set rsConsultingMatrix = CurrentDb.OpenRecordset(SQL4)
'Check to see if the recordset actually contains rows
If Not (rsConsultingMatrix.EOF And rsConsultingMatrix.BOF) Then
rsConsultingMatrix.MoveFirst
Do Until rsConsultingMatrix.EOF = True
'---ShowPractice logic here for child
'Roles 7,6,5 are typically practice 'ALL' (5)
'Role 4 can be ALL(5) or other
'Roles 3,2,1 are typically Practice
Select Case rsPARENT!RoleID
Case 7, 6, 5
ShowPractice = origPractice
Practice = 5
Case 4
SQL = "SELECT aID FROM tblSkills WHERE practiceID = " & Practice & " AND ProjectRoleID = " & rsCHILD!RoleID & " AND ServiceTypeID = " & rsServiceType!ServiceTypeID
Set rs = CurrentDb.OpenRecordset(SQL, dbOpenDynaset, dbFailOnError + dbSeeChanges)
'Debug.Print "Call B:Skills Role 4 SQL " & Format(Now, "h:nn:ss AM/PM")
If Not (rs.EOF And rs.BOF) Then
'records exist
ShowPractice = origPractice
Practice = origPractice
Else
'records do NOT exist
ShowPractice = origPractice
Practice = 5
End If
Case 3, 2, 1
ShowPractice = origPractice
Practice = 5
End Select
If rsConsultingMatrix!Behavioral = 1 Then ' 1 = Yes; 2 = No
'Yes do Behavioral Skills
SQL = "SELECT aID, SkillNo, Skill, SkillVersion, BusinessTypeID "
SQL = SQL + "FROM tblSKills "
SQL = SQL + "WHERE PracticeID = 5 AND ProjectRoleID = " & rsCHILD!RoleID & " AND "
SQL = SQL + "SkillTypeID = 2 AND ServiceTypeID = " & rsServiceType!ServiceTypeID & " AND "
SQL = SQL + "SkillVersion = (SELECT MAX(SkillVersion) FROM tblSkills WHERE PracticeID = 5 AND ProjectRoleID = " & rsCHILD!RoleID & " AND SkillTypeID = 2 AND ServiceTypeID = " & rsServiceType!ServiceTypeID & ") AND "
SQL = SQL + "ActiveYn = 1 AND BusinessTypeID = 1 "
'if Role = CRM(7), add filter for ShortForm
If rsCHILD!RoleID = 7 Then
SQL = SQL + "AND ShortForm = 1 "
Else
'Do nothing
End If 'ShortForm
SQL = SQL + "ORDER BY SkillNo ASC "
Set rsSkills = db.OpenRecordset(SQL, dbOpenDynaset, dbFailOnError + dbSeeChanges)
If Not (rsSkills.EOF And rsSkills.BOF) Then
rsSkills.MoveFirst
Do Until rsSkills.EOF = True
'Begin populating Behavioral Skills
SQL2 = "INSERT INTO tblSurveysGenerated (SurveyNumberID, ClientNumberID, ParentID, ChildID, SkillAutoID, SkillOrderNumber, SkillVersion, ShowPracticeID, PracticeID, RoleID, SkillTypeID, ServiceTypeID, BusinessTypeID, SurveyCreatedDate) "
SQL2 = SQL2 + "VALUES (" & SurveyNumber & ", " & Client & ", " & Chr(34) & rsPARENT!AssociateID & Chr(34) & ", " & Chr(34) & rsCHILD!AssociateID & Chr(34) & ", " & rsSkills!aID & ", " & rsSkills!SkillNo & ", " & rsSkills!SkillVersion & ", " & ShowPractice & ", " & Practice & ", " & rsCHILD!RoleID & ", 2, " & rsServiceType!ServiceTypeID & ", " & rsSkills!BusinessTypeID & ", '" & Date & "') "
db.Execute SQL2
' DoCmd.SetWarnings False
' DoCmd.RunSQL SQL2
' DoCmd.SetWarnings True
rsSkills.MoveNext
Loop 'Skills
Else 'Behavioral Skills
'Do not populate Behavioral Skills (0's)
End If 'Behavioral Skills
'Debug.Print "B:Skills done " & Format(Now, "h:nn:ss AM/PM")
End If 'rsConsultingMatrix!Behavioral
rsConsultingMatrix.MoveNext
Loop 'ConsultingMatrix
Else 'ConsultingMatrix
''Debug.Print "There are no records in the Consulting Matrix recordset."
End If 'ConsultingMatrix
'Debug.Print "Finished looping through CONSULTING MATRIX. " & Format(Now, "h:nn:ss AM/PM")
rsServiceType.MoveNext
Loop 'rsServiceType
Else 'rsServiceType
'Debug.Print "There are no records in the SERVICE TYPE recordset. " & Format(Now, "h:nn:ss AM/PM")
End If 'rsServiceType
s = "Behavioral Ends: " & Format(Now, "h:nn:ss AM/PM")
Debug.Print s
'' Print #n, s
'END Behavioral Skills
s = "Time: " & Format(Now, "h:nn:ss AM/PM") & " - ElapsedTime for Parent " & rsPARENT!RoleID & " with child " & rsCHILD!RoleID & " is " & HoursMinutes(StartTime, Now)
Debug.Print s
Print #n, s
rsCHILD.MoveNext
Loop 'rsCHILD
Else 'rsCHILD
'Debug.Print "There are no records in the CHILD recordset. " & Format(Now, "h:nn:ss AM/PM")
End If 'rsCHILD
'Debug.Print "Finished looping through CHILD records. " & Format(Now, "h:nn:ss AM/PM")
rsPARENT.MoveNext
Loop 'rsPARENT
Else 'rsPARENT
'Debug.Print "There are no records in the PARENT recordset. " & Format(Now, "h:nn:ss AM/PM")
End If 'rsPARENT
'Debug.Print "Finished looping through PARENT records. " & Format(Now, "h:nn:ss AM/PM")
rs.Close
rsPARENT.Close
rsCHILD.Close
rsServiceType.Close
rsConsultingMatrix.Close
Set rs = Nothing
Set rsPARENT = Nothing
Set rsCHILD = Nothing
Set rsServiceType = Nothing
Set rsConsultingMatrix = Nothing
Close #n
End Function