Looking for help with SetConsoleDisplayMode Api

[Unscruffed]

Hi everyone. Firstly, Merry Christmas and a prosperous and safe New Year to all.

I'm working on a VB6 app that will take control of the printing operations for a DOS application. The DOS app runs in fullscreen, and my VB app will need to display a preview window.

In short, what I need to achieve is:

1. Get the DisplayMode of the DOS app
2. If the DisplayMode is Fullscreen, change it to Windowed
3. Let the VB app do it's stuff
4. Restore the DOS app to Fullscreen if applicable.

I did achieve this a long time ago, but the source code was long lost after a drive failure. From memory I used OpenProcess or some kind of Pipe, but I can't remember which, or how I got the handle. The DisplayMode updating was done using GetConsoleDisplayMode and SetConsoleDisplayMode.

I have found plenty of C code, but every example seems to be (the same) code within the Console application itself. I can't find anything in VB, or anything that shows how to obtain the correct handle to use when targeting an external application.

Any help appreciated.

Cheers,
Scruff.

Heaven doesn't want me, and Hell's afraid I'll take over!

 

[Unscruffed]

SOLVED

The following is the code I worked out to hide then restore a Console window from a VB application. The result is 2 functions: HideConsole and RestoreConsole.

The HideConsole function stores the current DisplayMode of the Console (Windowed or FullScreen). If the current setting is FullScreen, it sets the DisplayMode to Windowed. Finally, it sets the WindowState to Minimized.
The HideConsole function takes one parameter:
ConsoleCaption: The Caption of the Console Window.
(This is usually the same as the name of the Shortcut that launched the Console.)

The RestoreConsole function first checks the original DisplayMode stored by the HideConsole function. If the result is FullScreen, it sets the DisplayMode accordingly. The WindowState is then set to Normal, and the Window brought to the foreground to ensure that it shows.
The RestoreConsole function has no parameters.

Both functions return True if successful, or False otherwise.

Module: modHideConsole.bas

Option Explicit

Private Type POINTAPI
    X As Long
    Y As Long
End Type

Private Type RECT
    Left As Long
    Top As Long
    Right As Long
    Bottom As Long
End Type

Private Type WindowPlacement
    Length As Long
    flags As Long
    showCmd As Long
    ptMinPosition As POINTAPI
    ptMaxPosition As POINTAPI
    rcNormalPosition As RECT
End Type

Private Declare Function FindWindow Lib "user32" Alias "FindWindowA" (ByVal _
    lpClassName As String, ByVal lpWindowName As String) As Long
Private Declare Function GetWindowThreadProcessId Lib "user32" (ByVal hWnd As _
    Long, lpdwProcessId As Long) As Long
Private Declare Function GetStdHandle Lib "kernel32" (ByVal nStdHandle As Long) _
    As Long
Private Declare Function AttachConsole Lib "kernel32" (ByVal hConsoleHandle As _
    Long) As Long
Private Declare Function GetConsoleDisplayMode Lib "kernel32" _
    (dwConsoleDisplayMode As Long) As Long
Private Declare Function SetConsoleDisplayMode Lib "kernel32" (ByVal _
    hConsoleHandle As Long, ByVal dwConsoleDisplayMode As Long, _
    dwPreviousDisplayMode As Long) As Long
Private Declare Function FreeConsole Lib "kernel32" () As Long
Private Declare Function GetWindowPlacement Lib "user32" (ByVal hWnd As Long, _
    lpwndpl As WindowPlacement) As Long
Private Declare Function SetWindowPlacement Lib "user32" (ByVal hWnd As Long, _
    lpwndpl As WindowPlacement) As Long
Private Declare Function SetForegroundWindow Lib "user32" (ByVal hWnd As Long) _
    As Long

Private Const STD_OUTPUT_HANDLE = -11&
Private Const CONSOLE_WINDOWED_MODE As Long = 0
Private Const CONSOLE_FULLSCREEN_MODE As Long = 1
Private Const CONSOLE_FULLSCREEN_HARDWARE As Long = 2
Private Const SW_SHOWNORMAL = 1
Private Const SW_SHOWMINIMIZED = 2

Private m_ConsoleCaption As String
Private m_PreviousMode As Long

' HideConsole
' Parameters: ConsoleCaption = Caption of the target Console Window.
' Return Value: Success = True (-1), Fail = False (0).
Public Function HideConsole(ConsoleCaption As String) As Boolean
    Dim lWnd As Long
    Dim lPID As Long
    Dim lHandle As Long
    Dim lRet As Long
    Dim lDummy As Long
    Dim WP As WindowPlacement
    m_ConsoleCaption = ConsoleCaption
    lWnd = FindWindow("ConsoleWindowClass", m_ConsoleCaption)
    If lWnd <> 0 Then
        lRet = GetWindowThreadProcessId(lWnd, lPID)
        If lRet <> 0 Then
            lRet = AttachConsole(lPID)
            If lRet <> 0 Then
                lHandle = GetStdHandle(STD_OUTPUT_HANDLE)
                lRet = GetConsoleDisplayMode(m_PreviousMode)
                If m_PreviousMode = (CONSOLE_FULLSCREEN_MODE Or _
                    CONSOLE_FULLSCREEN_HARDWARE) Then
                    lRet = SetConsoleDisplayMode(lHandle, CONSOLE_WINDOWED_MODE, _
                        lDummy)
                End If
            End If
            Call FreeConsole
        End If
        WP.Length = Len(WP)
        Call GetWindowPlacement(lWnd, WP)
        WP.flags = 0&
        WP.showCmd = SW_SHOWMINIMIZED
        Call SetWindowPlacement(lWnd, WP)
    End If
    HideConsole = (lWnd <> 0) And (lRet <> 0)
End Function

' RestoreConsole
' Return Value: Success = True (-1), Fail = False (0).
Public Function RestoreConsole() As Boolean
    Dim lWnd As Long
    Dim lPID As Long
    Dim lHandle As Long
    Dim lRet As Long
    Dim lDummy As Long
    Dim WP As WindowPlacement
    lWnd = FindWindow("ConsoleWindowClass", m_ConsoleCaption)
    If lWnd <> 0 Then
        lRet = GetWindowThreadProcessId(lWnd, lPID)
        If lRet <> 0 Then
            lRet = AttachConsole(lPID)
            If lRet <> 0 Then
                lHandle = GetStdHandle(STD_OUTPUT_HANDLE)
                If m_PreviousMode = (CONSOLE_FULLSCREEN_MODE Or _
                    CONSOLE_FULLSCREEN_HARDWARE) Then
                    lRet = SetConsoleDisplayMode(lHandle, _
                        CONSOLE_FULLSCREEN_MODE, lDummy)
                End If
            End If
            Call FreeConsole
        End If
        WP.Length = Len(WP)
        Call GetWindowPlacement(lWnd, WP)
        WP.flags = 0&
        WP.showCmd = SW_SHOWNORMAL
        Call SetWindowPlacement(lWnd, WP)
        Call SetForegroundWindow(lWnd)
    End If
    RestoreConsole = (lWnd <> 0) And (lRet <> 0)
End Function

Cheers,
Scruff.


Heaven doesn't want me, and Hell's afraid I'll take over!

 

[Unscruffed]

SOLUTION 2

After some fiddling, I came up with this solution to work on any Console window that is in fullscreen mode.

If a Console window is in Fullscreen mode, it is automatically placed at the top of the ZOrder. Since the FindWindow API starts it's search from the top of the ZOrder, we can target the first window found with the "ConsoleWindowClass" class name. You therefore don't need to know the window's caption.

The HideConsole function will find the topmost Console window and act in the following ways:
If no console found: Do nothing.
If the first console found is Windowed: Do nothing.
If the first console found is Fullscreen: Change to Windowed and Minimize.

The RestoreConsole does nothing unless HideConsole found a Console window in Fullscreen mode, in which case it will restore the window.

This code is more generic in that it will take any Console window out of Fullscreen mode and restore it afterwards, rather than the solution above which is more application specific.

Module: modHideConsole2.bas[/]:

Option Explicit

Private Type POINTAPI
    X As Long
    Y As Long
End Type

Private Type RECT
    Left As Long
    Top As Long
    Right As Long
    Bottom As Long
End Type

Private Type WindowPlacement
    Length As Long
    flags As Long
    showCmd As Long
    ptMinPosition As POINTAPI
    ptMaxPosition As POINTAPI
    rcNormalPosition As RECT
End Type

Private Declare Function FindWindow Lib "user32" Alias "FindWindowA" (ByVal lpClassName As String, ByVal lpWindowName As String) As Long
Private Declare Function GetWindowThreadProcessId Lib "user32" (ByVal hWnd As Long, lpdwProcessId As Long) As Long
Private Declare Function GetStdHandle Lib "kernel32" (ByVal nStdHandle As Long) As Long
Private Declare Function AttachConsole Lib "kernel32" (ByVal hConsoleHandle As Long) As Long
Private Declare Function GetConsoleDisplayMode Lib "kernel32" (dwConsoleDisplayMode As Long) As Long
Private Declare Function SetConsoleDisplayMode Lib "kernel32" (ByVal hConsoleHandle As Long, ByVal dwConsoleDisplayMode As Long, dwPreviousDisplayMode As Long) As Long
Private Declare Function FreeConsole Lib "kernel32" () As Long
Private Declare Function GetWindowPlacement Lib "user32" (ByVal hWnd As Long, lpwndpl As WindowPlacement) As Long
Private Declare Function SetWindowPlacement Lib "user32" (ByVal hWnd As Long, lpwndpl As WindowPlacement) As Long
Private Declare Function SetForegroundWindow Lib "user32" (ByVal hWnd As Long) As Long

Private Const STD_OUTPUT_HANDLE = -11&
Private Const CONSOLE_WINDOWED_MODE As Long = 0
Private Const CONSOLE_FULLSCREEN_MODE As Long = 1
Private Const CONSOLE_FULLSCREEN_HARDWARE As Long = 2
Private Const SW_SHOWNORMAL = 1
Private Const SW_SHOWMINIMIZED = 2

Private m_hWnd As Long

' HideConsole
' Return Value: Success = True (-1), Fail = False (0).
Public Function HideConsole() As Boolean
    Dim lWnd As Long
    Dim lPID As Long
    Dim lHandle As Long
    Dim lRet As Long
    Dim lDummy As Long
    Dim lDisplayMode As Long
    Dim WP As WindowPlacement
    m_hWnd = 0
    lWnd = FindWindow("ConsoleWindowClass", vbNullString)
    If lWnd <> 0 Then
        lRet = GetWindowThreadProcessId(lWnd, lPID)
        If lRet <> 0 Then
            lRet = AttachConsole(lPID)
            If lRet <> 0 Then
                lHandle = GetStdHandle(STD_OUTPUT_HANDLE)
                lRet = GetConsoleDisplayMode(lDisplayMode)
                If lDisplayMode = (CONSOLE_FULLSCREEN_MODE Or CONSOLE_FULLSCREEN_HARDWARE) Then
                    lRet = SetConsoleDisplayMode(lHandle, CONSOLE_WINDOWED_MODE, lDummy)
                    WP.Length = Len(WP)
                    Call GetWindowPlacement(lWnd, WP)
                    WP.flags = 0&
                    WP.showCmd = SW_SHOWMINIMIZED
                    Call SetWindowPlacement(lWnd, WP)
                    m_hWnd = lWnd
                End If
                Call FreeConsole
            End If
        End If
    End If
    HideConsole = (m_hWnd <> 0) And (lRet <> 0)
End Function

' RestoreConsole
' Return Value: Success = True (-1), Fail = False (0).
Public Function RestoreConsole() As Boolean
    Dim lPID As Long
    Dim lHandle As Long
    Dim lRet As Long
    Dim lDummy As Long
    Dim WP As WindowPlacement
    If m_hWnd <> 0 Then
        lRet = GetWindowThreadProcessId(m_hWnd, lPID)
        If lRet <> 0 Then
            lRet = AttachConsole(lPID)
            If lRet <> 0 Then
                lHandle = GetStdHandle(STD_OUTPUT_HANDLE)
                lRet = SetConsoleDisplayMode(lHandle, CONSOLE_FULLSCREEN_MODE, lDummy)
            End If
            Call FreeConsole
        End If
        WP.Length = Len(WP)
        Call GetWindowPlacement(m_hWnd, WP)
        WP.flags = 0&
        WP.showCmd = SW_SHOWNORMAL
        Call SetWindowPlacement(m_hWnd, WP)
        Call SetForegroundWindow(m_hWnd)
    End If
    RestoreConsole = (m_hWnd <> 0) And (lRet <> 0)
    m_hWnd = 0
End Function

Cheers,
Scruff.

Heaven doesn't want me, and Hell's afraid I'll take over!

 

[j252ewv]

Does anyone have a 'Hello World' application to write to a console window?