Java Specialists' Java Training Europehome of the java specialists' newsletter

The Java Specialists' Newsletter
Issue 1822010-05-12 Category: GUI Java version: 6+

GitHub Subscribe Free RSS Feed

Remote Screenshots

by Dr. Heinz M. Kabutz
Abstract:
In this newsletter, we describe how we can generate remote screen shots as compressed, scaled JPGs to build a more efficient remote control mechanism.

A warm welcome to the 182nd edition of The Java(tm) Specialists' Newsletter, sent to you from Regina in Saskatchewan Canada. I went to a fantastic restaurant last night called "The Diplomat", where I tasted a wonderful fillet with lemon potatoes. As soon as I put the first potato in my mouth, I knew the nationality of the chef - Greek! I asked the waitress and she was quite surprised that I knew. The potatoes tasted exactly like those made by Irene at her taverna up the road from where we live! A Java programmer from Albania subscribed recently, so our list of subscriber countries has grown to 121. Special welcome to Adonis Arifi!

Email subscribers would have seen the announcements for our live webinars (Click here to download the recordings). We had a lot of interest in these, with hundreds of participants from over 60 countries. Even though we were in a beta phase of presenting online webinars, they turned out surprisingly well. We are going to do this a lot more in future and are also going to have a Java Specialist Club where we can then discuss these webinars and deal better with questions afterwards. I will announce details of this in the next newsletter.

Last week we ran a train-the-trainer class in Crete, with instructors from the USA, South Africa, Germany, France and Spain joining us in Crete and New Zealand joining via a webinar. It was simply wonderful using the new conference room at our house. The setup is actually the nicest that I've taught in anywhere. We have a balcony from which you can view the sea, birds twittering around us. On the last day we ended our course with Greek lamb roasted slowly on coals. It was turning and dropping it's fat into the fire towards the end of the course, with the scents wafting into the room. Made my mouth water! Our lunches were held up the road at Irene's Taverna, where she served us delicious traditional foods.

In two weeks time, from the 25th to 28th May 2010, we are running our Java Specialist Master Course in Chania, Crete. We have still one place available. Price is EUR2500, which includes lunches, the course fees, coffee, water and an amazing Greek lamb feast on the last day. We can recommend hotels in different price categories. Flights are currently also quite affordable within Europe. In addition, I will offer you a money back guarantee on the course fees if you cannot come due to the volcanic activities in Iceland (or we will try to reschedule you for a later date if you prefer).

If you want to save yourself the trip to Crete, you are also welcome to attend the course using the webinar software. You will then only need to pay for the course fees and not hotel and flights. The benefit to you is that you can then attend the course from anywhere in the world. Just note that we will start at 6:00 GMT and run until 14:00 GMT. We tried it last week with someone from New Zealand and it worked really well. I do not like travelling and can understand if you feel the same way.

NEW: We have revised our "Advanced Topics" course, covering Reflection, Java NIO, Data Structures, Memory Management and several other useful topics for Java experts to master. 2 days of extreme fun and learning. Extreme Java - Advanced Topics.

Remote Screenshots

Programmers do not like it when you stand behind them watching how they code, especially if they are not confident on how to solve a problem. Many years ago, I wrote a very simple Java program that allowed me to watch my students' screens, with their knowledge of course, without the feeling of being watched.

Unfortunately my very basic program only scaled up to 6 PCs, after that it became too slow and it was hard to follow all those windows. It also did not work well over a DSL internet connection.

Last year I was away from home for 150 days, due to all the courses I had to present. Our plan is to offer more courses from our conference facility on our island in the Mediterranean. To do this, I needed a program to let me view the approach my students were using in solving the exercises. I thus took another look at my old program and improved it to reduce the bandwidth.

The way this program works is that we start a server on our side to which the students can connect. The server then sends a request to the student program to please send a "screen shot". It also tells the student program what zoom factor we want to see, which well help to reduce bandwidth on screens that I am not currently looking at. I can send mouse clicks to the student screen, which I would do if I needed to show them something or check that their answers are correct.

Similar to our first approach, we will use the interface RobotAction, which represents a command that I send to the client machine.

import java.awt.*;
import java.io.*;

public interface RobotAction extends Serializable {
  Object execute(Robot robot) throws IOException;
}
  

The RobotActionQueue extends a Deque implementation, so we can peek at the last element in the queue.

import java.util.concurrent.*;

public class RobotActionQueue extends
    LinkedBlockingDeque<RobotAction> {
}
  

The client machine would run the Student class (below), which would read in the RobotAction objects from the server and then insert them into a queue. One of the issues that we had with our old solution was that the same requests would start filling up the queue, so that new requests would take a very long time to be processed. We thus use jobs.peekLast() to see whether the same type of job is already waiting and if it is, we discard it.

When the robot action is executed, any non-null result is sent back to the server. In our current implementation, the only information that we want to send back to the Teacher is a screen shot as a byte[], so that is all that we would expect. However, we could easily expand this program to let the student press a button to call for help.

import java.awt.*;
import java.io.*;
import java.net.*;

public class Student {
  private final ObjectOutputStream out;
  private final ObjectInputStream in;
  private final Robot robot;
  private final RobotActionQueue jobs = new RobotActionQueue();
  private final ProcessorThread processor;
  private final ReaderThread reader;

  public Student(String serverMachine, String studentName)
      throws IOException, AWTException {
    Socket socket = new Socket(
        serverMachine, TeacherServer.PORT);
    robot = new Robot();
    out = new ObjectOutputStream(socket.getOutputStream());
    in = new ObjectInputStream(
        new BufferedInputStream(socket.getInputStream()));
    out.writeObject(studentName);
    out.flush();
    processor = new ProcessorThread();
    reader = new ReaderThread();
  }

  private class ReaderThread extends Thread {
    public void run() {
      try {
        RobotAction action;
        while ((action = (RobotAction) in.readObject()) != null) {
          if (!action.equals(jobs.peekLast())) {
            jobs.add(action);
            System.out.println("jobs = " + jobs);
          } else {
            System.out.println("Discarding duplicate request");
          }
        }
      } catch (EOFException eof) {
        System.out.println("Connection closed");
      } catch (Exception ex) {
        System.out.println("Connection closed abruptly: " + ex);
      }
    }
  }

  private class ProcessorThread extends Thread {
    public ProcessorThread() {
      super("ProcessorThread");
      setDaemon(true);
    }

    public void run() {
      try {
        while (!isInterrupted()) {
          try {
            RobotAction action = jobs.take();
            Object result = action.execute(robot);
            if (result != null) {
              out.writeObject(result);
              out.reset();
              out.flush();
            }
          } catch (InterruptedException e) {
            interrupt();
            break;
          }
        }
        out.close();
      } catch (IOException e) {
        System.out.println("Connection closed (" + e + ')');
      }
    }
  }

  public void start() {
    processor.start();
    reader.start();
  }

  public static void main(String[] args) throws Exception {
    if (args.length != 2) {
      System.err.println("Parameters: server studentname");
      System.exit(1);
    }
    Student student = new Student(args[0], args[1]);
    student.start();
  }
}
  

The first action we want to look at is MouseMove. We want to send a request to the client PC to move the mouse to a certain position. The student program would then call the execute method, which would move the pointer to the desired location.

Instead of using the standard hashcode algorithm, where we multiply the first field by 31 and then add the second, we will rather do bit shifting to get unique values every time. There is otherwise a very good chance of a collision. We are not using the MoveMouse objects for hashing, but on principle I think we should write good hashCode functions even if the object is currently not intended as a key.

import java.awt.*;
import java.awt.event.*;

public class MoveMouse implements RobotAction {
  private final int x;
  private final int y;

  public MoveMouse(Point to) {
    x = (int) to.getX();
    y = (int) to.getY();
  }

  public MoveMouse(MouseEvent event) {
    this(event.getPoint());
  }

  public Object execute(Robot robot) {
    robot.mouseMove(x, y);
    return null;
  }

  public String toString() {
    return "MoveMouse: x=" + x + ", y=" + y;
  }

  public boolean equals(Object o) {
    if (!(o instanceof MoveMouse)) return false;
    MoveMouse mm = (MoveMouse) o;
    return x == mm.x && y == mm.y;
  }

  public int hashCode() {
    return (x << 16) + y;
  }
}
  

The next action is ClickMouse. Here we specify which mouse button was pressed and how often. With these two actions, I can now control the remote PC to help anyone who gets stuck.

import java.awt.*;
import java.awt.event.*;

public class ClickMouse implements RobotAction {
  private final int mouseButton;
  private final int clicks;

  public ClickMouse(int mouseButton, int clicks) {
    this.mouseButton = mouseButton;
    this.clicks = clicks;
  }

  public ClickMouse(MouseEvent event) {
    this(event.getModifiers(), event.getClickCount());
  }

  public Object execute(Robot robot) {
    for (int i = 0; i < clicks; i++) {
      robot.mousePress(mouseButton);
      robot.mouseRelease(mouseButton);
    }
    return null;
  }

  public String toString() {
    return "ClickMouse: " + mouseButton + ", " + clicks;
  }

  public boolean equals(Object o) {
    if (!(o instanceof ClickMouse)) return false;
    ClickMouse cm = (ClickMouse) o;
    return clicks == cm.clicks && mouseButton == cm.mouseButton;
  }

  public int hashCode() {
    return 31 * mouseButton + clicks;
  }
}
  

The last robot action that we need to look at is the ScreenShot, where the Robot creates an image of the screen and then sends that over the wire as a JPEG. In this code we used several tricks to reduce the bandwidth used.

First off, when I am not actively watching what the student is doing, I set the zoom factor to 30%. This allows me to keep lots of student screens open. As soon as I select his screen, it changes the zoom factor to 100%. I tried several options to make the scaling look nice. After some experiments I decided to use SCALE_AREA_AVERAGING, but that is a bit slow. If the student machine is too slow to cope, then we can also change to the SCALE_FAST algorithm.

The second trick is to reduce the JPG quality, so that I can still recognize what is happening, but where we reduce the bandwidth significantly. This way we were able to reduce the images to 1/50th of their size. In our program we chose a compression quality of 30%. You can see the artifacts of the JPG compression if you look carefully, but at least it is really fast.

The third trick is that I only send the image if it has not changed since the last time, which I store in the ThreadLocal called "previous". It would be even better if we stored the raw image, so that we could only send across the region that was changed last. This is how the commercial solutions work.

import javax.imageio.*;
import javax.imageio.stream.*;
import java.awt.*;
import java.awt.image.*;
import java.io.*;
import java.util.*;

public class ScreenShot implements RobotAction {
  // this is used on the student JVM to optimize transfers
  private static final ThreadLocal<byte[]> previous =
      new ThreadLocal<byte[]>();
  private static final float JPG_QUALITY = 0.3f;

  private final double scale;

  public ScreenShot(double scale) {
    this.scale = scale;
  }

  public ScreenShot() {
    this(1.0);
  }

  public Object execute(Robot robot) throws IOException {
    long time = System.currentTimeMillis();
    Toolkit defaultToolkit = Toolkit.getDefaultToolkit();
    Rectangle shotArea = new Rectangle(
        defaultToolkit.getScreenSize());
    BufferedImage image = robot.createScreenCapture(shotArea);
    if (scale != 1.0) {
      image = getScaledInstance(image);
    }
    byte[] bytes = convertToJPG(image);
    time = System.currentTimeMillis() - time;
    System.out.println("time = " + time);
    // only send it if the picture has actually changed
    byte[] prev = previous.get();
    if (prev != null && Arrays.equals(bytes, prev)) {
      return null;
    }
    previous.set(bytes);
    return bytes;
  }

  private byte[] convertToJPG(BufferedImage img)
      throws IOException {
    ImageWriter writer =
        ImageIO.getImageWritersByFormatName("jpg").next();
    ImageWriteParam iwp = writer.getDefaultWriteParam();
    iwp.setCompressionMode(ImageWriteParam.MODE_EXPLICIT);
    iwp.setCompressionQuality(JPG_QUALITY);

    ByteArrayOutputStream bout = new ByteArrayOutputStream();
    writer.setOutput(new MemoryCacheImageOutputStream(bout));
    writer.write(null, new IIOImage(img, null, null), iwp);
    writer.dispose();
    bout.flush();
    return bout.toByteArray();
  }

  public BufferedImage getScaledInstance(BufferedImage src) {
    int width = (int) (src.getWidth() * scale);
    int height = (int) (src.getHeight() * scale);

    Image scaled = src.getScaledInstance(width, height,
        BufferedImage.SCALE_AREA_AVERAGING);
    BufferedImage result = new BufferedImage(
        width, height, BufferedImage.TYPE_INT_RGB
    );
    result.createGraphics().drawImage(
        scaled, 0, 0, width, height, null);
    return result;
  }

  public String toString() {
    return "ScreenShot(" + scale + ")";
  }

  public boolean equals(Object o) {
    if (!(o instanceof ScreenShot)) return false;
    return Double.compare(((ScreenShot) o).scale, scale) == 0;
  }

  public int hashCode() {
    long temp = Double.doubleToLongBits(scale);
    return (int) (temp ^ (temp >>> 32));
  }
}

TeacherServer

Our main class is the TeacherServer. When a new student connects, the first object that is sent is the student name. We then generate two threads to communicate with the student. In our system, the bottleneck is the network and not the number of threads. It thus does not matter that we are constructing two threads per connection. I would not be able to monitor that many students at a time anyway. However, if it ever became an issue, then we could simply replace the server with non-blocking IO.

The SocketWriterThread controls the zoom factor and the delay between ScreenShot requests. Whilst the window is active, we want to see updates as quickly as possible, so we set the zoom factor to 100% and the wait time to 300 milliseconds. When we stop looking at the window, the zoom goes back to 30% and the wait time to 3 seconds between taking screen shots.

import java.awt.event.*;
import java.io.*;
import java.util.concurrent.*;

class SocketWriterThread extends Thread {
  private final RobotActionQueue jobs = new RobotActionQueue();
  private final String studentName;
  private final ObjectOutputStream out;
  private volatile boolean active = false;

  public SocketWriterThread(String studentName,
                            ObjectOutputStream out) {
    super("Writer to " + studentName);
    this.studentName = studentName;
    this.out = out;
  }

  public void setActive(boolean active) {
    this.active = active;
    askForScreenShot();
  }

  private double getZoomFactor() {
    return active ? 1.0 : 0.3;
  }

  public long getWaitTime() {
    return active ? 500 : 3000;
  }

  public void clickEvent(MouseEvent e) {
    if (active) {
      jobs.add(new MoveMouse(e));
      jobs.add(new ClickMouse(e));
    }
    active = true;
    askForScreenShot();
  }

  private void askForScreenShot() {
    jobs.add(new ScreenShot(getZoomFactor()));
  }

  public void run() {
    askForScreenShot();
    try {
      while (!isInterrupted()) {
        try {
          RobotAction action = jobs.poll(
              getWaitTime(),
              TimeUnit.MILLISECONDS);
          if (action == null) {
            // we had a timeout, so do a screen capture
            askForScreenShot();
          } else {
            System.out.println("sending " + action +
                " to " + studentName);
            out.writeObject(action);
            out.reset();
            out.flush();
          }
        } catch (InterruptedException e) {
          interrupt();
          break;
        }
      }
      out.close();
    } catch (IOException e) {
      System.out.println("Connection to " + studentName +
          " closed (" + e + ')');
    }
    System.out.println("Closing connection to " + studentName);
  }
}
  

The SocketReaderThread reads the screen shot results from our student process and then passes them to the TeacherServer, who then displays them immediately. This is rather simple code:

import java.io.*;

class SocketReaderThread extends Thread {
  private final String studentName;
  private final ObjectInputStream in;
  private final TeacherServer server;

  public SocketReaderThread(
      String studentName,
      ObjectInputStream in,
      TeacherServer server) {
    super("Reader from " + studentName);
    this.studentName = studentName;
    this.in = in;
    this.server = server;
  }

  public void run() {
    while (true) {
      try {
        byte[] img = (byte[]) in.readObject();
        System.out.println("Received screenshot of " +
            img.length + " bytes from " + studentName);
        server.showScreenShot(img);
      } catch (Exception ex) {
        System.out.println("Exception occurred: " + ex);
        ex.printStackTrace();
        server.shutdown();
        return;
      }
    }
  }

  public void close() {
    try {
      in.close();
    } catch (IOException ignore) {
    }
  }
}

The TeacherServer class sets up the network connections with the server and creates the TeacherFrame, which then displays the screen shots sent by the student program.

import javax.swing.*;
import java.io.*;
import java.net.*;
import java.lang.reflect.*;

public class TeacherServer {
  public static final int PORT = 5555;

  private final SocketWriterThread writer;
  private final TeacherFrame frame;
  private final SocketReaderThread reader;

  public TeacherServer(Socket socket)
      throws IOException, ClassNotFoundException,
      InvocationTargetException, InterruptedException {
    ObjectOutputStream out = new ObjectOutputStream(
        socket.getOutputStream());
    ObjectInputStream in = new ObjectInputStream(
        new BufferedInputStream(
            socket.getInputStream()));
    System.out.println("waiting for student name ...");
    final String studentName = (String) in.readObject();

    reader = new SocketReaderThread(studentName, in, this);
    writer = new SocketWriterThread(studentName, out);

    final TeacherFrame[] temp = new TeacherFrame[1];
    SwingUtilities.invokeAndWait(new Runnable() {
      public void run() {
        temp[0] = new TeacherFrame(studentName,
            TeacherServer.this, writer);
      }
    });
    frame = temp[0];

    reader.start();
    writer.start();

    System.out.println("finished connecting to " + socket);
  }

  public void showScreenShot(byte[] bytes) throws IOException {
    frame.showScreenShot(bytes);
  }

  public void shutdown() {
    writer.interrupt();
    reader.close();
  }

  public static void main(String[] args) throws Exception {
    ServerSocket ss = new ServerSocket(PORT);
    while (true) {
      Socket socket = ss.accept();
      System.out.println("Connection From " + socket);
      new TeacherServer(socket);
    }
  }
}  

The last class we need is the TeacherFrame, where we display what our student can see. When we click on the image, we at the same time send an two events to our student, MoveMouse and ClickMouse. We use the windowActivated and windowDeactivated events to decide what zoom factor to send to our student.

import javax.imageio.*;
import javax.swing.*;
import java.awt.event.*;
import java.awt.image.*;
import java.io.*;

public class TeacherFrame extends JFrame {
  private final TeacherServer server;
  private final SocketWriterThread writer;
  private final JLabel iconLabel = new JLabel();

  public TeacherFrame(String studentName, TeacherServer server,
                      SocketWriterThread writer) {
    super("Screen from " + studentName);
    this.server = server;
    this.writer = writer;

    add(new JScrollPane(iconLabel));
    iconLabel.addMouseListener(new MouseAdapter() {
      public void mouseClicked(MouseEvent e) {
        TeacherFrame.this.writer.clickEvent(e);
      }
    });
    addWindowListener(new WindowAdapter() {
      public void windowActivated(WindowEvent e) {
        TeacherFrame.this.writer.setActive(true);
      }
      public void windowDeactivated(WindowEvent e) {
        TeacherFrame.this.writer.setActive(false);
      }
      public void windowClosing(WindowEvent e) {
        TeacherFrame.this.server.shutdown();
      }
    });

    pack();
    setVisible(true);
  }

  public void showScreenShot(byte[] bytes) throws IOException {
    final BufferedImage img = ImageIO.read(
        new ByteArrayInputStream(bytes));
    SwingUtilities.invokeLater(new Runnable() {
      public void run() {
        iconLabel.setIcon(new ImageIcon(img));
        pack();
      }
    });
  }
}

To try out this program, it would be best to use two computers. The first computer would run the TeacherServer program, for example:

    java eu.javaspecialists.tjsn.jmatia.server.TeacherServer
  

The second computer runs the student, for which we need to specify the address of the server and the name of the student, for example:

    java eu.javaspecialists.tjsn.jmatia.Student 192.168.1.7 "Max Guy"
  

It is really interesting how quickly the images are now sent from the student to the teacher. You can learn a lot by watching how people approach a new exercise. It certainly helps to then explain to them how it should have been answered. It also helps to be able to watch lots of students at once.

Kind regards, and hope to see you in Crete, either in person or via the internet :-)

Heinz

GUI Articles Related Java Course

Extreme Java - Concurrency and Performance for Java 8
Extreme Java - Advanced Topics for Java 8
Design Patterns
In-House Courses

© 2010-2016 Heinz Kabutz - All Rights Reserved Sitemap
Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners. JavaSpecialists.eu is not connected to Oracle, Inc. and is not sponsored by Oracle, Inc.