MVTableEngine (backup, encryption, various bugs)

h2/src/docsrc/html/changelog.html

@@ -18,7 +18,9 @@ Change Log
 <h1>Change Log</h1>
 
 <h2>Next Version (unreleased)</h2>
-<ul><li>To use the MVStore storage engine (which is still work in progress), append
+<ul><li>File system abstraction: if used directly, some file systems did not work correctly
+    with spliced byte buffers (the database engine doesn't use those).
+</li><li>To use the MVStore storage engine (which is still work in progress), append
     ";mv_store=true" to the database URL. Using the MVTableEngine when creating the table
     is no longer recommended.
 </li><li>To compile user defined functions, the javax.tools.JavaCompiler is now used if available,

h2/src/docsrc/html/mvstore.html

@@ -32,11 +32,11 @@ MVStore
 <a href="#transactions">- Transactions</a><br />
 <a href="#inMemory">- In-Memory Performance and Usage</a><br />
 <a href="#dataTypes">- Pluggable Data Types</a><br />
-<a href="#offHeap">- Off-Heap and Pluggable Storage</a><br />
 <a href="#blob">- BLOB Support</a><br />
 <a href="#pluggableMap">- R-Tree and Pluggable Map Implementations</a><br />
 <a href="#caching">- Concurrent Operations and Caching</a><br />
 <a href="#logStructured">- Log Structured Storage</a><br />
+<a href="#offHeap">- Off-Heap and Pluggable Storage</a><br />
 <a href="#fileSystem">- File System Abstraction, File Locking and Online Backup</a><br />
 <a href="#encryption">- Encrypted Files</a><br />
 <a href="#tools">- Tools</a><br />
@@ -283,19 +283,6 @@ Also, there is no inherent limit to the number of maps and chunks.
 Due to using a log structured storage, there is no special case handling for large keys or pages.
 </p>
 
-<h3 id="offHeap">Off-Heap and Pluggable Storage</h3>
-<p>
-Storage is pluggable. The default storage is to a single file (unless pure in-memory operation is used).
-</p>
-<p>
-An off-heap storage implementation is available. This storage keeps the data in the off-heap memory,
-meaning outside of the regular garbage collected heap. This allows to use very large in-memory
-stores without having to increase the JVM heap (which would increase Java garbage collection
-cost a lot). Memory is allocated using <code>ByteBuffer.allocateDirect</code>.
-One chunk is allocated at a time (each chunk is usually a few MB large), so that
-allocation cost is low.
-</p>
-
 <h3 id="blob">BLOB Support</h3>
 <p>
 There is a mechanism that stores large binary objects by splitting them into smaller blocks.
@@ -386,11 +373,25 @@ But temporarily, disk space usage might actually be a bit higher than for a regu
 as disk space is not immediately re-used (there are no in-place updates).
 </p>
 
+<h3 id="offHeap">Off-Heap and Pluggable Storage</h3>
+<p>
+Storage is pluggable. The default storage is to a single file (unless pure in-memory operation is used).
+</p>
+<p>
+An off-heap storage implementation is available. This storage keeps the data in the off-heap memory,
+meaning outside of the regular garbage collected heap. This allows to use very large in-memory
+stores without having to increase the JVM heap (which would increase Java garbage collection
+pauses a lot). Memory is allocated using <code>ByteBuffer.allocateDirect</code>.
+One chunk is allocated at a time (each chunk is usually a few MB large), so that
+allocation cost is low.
+</p>
+
 <h3 id="fileSystem">File System Abstraction, File Locking and Online Backup</h3>
 <p>
 The file system is pluggable (the same file system abstraction is used as H2 uses).
-The file can be encrypted using an encrypting file system.
+The file can be encrypted using a encrypting file system wrapper.
 Other file system implementations support reading from a compressed zip or jar file.
+The file system abstraction closely matches the Java 7 file system API.
 </p>
 <p>
 Each store may only be opened once within a JVM.
@@ -399,11 +400,14 @@ the file can only be changed from within one process.
 Files can be opened in read-only mode, in which case a shared lock is used.
 </p>
 <p>
-The persisted data can be backed up to a different file at any time,
+The persisted data can be backed up at any time,
 even during write operations (online backup).
 To do that, automatic disk space reuse needs to be first disabled, so that
 new data is always appended at the end of the file.
 Then, the file can be copied (the file handle is available to the application).
+It is recommended to use the utility class <code>FileChannelInputStream</code> to do this.
+For encrypted databases, both the encrypted (raw) file content, 
+as well as the clear text content, can be backed up.
 </p>
 
 <h3 id="encryption">Encrypted Files</h3>

h2/src/docsrc/html/roadmap.html

@@ -22,6 +22,10 @@ Of course, patches are always welcome, but are not always applied as is.
 See also <a href="build.html#providing_patches">Providing Patches</a>.
 </p>
 
+<h2>Version 1.4.x: Planned Changes</h2>
+<ul><li>Replace file password hash with file encryption key; validate encryption key when connecting.
+</li></ul>
+
 <h2>Version 1.4.x: Planned Changes</h2>
 <ul><li>Build the jar file for Java 6 by default (JDBC API 4.1).
 </li><li>Enable the new storage format for dates (system property "h2.storeLocalTime").

h2/src/main/org/h2/command/dml/BackupCommand.java

@@ -21,6 +21,7 @@ import org.h2.engine.Database;
 import org.h2.engine.Session;
 import org.h2.expression.Expression;
 import org.h2.message.DbException;
+import org.h2.mvstore.MVStore;
 import org.h2.mvstore.db.MVTableEngine.Store;
 import org.h2.result.ResultInterface;
 import org.h2.store.FileLister;
@@ -82,8 +83,15 @@ public class BackupCommand extends Prepared {
                         backupFile(out, base, n);
                     }
                     if (n.endsWith(Constants.SUFFIX_MV_FILE)) {
-                        InputStream in = db.getMvStore().getInputStream();
-                        backupFile(out, base, n, in);
+                        MVStore s = store.getStore();
+                        boolean before = s.getReuseSpace();
+                        s.setReuseSpace(false);
+                        try {
+                            InputStream in = store.getInputStream();
+                            backupFile(out, base, n, in);
+                        } finally {
+                            s.setReuseSpace(before);
+                        }
                     }
                 }
             }

h2/src/main/org/h2/engine/ConnectionInfo.java

@@ -19,6 +19,7 @@ import org.h2.constant.ErrorCode;
 import org.h2.constant.SysProperties;
 import org.h2.message.DbException;
 import org.h2.security.SHA256;
+import org.h2.store.fs.FilePathCrypt;
 import org.h2.store.fs.FilePathRec;
 import org.h2.store.fs.FileUtils;
 import org.h2.util.New;
@@ -37,6 +38,7 @@ public class ConnectionInfo implements Cloneable {
     private String url;
     private String user;
     private byte[] filePasswordHash;
+    private byte[] fileEncryptionKey;
     private byte[] userPasswordHash;
 
     /**
@@ -118,6 +120,7 @@ public class ConnectionInfo implements Cloneable {
         ConnectionInfo clone = (ConnectionInfo) super.clone();
         clone.prop = (Properties) prop.clone();
         clone.filePasswordHash = Utils.cloneByteArray(filePasswordHash);
+        clone.fileEncryptionKey = Utils.cloneByteArray(fileEncryptionKey);
         clone.userPasswordHash = Utils.cloneByteArray(userPasswordHash);
         return clone;
     }
@@ -311,6 +314,7 @@ public class ConnectionInfo implements Cloneable {
             System.arraycopy(password, 0, filePassword, 0, space);
             Arrays.fill(password, (char) 0);
             password = np;
+            fileEncryptionKey = FilePathCrypt.getPasswordBytes(filePassword);
             filePasswordHash = hashPassword(passwordHash, "file", filePassword);
         }
         userPasswordHash = hashPassword(passwordHash, user, password);
@@ -398,9 +402,13 @@ public class ConnectionInfo implements Cloneable {
      *
      * @return the password hash or null
      */
-    byte[] getFilePasswordHash() {
+    public byte[] getFilePasswordHash() {
         return filePasswordHash;
     }
+    
+    byte[] getFileEncryptionKey() {
+        return fileEncryptionKey;   
+    }
 
     /**
      * Get the name of the user.
@@ -541,6 +549,10 @@ public class ConnectionInfo implements Cloneable {
     public void setFilePasswordHash(byte[] hash) {
         this.filePasswordHash = hash;
     }
+    
+    public void setFileEncryptionKey(byte[] key) {
+        this.fileEncryptionKey = key;
+    }
 
     /**
      * Overwrite a property.

h2/src/main/org/h2/engine/Constants.java

@@ -73,10 +73,15 @@ public class Constants {
     public static final int TCP_PROTOCOL_VERSION_11 = 11;
 
     /**
-     * The TCP protocol version number 11.
+     * The TCP protocol version number 12.
      */
     public static final int TCP_PROTOCOL_VERSION_12 = 12;
 
+    /**
+     * The TCP protocol version number 13.
+     */
+    public static final int TCP_PROTOCOL_VERSION_13 = 13;
+
     /**
      * The major version of this database.
      */

h2/src/main/org/h2/engine/Database.java

@@ -92,6 +92,7 @@ public class Database implements DataHandler {
     private final String databaseURL;
     private final String cipher;
     private final byte[] filePasswordHash;
+    private final byte[] fileEncryptionKey;
 
     private final HashMap<String, Role> roles = New.hashMap();
     private final HashMap<String, User> users = New.hashMap();
@@ -192,6 +193,7 @@ public class Database implements DataHandler {
         this.compareMode = CompareMode.getInstance(null, 0, false);
         this.persistent = ci.isPersistent();
         this.filePasswordHash = ci.getFilePasswordHash();
+        this.fileEncryptionKey = ci.getFileEncryptionKey();
         this.databaseName = name;
         this.databaseShortName = parseDatabaseShortName();
         this.maxLengthInplaceLob = SysProperties.LOB_IN_DATABASE ?
@@ -2574,8 +2576,8 @@ public class Database implements DataHandler {
         throw DbException.throwInternalError();
     }
 
-    public byte[] getFilePasswordHash() {
-        return filePasswordHash;
+    public byte[] getFileEncryptionKey() {
+        return fileEncryptionKey;
     }
 
     @Override

h2/src/main/org/h2/engine/SessionRemote.java

@@ -103,7 +103,7 @@ public class SessionRemote extends SessionWithState implements DataHandler {
         trans.setSSL(ci.isSSL());
         trans.init();
         trans.writeInt(Constants.TCP_PROTOCOL_VERSION_6);
-        trans.writeInt(Constants.TCP_PROTOCOL_VERSION_12);
+        trans.writeInt(Constants.TCP_PROTOCOL_VERSION_13);
         trans.writeString(db);
         trans.writeString(ci.getOriginalURL());
         trans.writeString(ci.getUserName());
@@ -118,6 +118,11 @@ public class SessionRemote extends SessionWithState implements DataHandler {
             done(trans);
             clientVersion = trans.readInt();
             trans.setVersion(clientVersion);
+            if (clientVersion >= Constants.TCP_PROTOCOL_VERSION_13) {
+                if (ci.getFileEncryptionKey() != null) {
+                    trans.writeBytes(ci.getFileEncryptionKey());
+                }
+            }
             trans.writeInt(SessionRemote.SESSION_SET_ID);
             trans.writeString(sessionId);
             done(trans);

h2/src/main/org/h2/mvstore/FileStore.java

@@ -130,9 +130,9 @@ public class FileStore {
         try {
             file = f.open(readOnly ? "r" : "rw");
             if (encryptionKey != null) {
-                byte[] password = FilePathCrypt.getPasswordBytes(encryptionKey);
+                byte[] key = FilePathCrypt.getPasswordBytes(encryptionKey);
                 encryptedFile = file;
-                file = new FilePathCrypt.FileCrypt(fileName, password, file);
+                file = new FilePathCrypt.FileCrypt(fileName, key, file);
             }
             file = FilePathCache.wrap(file);
             fileSize = file.size();

h2/src/main/org/h2/mvstore/MVStore.java

@@ -103,6 +103,8 @@ MVStore:
 - storage that splits database into multiple files,
     to speed up compact and allow using trim
     (by truncating / deleting empty files)
+- add new feature to file systems that avoid copying data
+    (reads should return a ByteBuffer, not write into one)
 
 */
 

h2/src/main/org/h2/mvstore/db/MVTableEngine.java

@@ -46,7 +46,7 @@ public class MVTableEngine implements TableEngine {
         if (store != null) {
             return store;
         }
-        byte[] key = db.getFilePasswordHash();
+        byte[] key = db.getFileEncryptionKey();
         String dbPath = db.getDatabasePath();
         MVStore.Builder builder = new MVStore.Builder();
         if (dbPath == null) {
@@ -67,9 +67,9 @@ public class MVTableEngine implements TableEngine {
                 }
             }
             if (key != null) {
-                char[] password = new char[key.length];
-                for (int i = 0; i < key.length; i++) {
-                    password[i] = (char) key[i];
+                char[] password = new char[key.length / 2];
+                for (int i = 0; i < password.length; i++) {
+                    password[i] = (char) (((key[i + i] & 255) << 16) | ((key[i + i + 1]) & 255));
                 }
                 builder.encryptionKey(password);
             }
@@ -87,12 +87,12 @@ public class MVTableEngine implements TableEngine {
                 int errorCode = DataUtils.getErrorCode(e.getMessage());
                 if (errorCode == DataUtils.ERROR_FILE_CORRUPT) {
                     if (key != null) {
-                        throw DbException.get(ErrorCode.FILE_ENCRYPTION_ERROR_1, fileName);
+                        throw DbException.get(ErrorCode.FILE_ENCRYPTION_ERROR_1, e, fileName);
                     }
                 } else if (errorCode == DataUtils.ERROR_FILE_LOCKED) {
-                    throw DbException.get(ErrorCode.DATABASE_ALREADY_OPEN_1, fileName);
+                    throw DbException.get(ErrorCode.DATABASE_ALREADY_OPEN_1, e, fileName);
                 }
-                throw DbException.get(ErrorCode.FILE_CORRUPTED_1, fileName);
+                throw DbException.get(ErrorCode.FILE_CORRUPTED_1, e, fileName);
             }
         }
         db.setMvStore(store);

h2/src/main/org/h2/mvstore/db/ValueDataType.java

@@ -90,7 +90,8 @@ public class ValueDataType implements DataType {
             int bl = bx.length;
             int len = Math.min(al, bl);
             for (int i = 0; i < len; i++) {
-                int comp = compareValues(ax[i], bx[i], sortTypes[i]);
+                int sortType = sortTypes[i];
+                int comp = compareValues(ax[i], bx[i], sortType);
                 if (comp != 0) {
                     return comp;
                 }

h2/src/main/org/h2/server/TcpServerThread.java

@@ -84,12 +84,12 @@ public class TcpServerThread implements Runnable {
                 int minClientVersion = transfer.readInt();
                 if (minClientVersion < Constants.TCP_PROTOCOL_VERSION_6) {
                     throw DbException.get(ErrorCode.DRIVER_VERSION_ERROR_2, "" + clientVersion, "" + Constants.TCP_PROTOCOL_VERSION_6);
-                } else if (minClientVersion > Constants.TCP_PROTOCOL_VERSION_12) {
-                    throw DbException.get(ErrorCode.DRIVER_VERSION_ERROR_2, "" + clientVersion, "" + Constants.TCP_PROTOCOL_VERSION_12);
+                } else if (minClientVersion > Constants.TCP_PROTOCOL_VERSION_13) {
+                    throw DbException.get(ErrorCode.DRIVER_VERSION_ERROR_2, "" + clientVersion, "" + Constants.TCP_PROTOCOL_VERSION_13);
                 }
                 int maxClientVersion = transfer.readInt();
-                if (maxClientVersion >= Constants.TCP_PROTOCOL_VERSION_12) {
-                    clientVersion = Constants.TCP_PROTOCOL_VERSION_12;
+                if (maxClientVersion >= Constants.TCP_PROTOCOL_VERSION_13) {
+                    clientVersion = Constants.TCP_PROTOCOL_VERSION_13;
                 } else {
                     clientVersion = minClientVersion;
                 }
@@ -140,6 +140,11 @@ public class TcpServerThread implements Runnable {
                 transfer.writeInt(SessionRemote.STATUS_OK);
                 transfer.writeInt(clientVersion);
                 transfer.flush();
+                if (clientVersion >= Constants.TCP_PROTOCOL_VERSION_13) {
+                    if (ci.getFilePasswordHash() != null) {
+                        ci.setFileEncryptionKey(transfer.readBytes());
+                    }
+                }
                 server.addConnection(threadId, originalURL, ci.getUserName());
                 trace("Connected");
             } catch (Throwable e) {

h2/src/main/org/h2/store/FileLock.java

@@ -238,7 +238,7 @@ public class FileLock implements Runnable {
             transfer.setSocket(socket);
             transfer.init();
             transfer.writeInt(Constants.TCP_PROTOCOL_VERSION_6);
-            transfer.writeInt(Constants.TCP_PROTOCOL_VERSION_12);
+            transfer.writeInt(Constants.TCP_PROTOCOL_VERSION_13);
             transfer.writeString(null);
             transfer.writeString(null);
             transfer.writeString(id);

h2/src/main/org/h2/tools/Backup.java

@@ -165,7 +165,6 @@ public class Backup extends Tool {
                     out.println("Processed: " + fileName);
                 }
             }
-            zipOut.closeEntry();
             zipOut.close();
         } catch (IOException e) {
             throw DbException.convertIOException(e, zipFileName);

h2/src/main/org/h2/tools/ChangeFileEncryption.java

@@ -6,12 +6,22 @@
  */
 package org.h2.tools;
 
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.channels.FileChannel;
 import java.sql.SQLException;
 import java.util.ArrayList;
+
+import org.h2.engine.Constants;
 import org.h2.message.DbException;
 import org.h2.security.SHA256;
 import org.h2.store.FileLister;
 import org.h2.store.FileStore;
+import org.h2.store.fs.FileChannelInputStream;
+import org.h2.store.fs.FileChannelOutputStream;
+import org.h2.store.fs.FilePath;
+import org.h2.store.fs.FilePathCrypt;
 import org.h2.store.fs.FileUtils;
 import org.h2.util.Tool;
 
@@ -28,6 +38,8 @@ public class ChangeFileEncryption extends Tool {
     private String cipherType;
     private byte[] decrypt;
     private byte[] encrypt;
+    private byte[] decryptKey;
+    private byte[] encryptKey;
 
     /**
      * Options are case sensitive. Supported options are:
@@ -140,12 +152,16 @@ public class ChangeFileEncryption extends Tool {
                     throw new SQLException("The file password may not contain spaces");
                 }
             }
+            change.encryptKey = FilePathCrypt.getPasswordBytes(encryptPassword);
+            change.encrypt = getFileEncryptionKey(encryptPassword);
+        }
+        if (decryptPassword != null) {
+            change.decryptKey = FilePathCrypt.getPasswordBytes(decryptPassword);
+            change.decrypt = getFileEncryptionKey(decryptPassword);
         }
         change.out = out;
         change.directory = dir;
         change.cipherType = cipher;
-        change.decrypt = getFileEncryptionKey(decryptPassword);
-        change.encrypt = getFileEncryptionKey(encryptPassword);
 
         ArrayList<String> files = FileLister.getDatabaseFiles(dir, db, true);
         FileLister.tryUnlockDatabase(files, "encryption");
@@ -173,6 +189,14 @@ public class ChangeFileEncryption extends Tool {
     }
 
     private void process(String fileName) {
+        if (fileName.endsWith(Constants.SUFFIX_MV_FILE)) {
+            try {
+                copy(fileName);
+            } catch (IOException e) {
+                throw DbException.convertIOException(e, "Error encrypting / decrypting file " + fileName);
+            }
+            return;
+        }
         FileStore in;
         if (decrypt == null) {
             in = FileStore.open(null, fileName, "r");
@@ -182,6 +206,42 @@ public class ChangeFileEncryption extends Tool {
         in.init();
         copy(fileName, in, encrypt);
     }
+    
+    private void copy(String fileName) throws IOException {
+        if (FileUtils.isDirectory(fileName)) {
+            return;
+        }
+        FileChannel fileIn = FilePath.get(fileName).open("r");
+        if (decryptKey != null) {
+            fileIn = new FilePathCrypt.FileCrypt(fileName, decryptKey, fileIn);
+        }
+        InputStream inStream = new FileChannelInputStream(fileIn, true);
+        String temp = directory + "/temp.db";
+        FileUtils.delete(temp);
+        FileChannel fileOut = FilePath.get(temp).open("rw");
+        if (encryptKey != null) {
+            fileOut = new FilePathCrypt.FileCrypt(temp, encryptKey, fileOut);
+        }
+        OutputStream outStream = new FileChannelOutputStream(fileOut, true);
+        byte[] buffer = new byte[4 * 1024];
+        long remaining = fileIn.size();
+        long total = remaining;
+        long time = System.currentTimeMillis();
+        while (remaining > 0) {
+            if (System.currentTimeMillis() - time > 1000) {
+                out.println(fileName + ": " + (100 - 100 * remaining / total) + "%");
+                time = System.currentTimeMillis();
+            }
+            int len = (int) Math.min(buffer.length, remaining);
+            len = inStream.read(buffer, 0, len);
+            outStream.write(buffer, 0, len);
+            remaining -= len;
+        }
+        inStream.close();
+        outStream.close();
+        FileUtils.delete(fileName);
+        FileUtils.moveTo(temp, fileName);
+    }
 
     private void copy(String fileName, FileStore in, byte[] key) {
         if (FileUtils.isDirectory(fileName)) {

h2/src/test/org/h2/test/db/TestMemoryUsage.java

@@ -59,7 +59,7 @@ public class TestMemoryUsage extends TestBase {
         deleteDb("memoryUsage");
         conn = getConnection("memoryUsage");
         eatMemory(4000);
-        for (int i = 0; i < 40000; i++) {
+        for (int i = 0; i < 4000; i++) {
             Connection c2 = getConnection("memoryUsage");
             c2.createStatement();
             c2.close();
@@ -105,18 +105,18 @@ public class TestMemoryUsage extends TestBase {
         deleteDb("memoryUsage");
         conn = getConnection("memoryUsage");
         Statement stat = conn.createStatement();
-        stat.execute("SET MAX_LENGTH_INPLACE_LOB 32768");
+        stat.execute("SET MAX_LENGTH_INPLACE_LOB 16384");
         stat.execute("SET CACHE_SIZE 8000");
         stat.execute("CREATE TABLE TEST(ID IDENTITY, DATA CLOB)");
         freeSoftReferences();
         try {
             int base = Utils.getMemoryUsed();
             for (int i = 0; i < 4; i++) {
-                stat.execute("INSERT INTO TEST(DATA) SELECT SPACE(32000) FROM SYSTEM_RANGE(1, 200)");
+                stat.execute("INSERT INTO TEST(DATA) SELECT SPACE(16000) FROM SYSTEM_RANGE(1, 400)");
                 freeSoftReferences();
                 int used = Utils.getMemoryUsed();
                 if ((used - base) > 16000) {
-                    fail("Used: " + (used - base));
+                    fail("Used: " + (used - base) + " i: " + i);
                 }
             }
         } finally {

h2/src/test/org/h2/test/store/TestConcurrent.java

@@ -9,7 +9,6 @@ import java.io.BufferedInputStream;
 import java.io.ByteArrayOutputStream;
 import java.io.FileOutputStream;
 import java.io.InputStream;
-import java.nio.channels.Channels;
 import java.nio.channels.FileChannel;
 import java.util.ConcurrentModificationException;
 import java.util.Iterator;
@@ -20,6 +19,7 @@ import java.util.concurrent.atomic.AtomicInteger;
 import org.h2.mvstore.MVMap;
 import org.h2.mvstore.MVMapConcurrent;
 import org.h2.mvstore.MVStore;
+import org.h2.store.fs.FileChannelInputStream;
 import org.h2.store.fs.FileUtils;
 import org.h2.test.TestBase;
 import org.h2.util.Task;
@@ -155,12 +155,7 @@ public class TestConcurrent extends TestMVStore {
 
     private static byte[] readFileSlowly(FileChannel file, long length) throws Exception {
         file.position(0);
-        InputStream in = new BufferedInputStream(Channels.newInputStream(file)) {
-            @Override
-            public void close() {
-                // don't close
-            }
-        };
+        InputStream in = new BufferedInputStream(new FileChannelInputStream(file, false));
         ByteArrayOutputStream buff = new ByteArrayOutputStream();
         for (int j = 0; j < length; j++) {
             int x = in.read();