I have been working in a large organization for quite a while now where I have seen a lot of testing going on. As an information security engineer, I naturally aligned more with testing and indeed, information security assurance does align well with testing: It’s done on a continuous basis, its results usually mean work for developers, operations people, system architects, etc. and not caring about it is equivalent to accepting unknown risks. Since I have been working in an environment where testing was paramount, I have been digging more and more into the testing literature.
All posts by msoos
Testing CryptoMiniSat using GoogleTest
Lately, I have been working quite hard on writing module tests for CryptoMinisat using GoogleTest. I’d like to share what I’ve learnt and what surprised me most about this exercise.
An example
First of all, let me show how a typical test looks like:
TEST_F(intree, fail_1)
{
s->add_clause_outer(str_to_cl(" 1, 2"));
s->add_clause_outer(str_to_cl("-2, 3"));
s->add_clause_outer(str_to_cl("-2, -3"));
inp->intree_probe();
check_zero_assigned_lits_contains(s, "-2");
}
Here we are checking that intree probing finds that the set of three binary clauses cause a failure and it enqueues “-2” at top level. If one looks at it, it’s a fairly trivial test. It turns out that most are in fact, fairly trivial if the system is set up well. This test’s setup is the following test fixture:
struct intree : public ::testing::Test {
intree()
{
must_inter = false;
s = new Solver(NULL, &must_inter);
s->new_vars(30);
inp = s->intree;
}
~intree()
{
delete s;
}
Solver* s;
InTree* inp;
bool must_inter;
};
CryptoMiniSat: 8000 commits later
The GitHub repository for CryptoMiniSat just hit 8000 commits. To celebrate this rather weird and crazy fact, let me put together a bit of a history.
The Beginnings
CryptoMiniSat began as a way of trying to prove that a probabilistic cryptographic scheme was not possible to break using SAT solvers. This was the year 2009, and I was working in Grenoble at INRIA. It was a fun time and I was working really hard to prove what I wanted to prove, to the point that I created a probabilistic SAT solver where one could add probability weights to clauses. The propagations and conflict engine would only work if the conflict or propagation was supported by multiple clauses. Thus began a long range of my unpublished work in SAT. This SAT solver, curiously, still works and solves problems quite well. It’s a lot of fun, actually — just add some random clause into the database that bans the only correct solution and the solver will find the correct solution. A bit of a hackery, but hey, it works. Also, it’s so much faster than doing that solving with the “right” systems, it’s not even worth comparing.
Continue reading CryptoMiniSat: 8000 commits later
STP and CryptoMiniSat in the competitions of 2015
Both the SAT Race and the SMT competition of 2015 are now over. The results are in and it’s time to draw some conclusions. CryptoMiniSat won the SAT Race’s incremental solving track, and the SMT solver that uses (among other solvers) CryptoMiniSat, CVC4, won the SMT competiton. These two wins are not unrelated — CryptoMiniSat is used in an incremental way by the CVC4 team. What CryptoMiniSat and STP (the other project I work on) hasn’t done is win their respective main competitions. Let me reflect on these quickly.
The SMT competition
The SMT competition’s quantifier free boolean vector track (QF_BV) was won by Boolector. Boolector is an amazing piece of engineering and so is the SAT solver it uses, lingeling. Second was CVC using CryptoMiniSat and third was STP. The reason for STP being third was because I managed to add a bug into STP and never fuzzed it. So now fuzzing is part of the continuous integration suite. Without this bug, STP would have come pretty close to Boolector, of which I’m proud of.
In general, STP needs to be greatly overhauled. It’s a legacy codebase with global variables, weird dependencies and long list of weirdnesses. It’s also performing amazingly well, so one needs to be careful how to approach it. I have been trying to cut it up and slice it any way I can so it makes more sense to read the code and work on it, but it’s still quite hard.
The SAT competition
The SAT competition had its share of surprises. One such surprise was that lingeling didn’t win any main prizes. I wonder why it wasn’t submitted to the incremental track, for example. Another surprise was the 1st and 2nd places of two solvers by the same author, Jingchao Chen, who unfortunately I have never met or conversed with. I was also a bit disappointed that CryptoMiniSat didn’t make it too high on the list. There have been a number of issues I tried to fix, but apparently that wasn’t enough. I will be interested in the sources of the winning solvers, see if I can learn something from them.
It’s really hard to create winning SAT solvers that behave well in all circumstances because it takes massive amount of time to code them up and properly test them. Especially the latter is really complicated and writing good test harnesses takes concentrated effort. My test scripts are easily over a thousand lines of code, mostly python. The continuous integration and build systems (both necessary for timely releases) are another thousand, and my AWS scripts for performance testing is almost a thousand as well. That’s more code than some early, well-performing SAT solvers had.
Continue reading STP and CryptoMiniSat in the competitions of 2015
Setting up encrypted mail in Chrome and Gmail
The use of Gmail is now ubiquitous. Unfortunately, it’s easy to read email in transit and some national governments abuse their power to read email in transit. I have always been using PGP to encrypt email, and today I thought I’d put down how to communicate with me, or with your friends, using signed and encrypted mail. I think the biggest reason email encryption is not being used is because it’s hard to set up. So, here is a simple, step-by-step tutorial that is easy to follow.
Installing and creating a key
- Install Mailvelope . Click “add to chrome”, pop-up appears, click “add”
- little padlock icon appears on the top right of your Chrome
- Click little padlock icon, click “Options”
- At the bottom, click “Generate key”
- Fill in Name (you can put fictitious name, it’s good!), Email (the email you want to use, e.g. Jon.Doe@gmail.com), put in a password that you will remember. This password is never sent anywhere. It’s used so that when you want to read email that is encrypted to you, the encryption keys can be accessed.
- Click submit, wait for the generation to finish.
- Setup is done!
Continue reading Setting up encrypted mail in Chrome and Gmail